Files beside the weights · .json

Hugging Face tokenizer.json

A JSON tokenizer file, not a tensor container. Converters need it to rebuild GGUF tokenizer.ggml.* keys. At tokenizers a73dbb3f65b2 the normative writer/reader is version "2.0", specified in tokenizers/tk-serialize/SPEC.md. Version "1.0" is what existing Hub files use. The slim reader refuses 1.0 until tk-convert rewrites it. This spec records both shapes and does not list every normalizer type beyond the SPEC.md tables.

.json

What it is: beside the weights
Compression: none
Metadata: json
Loading: Loading it into a library does not execute code.

How to read it

Where the values live

nbytes: 0

TokenizerV2

version: string. constant 2.0
truncation: null or object
padding: null or object
added_tokens: array. objects with id, content, single_word, lstrip, rstrip, normalized, special
normalizer: component or null
pre_tokenizer: component or null
post_processor: TemplateProcessing or null
decoder: component or null
model: BPE or Unigram or WordPiece or WordLevel

BpeModel

type: string. constant BPE
vocab: object. token string to id
merges: array. [left, right] pairs
byte_level: bool

Reader

The procedure that finds each tensor's bytes.

read

pseudo-python

def read_tokenizer_json(path):
    doc = json.load(open(path, encoding="utf-8"))
    ver = doc.get("version")
    if ver == "2.0":
        model = doc["model"]
        if "type" not in model: raise Invalid("2.0 model type")
        return doc
    if ver == "1.0" or "model" in doc:
        raise Unsupported("tokenizer.json 1.0; run tk-convert before the 2.0 reader")
    raise Invalid("not a tokenizer.json")

Reject the file when

Listed in the spec's order. Only rules marked MUST that reject the file are shown. The other IDs are warnings or implementation limits.

TOK-001. Parse as UTF-8 JSON. An object is required.
TOK-002. version 2.0 follows SPEC.md. A 2.0 reader must reject version 1.0 instead of guessing a hybrid.
TOK-004. BPE merge rank in 2.0 is the array index, not a score stored beside the pair

Sources

Nick Reynolds · @goldnode · rev 2.4