Files beside the weights · .model

SentencePiece model (tokenizer.model)

A binary protobuf of sentencepiece.ModelProto, usually named tokenizer.model. It is not a tensor file. Converters read it to fill a Unigram or BPE vocab. There is no magic number; identification is the protobuf schema. The fields that change tokenization are pieces, trainer_spec, normalizer_spec, and denormalizer_spec. Field numbers below are the proto numbers at ab49d75a87c1.

.model

What it is
beside the weights
Byte order
little
Compression
none
Metadata
protobuf
Loading
Loading it into a library does not execute code.

How to read it

Where the values live

nbytes
0

ModelProto

pieces
repeated SentencePiece (field 1)
trainer_spec
TrainerSpec (field 2)
normalizer_spec
NormalizerSpec (field 3)
denormalizer_spec
NormalizerSpec (field 5)

SentencePiece

piece
string (field 1). must not be empty
score
float (field 2)
type
enum (field 3). NORMAL=1, UNKNOWN=2, CONTROL=3, USER_DEFINED=4, UNUSED=5, BYTE=6. Default NORMAL.

TrainerSpec

Only the fields a tokenizer loader needs. The proto has many training-only fields.

model_type
enum (field 3). UNIGRAM=1, BPE=2, WORD=3, CHAR=4. Default UNIGRAM.
vocab_size
int32 (field 4). default 8000

NormalizerSpec

name
string (field 1)
precompiled_charsmap
bytes (field 2)
add_dummy_prefix
bool (field 3). default true
remove_extra_whitespaces
bool (field 4). default true
escape_whitespaces
bool (field 5). default true

Reader

The procedure that finds each tensor's bytes.

read

pseudo-python
def read_sentencepiece(path):
    model = ModelProto()
    model.ParseFromString(open(path, "rb").read())
    pieces = [(p.piece, p.score, p.type or 1) for p in model.pieces]
    kind = model.trainer_spec.model_type or 1   # default UNIGRAM
    norm = model.normalizer_spec
    return pieces, kind, norm

Reject the file when

Listed in the spec's order. Only rules marked MUST that reject the file are shown. The other IDs are warnings or implementation limits.

  • SP-001. Decode as sentencepiece.ModelProto. Do not scan for a magic.
  • SP-002. piece type BYTE is 6 and UNUSED is 5
  • SP-003. An empty piece string is invalid

Sources

Nick Reynolds · @goldnode · rev 2.4