Quantized weight packs · .safetensors

GPTQ packed checkpoint (safetensors / torch)

GPTQ does not define a container. A published checkpoint is safetensors or a torch state dict whose linear layers are stored as qweight, qzeros, scales, and g_idx, plus quantize_config.json (or quantization_config). This spec specifies the pack written by AutoGPTQ qlinear_cuda.QuantLinear (QUANT_TYPE "cuda") at 9f7d37072917. Marlin and the ExLlama kernels in the same tree have their own pack methods and are not this layout. The container bytes are the safetensors spec or the pytorch spec.

.safetensors .pt .bin

What it is: weight pack
Byte order: little
Dimension order: row major
Compression: none
Metadata: json
Loading: Loading it into a library can execute code, depending on the loader and the file.

How to read it

Where the values live

name: module prefix; the logical weight is prefix + '.qweight' plus the sibling tensors
dtype: none as a canonical element type; the packed words are int32. Logical values are unsigned bits-wide integers.
shape: [out_features, in_features] after unpack. qweight's on-disk shape is [in_features // 32 * bits, out_features].
data_offset: container offset of the qweight tensor
nbytes: qweight.nbytes + qzeros.nbytes + scales.nbytes + g_idx.nbytes
alignment: 1. no extra alignment beyond the container

GptqLayer

One quantized linear, stored as several tensors rather than one struct.

qweight: int32. shape [in_features // 32 * bits, out_features]
qzeros: int32. shape [ceil(in_features / group_size), out_features // 32 * bits]. Packed after zeros -= 1.
scales: float16 or the layer weight dtype. shape [ceil(in_features / group_size), out_features]
g_idx: int32. shape [in_features]. Group index of each input feature. Sequential i//group_size unless desc_act reordered groups.
bias: same dtype as scales. optional, shape [out_features]

QuantizeConfig

BaseQuantizeConfig.to_dict keys that select the pack. Other keys (damp_percent, sym, true_sequential) do not change the byte layout.

bits: int. 2, 3, 4, or 8 for this pack
group_size: int. -1 means one group covering in_features
desc_act: bool. when true, g_idx may be a permutation of groups rather than i//group_size
quant_method: string
checkpoint_format: string. gptq for this pack; marlin is a different pack

Reader

The procedure that finds each tensor's bytes.

read

pseudo-python

def unpack_gptq(qweight, qzeros, scales, g_idx, bits):
    # qweight: int32 [in_features/32*bits, out_features], row-major, little-endian
    # pack() advances i by 32 // bits once per qweight row, so in_features = rows * pack
    out_f = qweight.shape[1]
    pack = 32 // bits
    if bits in (2, 4, 8):
        in_f = qweight.shape[0] * pack
        w = zeros((in_f, out_f), uint32)
        row = 0
        i = 0
        while row < qweight.shape[0]:
            for j in range(pack):
                w[i + j] = (qweight[row] >> (bits * j)) & ((1 << bits) - 1)
            i += pack
            row += 1
    elif bits == 3:
        w = unpack_gptq_3bit(qweight)      # inverse of the 10 / split / 10 loop in pack()
    else:
        raise Unsupported(bits)
    z = unpack_zeros(qzeros, bits) + 1     # writer did zeros -= 1
    # dequant: intweight[k, n] = round((W[n, k] + scale_zeros[g]) / scales[g])
    # so W[n, k] = (code - zero) * scale, with g = g_idx[k]
    out = empty((out_f, w.shape[0]), float32)
    for k in range(w.shape[0]):
        g = g_idx[k]
        out[:, k] = (w[k].float() - z[g].float()) * scales[g].float()
    return out                               # [out_features, in_features]

Reject the file when

Listed in the spec's order. Only rules marked MUST that reject the file are shown. The other IDs are warnings or implementation limits.

GPTQ-001. bits in {2, 3, 4, 8} for the cuda pack
GPTQ-002. qweight.shape == [in_features // 32 * bits, out_features] and the in_features axis is divisible by 32
GPTQ-003. qzeros.shape[1] == out_features // 32 * bits
GPTQ-004. g_idx length == in_features and each index selects a scale/zero row
GPTQ-005. Reject checkpoint_format marlin rather than unpacking it with this routine
GPTQ-006. Add 1 to each unpacked zero before dequant. The writer subtracts 1 before packing.

Example

Example file. AutoGPTQ QuantLinear.pack wrote this safetensors file. The CUDA extension was not installed, so pack() is numpy. Names, dtypes, shapes, and stored bytes match the recorded hashes. This page does not unpack the weights. The header below is spaced for reading; the file pads that JSON with spaces out to a multiple of 8.

small-gptq.safetensors6,568 bytes

Name	Dtype	Shape	Bytes
gptq.scales	F32float32	8 × 32	1,024
gptq.g_idx	I32int32	256	1,024
gptq.qweight	I32int32	32 × 32	4,096
gptq.qzeros	I32int32	8 × 4	128

Header

{
  "gptq.scales": {
    "dtype": "F32",
    "shape": [
      8,
      32
    ],
    "data_offsets": [
      0,
      1024
    ]
  },
  "gptq.g_idx": {
    "dtype": "I32",
    "shape": [
      256
    ],
    "data_offsets": [
      1024,
      2048
    ]
  },
  "gptq.qweight": {
    "dtype": "I32",
    "shape": [
      32,
      32
    ],
    "data_offsets": [
      2048,
      6144
    ]
  },
  "gptq.qzeros": {
    "dtype": "I32",
    "shape": [
      8,
      4
    ],
    "data_offsets": [
      6144,
      6272
    ]
  }
}

Sources

Nick Reynolds · @goldnode · rev 2.3