AWQ GEMM checkpoints store each linear as qweight, qzeros, and scales inside safetensors or a torch state dict. This spec is the pack in awq/modules/linear/gemm.py WQLinear_GEMM.from_linear at 88e4c76b20755. The quantizer selects that class when version == "gemm". GEMV, GEMV-fast, Marlin, and ExLlama classes in the same repo are different layouts and are not specified here. Group size and bit width come from quantization_config, not from the tensor file.
.safetensors .pt .bin
What it is
weight pack
Byte order
little
Dimension order
row major
Compression
none
Metadata
json
Loading
Loading it into a library can execute code, depending on the loader and the file.
float16. shape [in_features // group_size, out_features] after the gemm transpose in quantizer.py
bias
float16. optional, shape [out_features]
Reader
The procedure that finds each tensor's bytes.
read
pseudo-python
ORDER = [0, 2, 4, 6, 1, 3, 5, 7]
def unpack_awq_gemm(qweight, qzeros, scales, group_size):
# qweight int32 [in_features, packed_out]
in_f, packed_out = qweight.shape
out_f = packed_out * 8
w = empty((in_f, out_f), int32)
for col in range(packed_out):
for i in range(8):
w[:, col * 8 + ORDER[i]] = (qweight[:, col] >> (i * 4)) & 0xF
z = empty((qzeros.shape[0], out_f), int32)
for col in range(qzeros.shape[1]):
for i in range(8):
z[:, col * 8 + ORDER[i]] = (qzeros[:, col] >> (i * 4)) & 0xF
# dequant along in_features: (code - zero) * scale, scale/zero row = k // group_size
out = empty((out_f, in_f), float32)
for k in range(in_f):
g = k // group_size
out[:, k] = (w[k].float() - z[g].float()) * scales[g].float()
return out # [out_features, in_features]
Reject the file when
Listed in the spec's order. Only rules marked MUST that reject the file are shown. The other IDs are warnings or implementation limits.
AWQ-004. Unpack with order_map [0, 2, 4, 6, 1, 3, 5, 7], not sequential nibbles
AWQ-005. Do not add 1 to zeros. AWQ stores the zero that dequant subtracts.
Example
Example file. AutoAWQ WQLinear_GEMM.from_linear wrote the packed tensors on CPU. That writer stores scales as float16; this file stores them as float32. Names, dtypes, shapes, and stored bytes match the recorded hashes. This page does not unpack the weights. The header below is spaced for reading; the file pads that JSON with spaces out to a multiple of 8.