$ whoami

• Kumar Shivendu

• Engineer @ Qdrant

• I search, databases, and performance.

• Token-Native Storage

Topics to cover

• The payload nobody compresses

• The compression ladder, and why every rung falls short

• Tokens as the storage format: free compression

• An easy win in every BPE tokenizer

• The second win: the agent read/write flip

• What breaks, and what the ecosystem needs

The payload nobody compresses

• A vector DB record is a vector plus a text payload

• We compress the vector obsessively: product quantization, binary quantization, Matryoshka

• The text payload gets raw UTF-8, or LZ4 if you're lucky. On English, 1.27x

• Text is the heavy field: thousands of characters, one byte each

What our engines actually do

• Qdrant, Elasticsearch, Postgres: the payload gets an LZ-family codec, usually LZ4

• On English, 512-token chunks: 1.27x

• Cons:

  • Barely compresses. 100 GB of text becomes 79 GB
  • Every copy pays it again: snapshots, backups, WAL, replicas, network egress

Compress harder? Train a dictionary?

• gzip -9 1.92x · zstd -19 1.94x · brotli q11 2.57x

• zstd --train learns a 112 KB dictionary from your corpus: 2.72x on English, 4.52x on Hindi. The fairest competitor in this talk

• Cons:

  • brotli takes 2,777us to encode one 512-token chunk, ~1,000x LZ4's 2.9us
  • The dictionary is yours alone. It ships with your data, nobody else can read it
  • Neither vocabulary is shared or standard

What if we stored the model's own format?

• Every rung optimizes the container and never questions the contents

• Whatever we store, the model turns it into token IDs before it can read a word

• So what if the token IDs were the stored form?

The napkin math

avg English word = 5 chars + 1 space = 6 bytes/word

UTF-8:  6 bytes/word x 3/4 word/token = 4.5 bytes/token
Tokens: 1 r50k token ID as uint16     = 2.0 bytes/token

ratio: 4.5 / 2.0 = ~2.25x

• One BPE token covers about 3/4 of a word

• r50k's vocabulary is 50,257 tokens, which fits in a uint16

• This is the whole idea. Everything after this slide is checking it

Tokens

• BPE (Byte Pair Encoding) starts from raw bytes and repeatedly merges the most frequent adjacent pair

• "storage" is one token. "Token-native" is three: Token + - + native

• Every word carries its leading space into the token: " cat" is 4 bytes of UTF-8, but 1 token

• r50k 50,257 (2 bytes) · cl100k 100,277 · o200k 200,019 (3 bytes)

Two levers on top of the IDs

• Asymmetric Numeral Systems (ANS) is an entropy coder: frequent tokens get shorter codes. "the" is ~40x more common than "embeddings", so it earns fewer bits

• Or re-rank the IDs by frequency and pack them with streamvbyte, a variable-length integer codec

• One table, trained once on a corpus, reused for every document. Not per-document, or you would ship ~900 bytes of table with each 512-token chunk

An easy win in every BPE tokenizer

• Running the frequency histogram, I found BPE hands out IDs in merge-discovery order, not by how often a token is used

• A token you use constantly can sit at ID 40,000. A rare one sits at ID 12

• Variable-length integer codecs pay for big numbers, so this ordering leaves compression on the table for everyone downstream

• Re-ranking by frequency on English: 2.13x → 2.60x. Half of +freq's gain is the remap alone

Fixing it:

# BPE numbers tokens by merge order. streamvbyte pays for big integers.
# So renumber once, by real-world frequency, and every document gets smaller.
corpus_ids = np.array(enc.encode(corpus_text), dtype=np.int64)
counts  = np.bincount(corpus_ids, minlength=VOCAB)
order   = np.argsort(-counts)                        # most -> least frequent
rank_of = np.empty(VOCAB, dtype=np.uint32)
rank_of[order] = np.arange(VOCAB, dtype=np.uint32)   # token id   -> freq rank
token_of_rank  = order                               # freq rank  -> token id

def compress(text):
    ids   = np.array(enc.encode(text), dtype=np.int64)
    ranks = rank_of[ids]                    # SAME tokens, new numbers
    out = np.zeros(len(ranks) * 2 + 1024, dtype=np.uint32)
    n = codec.encodeArray(ranks, len(ranks), out, len(out))       # streamvbyte
    return len(ranks).to_bytes(4, "big") + out[:n].tobytes()

Two representations, paid for twice

   WRITE (agent)                    READ (agent)
   +------------+                   +------------+
   | token IDs  |                   | token IDs  |
   +------------+                   +------------+
         | detokenize  50us               ^ tokenize  237us
         v                                |
   +------------+                   +------------+
   | UTF-8 text |                   | UTF-8 text |
   +------------+                   +------------+
         | LZ4 compress  2.9us            ^ LZ4 decompress  1.0us
         v                                |
   +----------------[ DISK ]-----------------+

• Stored once, kept in two forms, translated on every access

Agent read

• LZ4 decompresses in 1.0us, then spends 236.7us tokenizing text the model will immediately consume as IDs

• Token-native serves the IDs directly: 3.6us with +freq, 28.8us with +ANS

• That's ~66x on the fastest token-native path

• Read is where it compounds: it happens on every retrieval, forever. Writing happens once

Detokenize, then tokenize again on every read

But humans still read this data

• True cost: token-native pays ~50us to detokenize before a human sees anything

• But in a RAG or agent loop, a search returns 10 chunks and the agent reads all of them

• The human sees one answer, once, at the end

• So detokenize once, at the edge. Maybe in the frontend

Works well, but... tokenizers got faster

• My whole read argument assumes tokenizing costs ~237us. I measured that with tiktoken

• gigatoken encodes the same chunk in 13.3us, 13.9x faster. Decode barely moves, 1.19x

• My blog post said a 30M-request/month workload wastes 42 hours/month re-tokenizing

• With a fast tokenizer that becomes ~1.1 hours/month. I was off by 38x

So which claims actually survive?

• The compression and write arguments never depended on a slow tokenizer. Part of the read argument did

Limitations

• Pays off end to end only if reader and writer share a tokenizer. Anthropic and Google (except Gemma) haven't published theirs

• Hosted LLM APIs take text and return text, so you need to own the inference stack

• My frequency table is corpus-specific. Point it at a corpus it wasn't built on and the ratio drops

• mxbai-embed-large-v1 compresses better at 3.56x, but 80.4% of articles decode corrupted. BERT lowercases: "Qdrant" becomes "qdrant"

Interface

// One-time: register the tokenizer for a field.
PUT /collections/documents/index
{ "schema": { "text": { "type": "token", "tokenizer": "o200k" } } }
// Write: hand over the IDs the model just produced. A plain string also works.
PUT /collections/documents/points
{ "points": [{ "id": 123, "vector": [0.12, -0.34],
    "payload": { "text": [1858, 6427, 20272, 318, 257] } }] }
// Read: ask per field. Default stays "text", existing clients see no change.
POST /collections/documents/points/search
{ "vector": [0.1], "limit": 10, "with_payload": { "text": "tokens" } }

• Ask for "tokens" to skip detokenization. Swapping codec is not a data migration

Two asks for the AI labs

• Sort the vocabulary by corpus frequency before you publish it. It costs one sort, and it hands every downstream user free compression

• Publish the tokenizers. We need a UTF-8-like standard for tokens, so a stored payload isn't locked to one vendor's model version

• You don't have to wait for either of these. Remap on your own corpus and you'll beat the vendor's ordering anyway

Summary

• A tokenizer that covers your script is free compression: 2.25x raw, 3.40x with a coder

• The gain is the tokenizer, not the coder. And BPE's merge-order IDs leave more on the table

• A byte store re-tokenizes on every read. Store what the model speaks

• Find me at

  • kshivendu.dev/twitter

References

• Paper: Token-Native Storage (arXiv 2608.02376)

• Post: kshivendu.dev/blog/token-storage

• Benchmarks: github.com/KShivendu/token-storage

• tiktoken · constriction (ANS) · streamvbyte · gigatoken

• Kalcher, Compressing token IDs with frequency ordering (2026) · NVIDIA Megatron-Core tokenized .bin corpora