Text Classification with Python 3.14's ZSTD Module

Python 3.14 introduces the compression.zstd module to the standard library, and it unlocks a surprisingly powerful approach to text classification without machine learning models. By measuring how well a compressor can squeeze two texts together, you can determine their similarity — a technique called Normalized Compression Distance (NCD) — and now Zstandard makes it fast enough for production workloads.

How Does Compression-Based Text Classification Actually Work?

The core idea behind compression-based classification is rooted in information theory. When a compression algorithm like Zstandard encounters a block of text, it builds an internal dictionary of patterns. If two texts share similar vocabulary, syntax, and structure, compressing them together produces a result only slightly larger than compressing the bigger text alone. If they are unrelated, the concatenated compressed size approaches the sum of both individual sizes.

This relationship is captured by the Normalized Compression Distance formula: NCD(x, y) = (C(xy) - min(C(x), C(y))) / max(C(x), C(y)), where C(x) is the compressed size of text x, and C(xy) is the compressed size of the two texts concatenated. An NCD value near 0 means the texts are highly similar, while a value near 1 means they share almost no informational content.

What makes this technique remarkable is that it requires no training data, no tokenization, no embeddings, and no GPU. The compressor itself acts as the learned model of the text's structure. Research published in papers like "Low-Resource Text Classification: A Parameter-Free Classification Method with Compressors" (2023) demonstrated that gzip-based NCD rivalled BERT on certain benchmarks, sparking renewed interest in the approach.

Why Is Python 3.14's Zstandard Module a Game-Changer for NCD?

Before Python 3.14, using Zstandard required installing the third-party python-zstandard package. The new compression.zstd module, introduced via PEP 784, ships directly with CPython. This means zero dependency overhead and a guaranteed, stable API backed by Meta's battle-tested libzstd. For classification tasks specifically, Zstandard offers several advantages over gzip or bzip2:

• Speed: Zstandard compresses 3-5x faster than gzip at comparable ratios, making batch classification over thousands of documents viable in seconds rather than minutes
• Tunable compression levels: Levels 1 through 22 let you trade speed for ratio, allowing you to calibrate NCD precision against throughput requirements
• Dictionary support: Pre-trained Zstandard dictionaries can dramatically improve compression of small texts (under 4KB), which is exactly the document size range where NCD accuracy matters most
• Streaming API: The module supports incremental compression, enabling classification pipelines that process texts without loading entire corpora into memory
• Standard library stability: No version conflicts, no supply chain risk — from compression import zstd works on every Python 3.14+ installation

Key insight: Compression-based classification works best when you need a quick, dependency-free baseline that handles multilingual text natively. Because compressors operate on raw bytes rather than language-specific tokens, they classify Chinese, Arabic, or mixed-language documents just as effectively as English — no language model required.

What Does a Practical Implementation Look Like?

A minimal NCD classifier in Python 3.14 fits in under 30 lines. You encode each reference text (one per category), then for each new document, compute the NCD against every reference and assign the category with the lowest distance. Here is the core logic:

First, import the module with from compression import zstd. Define a function that accepts two byte strings, compresses each individually, compresses their concatenation, and returns the NCD score. Then build a dictionary mapping category labels to representative sample texts. For each incoming document, iterate over categories, compute NCD, and select the minimum.

In benchmarks against the AG News dataset (four-class news classification), this approach using Zstandard at compression level 3 achieves roughly 62-65% accuracy — no training step, no model download, and classification speed of approximately 8,000 documents per second on a single CPU core. Raising the compression level to 10 pushes accuracy to around 68% at the cost of reducing throughput to about 2,500 documents per second. These numbers do not match fine-tuned transformers, but they provide a strong baseline for prototyping, data labeling triage, or environments where installing ML dependencies is impractical.

How Does NCD Compare to Traditional ML Classification?

The honest answer is that NCD is not a replacement for transformer-based classifiers in high-stakes production systems. Models like BERT or GPT-based classifiers achieve 94%+ accuracy on standard benchmarks. However, NCD with Zstandard occupies a unique niche. It excels in cold-start scenarios where you have fewer than 50 labeled examples per class — a situation where even fine-tuned models struggle. It requires zero training time, handles any language or encoding without modification, and runs entirely on CPU with constant memory.

For businesses managing large volumes of incoming content — support tickets, social media mentions, product reviews — a Zstandard NCD classifier can serve as a first-pass router that categorises documents in real time before more expensive models refine the results. This two-stage pipeline reduces inference costs significantly while maintaining overall accuracy. Platforms processing user-generated content at scale, such as Mewayz's 207-module business OS used by over 138,000 entrepreneurs, benefit from lightweight classification to route messages, tag content, and personalise user experiences without heavy infrastructure.

What Are the Limitations and Best Practices?

Compression-based classification has known limitations you should account for. Short texts (under 100 bytes) produce unreliable NCD scores because the compressor does not have enough data to build meaningful patterns. The technique is also sensitive to the choice of reference texts — poorly chosen representatives degrade accuracy sharply. And because NCD is a distance metric rather than a probabilistic model, it does not naturally produce confidence scores.

To get the most from this approach: use reference texts of at least 500 bytes per category, experiment with concatenating multiple examples per class (2-3 representative documents joined together yield better compression dictionaries), normalise text casing and whitespace before compression, and benchmark across Zstandard compression levels 3, 6, and 10 to find your speed-accuracy sweet spot. For small-text classification, pre-train a Zstandard dictionary on your domain corpus — this single step can improve accuracy by 8-12 percentage points on short documents.

Frequently Asked Questions

Does compression-based classification work for sentiment analysis?

It can, but with caveats. Sentiment analysis requires detecting subtle tonal differences within structurally similar texts. NCD works better for topic classification where documents in different categories use distinct vocabularies. For sentiment, accuracy typically lands around 55-60% — better than random, but not production-ready on its own. Combining NCD features with a lightweight logistic regression model improves results considerably.

Can I use the compression.zstd module in Python versions before 3.14?

No. The compression.zstd module is new in Python 3.14. For earlier versions, install the python-zstandard package from PyPI, which provides equivalent compress() and decompress() functions. The NCD logic remains identical — only the import statement changes. Once you upgrade to 3.14, you can drop the third-party dependency entirely.

How does Zstandard NCD perform compared to TF-IDF with cosine similarity?

On multi-class topic classification with balanced datasets, TF-IDF plus cosine similarity typically achieves 75-82% accuracy compared to Zstandard NCD's 62-68%. However, TF-IDF requires a fitted vectoriser, a defined vocabulary, and language-specific stopword lists. Zstandard NCD requires none of this preprocessing, works across languages out of the box, and classifies new documents in constant time regardless of vocabulary size. For rapid prototyping or multilingual environments, NCD is often the faster path to a working system.

Whether you are building automated content pipelines, routing customer messages, or prototyping classification logic for your digital business, Python 3.14's built-in Zstandard support makes compression-based NCD more accessible than ever. If you are looking for an all-in-one platform to manage your business content, products, courses, and customer interactions, start building with Mewayz today and put these techniques to work across your entire operation.