Build a text classifier

An isolated Python environment with pinned library versions.

All the preparation — downloading, comparing, converting, clustering — uses Python libraries with no real equivalent elsewhere. Python is the workshop; only the finished files reach whatever you ship.

brew install uv                      # or: pip install uv
mkdir text-classifier && cd text-classifier
uv venv --python 3.12
source .venv/bin/activate

uv pip install torch transformers huggingface-hub datasets \
               onnx onnxruntime numpy scikit-learn umap-learn pandas
uv pip install sentence-transformers model2vec   # convenience, for the bake-off
uv pip freeze > requirements-lock.txt            # freeze the exact versions

A local, dated snapshot of text only, with any labels you happen to have locked in a separate file.

from datasets import load_dataset
import pandas as pd

ds = load_dataset("PolyAI/banking77")
train = pd.DataFrame(ds["train"])      # 10,003 rows: text, label
test  = pd.DataFrame(ds["test"])       #  3,080 rows

# The whole point: hide the labels from yourself.
train[["text"]].to_parquet("work/texts.parquet")        # what you may look at
train[["label"]].to_parquet("sealed/referee.parquet")   # only for grading

# Read-only, into a dated folder, scrubbing as you read.
import pandas as pd, glob
df = pd.concat(pd.read_parquet(f) for f in glob.glob("raw/2026-10/**/*.parquet", recursive=True))
df = df[df["source"] == "<the one free-text field you care about>"]
df["text"] = df["text"].map(scrub)      # see the table below
df[["id", "text"]].to_parquet("work/texts.parquet")

A list of unique, genuinely typed texts, each with a fair weight, split so the test set is actually hard.

1. Drop junk — empty, under 3 characters, or fewer than 2 letters.
2. Set aside text nobody typed. Preset replies, template phrases, autofill, dropdown options and form defaults are not free text. Answer them with a lookup table and keep them out of everything downstream — otherwise a handful of fixed phrases dominate every cluster you find.
3. Normalise for comparing only — lowercase, straighten quotes, collapse whitespace. Keep the original wording for the model to read.
4. Merge duplicates and near-duplicates (about 92% character similarity catches typo variants without merging genuinely different messages).
5. Cap any one person's influence: weight = min(times_seen, distinct_users × 3). One prolific user should not define a label.
6. Split train / validation / test by group, not at random. Cluster near-duplicates first and send a whole group to one side. Random splitting puts "my card hasn't arrived" in train and "my card still hasn't arrived" in test, and your test score becomes a measure of how well you memorised.

A number the clever pipeline has to beat, produced in about fifteen lines.

Word counts with a linear model on top are a genuinely strong text classifier, and they take a minute to train. If you skip this you will have no idea whether your 25 MB encoder is earning its keep.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import f1_score

baseline = make_pipeline(
    TfidfVectorizer(sublinear_tf=True, ngram_range=(1, 2), min_df=2),
    LogisticRegression(max_iter=2000, class_weight="balanced"),
)
baseline.fit(train_texts, train_labels)                  # needs labels — see below
pred = baseline.predict(test_texts)
print(f1_score(test_labels, pred, average="macro"))      # macro: every label counts equally

Four candidates on the bench, one from each family, chosen for what they are rather than for a leaderboard position.

Hugging Face has thousands of embedding models. They collapse into a handful of kinds, and the kind decides far more than the individual model does. Pick one from each row and let Step 7 settle it.

For every candidate, four facts written down before you run anything.

These are not performance details. Get any of them wrong and the model silently returns worse vectors, with no error anywhere.

all-MiniLM-L6-v2      pooling_mode_mean_tokens: true   → mean
bge-small-en-v1.5     pooling_mode_cls_token:   true   → CLS
e5-small-v2           pooling_mode_mean_tokens: true   → mean

e5-small-v2     tok("query: my card hasn't arrived")    # mandatory, even for similarity
bge-small-v1.5  tok("my card hasn't arrived")           # instruction optional for v1.5
MiniLM-L6-v2    tok("my card hasn't arrived")           # nothing

One table, same protocol for every candidate, measured on your data rather than someone else's benchmark.

Leaderboard rankings are averages over tasks that are not yours. The differences between the models in Step 5 on your text are routinely smaller than the differences between runs — which is itself the most useful thing you can learn, because it means you should take the small one.

from sentence_transformers import SentenceTransformer
import numpy as np, time

CANDIDATES = [
    "minishlab/potion-base-8M",                 # static
    "sentence-transformers/all-MiniLM-L6-v2",    # small, mean pooling
    "BAAI/bge-small-en-v1.5",                   # small, CLS pooling
    "sentence-transformers/all-mpnet-base-v2",   # base
]

for name in CANDIDATES:
    m = SentenceTransformer(name)                # handles pooling per its card
    E = m.encode(texts, normalize_embeddings=True, batch_size=64)
    scores = [proto_macro_f1(E, referee_labels, seed=s) for s in range(5)]
    t = time.perf_counter(); m.encode([texts[0]]); ms = (time.perf_counter()-t)*1000
    print(f"{name:48s} {np.mean(scores):.3f} ± {np.std(scores):.3f}   {E.shape[1]:4d}d  {ms:6.1f} ms")

sentence-transformers applies each model's own pooling, so use it for the bake-off even though Part C exports ONNX by hand. Comparing candidates and shipping one are different jobs.

One model chosen, with the reasoning recorded somewhere a future colleague will find it.

The rule that holds up: take the smallest candidate whose score is inside the noise band of the best one. If mpnet-base scores 0.81 ± 0.02 and MiniLM scores 0.79 ± 0.02, those are the same number, and one of them is a fifth of the size.

Encoder: sentence-transformers/all-MiniLM-L6-v2  (Apache-2.0)
Chosen:  2026-10-05, from the Step 7 bake-off on 400 referee-labelled texts.
Because: within one standard deviation of all-mpnet-base-v2 (the best scorer)
         at 1/5 the parameters, 1/2 the vector width and no required prefix.
         Rejected potion-base-8M: clearly below the band, though 40x faster.
Revisit: if p99 text length exceeds 256 sub-words (its cap), or macro-F1 on
         the monthly fresh sample drops more than 0.05 below this baseline.

One file that takes sub-word IDs and returns one normalised sentence vector.

ONNX is a portable model format that runs almost anywhere via onnxruntime — browsers, phones, servers, embedded. Wrap the model in a small class that does the pooling and normalising inside the graph, then export the whole thing, so nothing downstream has to reimplement that arithmetic.

import torch
from transformers import AutoModel, AutoTokenizer

NAME = "sentence-transformers/all-MiniLM-L6-v2"   # whatever Step 8 chose
POOLING = "mean"                                   # from Step 6 — not a guess

tok   = AutoTokenizer.from_pretrained(NAME)
model = AutoModel.from_pretrained(NAME)

class SentenceEncoder(torch.nn.Module):
    def __init__(self, m, pooling): super().__init__(); self.m, self.pooling = m, pooling
    def forward(self, input_ids, attention_mask):
        words = self.m(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        if self.pooling == "cls":
            sent = words[:, 0]                                       # first token
        else:
            mask = attention_mask.unsqueeze(-1).float()
            sent = (words * mask).sum(1) / mask.sum(1).clamp(min=1e-9)  # mean, skip padding
        return torch.nn.functional.normalize(sent, dim=1)            # length 1

sample = tok("a sample sentence", return_tensors="pt",
             padding="max_length", truncation=True, max_length=128)

torch.onnx.export(
    SentenceEncoder(model, POOLING).eval(),
    (sample["input_ids"], sample["attention_mask"]),
    "model.onnx",
    input_names=["input_ids", "attention_mask"],
    output_names=["embedding"],
    dynamic_axes={"input_ids": {0: "batch", 1: "seq"},
                  "attention_mask": {0: "batch", 1: "seq"},
                  "embedding": {0: "batch"}},
    opset_version=17,
    dynamo=False,
)

About 4× smaller, with no measurable change in which label wins.

Weights are stored as 32-bit decimals — 4 bytes each. Rounding each to a 1-byte integer is called quantisation, and for an encoder like this it is close to free.

from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic("model.onnx", "model_int8.onnx", weight_type=QuantType.QInt8)
# then delete model.onnx — only the small one ships

Everything whatever-you-ship needs to turn text into exactly the IDs the model expects.

The model reads IDs, not text. The target has to do that conversion itself, so it needs the vocabulary (one sub-word per line, line number = ID) and every rule that was applied on the way.

vocab   = tok.get_vocab()
ordered = [t for t, _ in sorted(vocab.items(), key=lambda kv: kv[1])]
open("vocab.txt", "w").write("\n".join(ordered))

meta = {   # read every value off the tokenizer object — never type them by hand
  "model":   NAME,
  "pooling": POOLING, "dims": 384, "max_len": 128,
  "prefix":  "",      # e.g. "query: " for E5 — see Step 6
  "special_tokens": {"cls_id": tok.cls_token_id, "sep_id": tok.sep_token_id,
                     "pad_id": tok.pad_token_id, "unk_id": tok.unk_token_id},
  "tokenizer": {"lowercase": tok.do_lower_case, "strip_accents": True,
                "continuing_subword_prefix": "##"},
}

"my card is missing"  →  [CLS] my card is missing [SEP]   # markers wrap every text
"overdraft"           →  over ##draft                     # ## = glued to the previous piece
"Café"                →  cafe                             # lowercased, accent stripped
"thanks 🙏"           →  thanks [UNK]                      # unknown symbol

Proof that the converted file answers like the original — with each stage checked separately.

Encode the same sentences with the original model and with your files, and compare with a dot product (1.0 = identical).

Then the check that actually matters: run a few hundred real texts through both and confirm which label wins does not change. Vectors drift by a few percent; decisions should not.

One vector per text, saved alongside a matching list of ids.

import onnxruntime as ort, numpy as np
sess = ort.InferenceSession("model_int8.onnx")
vecs = []
for i in range(0, len(texts), 32):
    b = tok(texts[i:i+32], padding=True, truncation=True, max_length=128, return_tensors="np")
    vecs.append(sess.run(None, {"input_ids": b["input_ids"],
                                "attention_mask": b["attention_mask"]})[0])
E = np.vstack(vecs)                         # (n_texts, dims)
np.save("embeddings.npy", E)                # + save the ids in the same order

Similarity is now a dot product. Absolute values stay modest even for obvious matches — what matters is the gap between related and unrelated pairs:

0.71   "my card hasn't arrived"   vs  "still waiting on my new card"
0.15   "my card hasn't arrived"   vs  "how do I change my address"

Groups of "the same kind of thing", found without any labels, and reproducible forever after.

import umap, joblib
from sklearn.mixture import GaussianMixture

reducer = umap.UMAP(n_components=10, n_neighbors=15, min_dist=0.0,
                    metric="cosine", random_state=42).fit(E)       # 384 → 10 numbers
gmm     = GaussianMixture(n_components=40, covariance_type="full",
                          random_state=42).fit(reducer.embedding_)
cluster = gmm.predict(reducer.embedding_)

joblib.dump(reducer, "frozen/v1/umap.joblib")       # freeze — never refit v1
joblib.dump(gmm,     "frozen/v1/gmm.joblib")

Why squash to 10 dimensions first? In 384 dimensions, distances between points concentrate — almost everything looks roughly equally far from everything else — and clustering struggles. Reducing first gives it a space with real structure in it.

How many groups? Do not trust the textbook criterion (BIC). On text data it tends to keep asking for more groups indefinitely. Ask the question you actually care about instead: if each group became a label, how often would nearest-prototype get it right? Sweep the count, score each against your referee labels, and take the peak.

A readable name and a written description for every group worth keeping.

1. Pull keywords — words common inside the group and rare outside it (c-TF-IDF).
2. Draft a name from the top keywords, or have an LLM draft one from twenty sampled texts.
3. Test each group's coherence: build its prototype from half its texts and check how often the other half lands back on it. Below about 0.6, the group is two things wearing one name — split it or drop it.
4. Reject invented groups: if you added any synthetic or templated text, drop groups that are mostly synthetic. Those are labels for things no real person said.
5. Read the texts and rename by hand. Twenty per group. There is no shortcut and this is the step that makes the project good.

Cover the genuinely different ways people say the same thing.

A label's prototype is the average of its texts' vectors. But one label often holds several distinct phrasings — "card not delivered" and "tracking says delivered but nothing arrived" — and a single average of both lands between them, close to neither. So cluster again inside each label and keep one average per sub-group. A label's score is then the best of its prototypes.

from sklearn.cluster import KMeans
X  = E[label_texts]                          # this label's REAL texts only
m  = min(5, len(X) // 4)                     # at least 4 texts behind every vector
km = KMeans(n_clusters=m, random_state=42).fit(X)
prototypes = [normalise(X[km.labels_ == j].mean(0)) for j in range(m)]

Labels, prototypes and decision rules in one file the target reads at startup.

{
  "labels":           ["card_not_arrived", "fee_dispute", …, "none"],
  "proto_label":      [0,0,0,0,0, 1,1,1,1,1, …],   // which label each prototype belongs to
  "prototypes_int8":  "<base64>",                 // the vectors, 1 byte per number
  "prototype_scales": [0.153, …],
  "min_margin":       0.03,                        // tie → say nothing
  "tau":              0.05,                        // only shapes the confidence %
  "encoder":          {"model": "all-MiniLM-L6-v2", "dims": 384,
                       "pooling": "mean", "prefix": ""}
}

scale = abs(P).max(axis=1, keepdims=True)
Q     = round(P / scale * 127).astype(int8)       # target decodes: byte / 127 * scale

• none has zero prototypes — it can never win on similarity. It is only where "say nothing" goes, so that downstream code has one label vocabulary rather than a label plus a null case.
• encoder lets the target refuse an index built for a different model. A mismatched model-and-index pair produces confident nonsense, not an error, so this field is load-bearing.

From a sentence vector to a label, by a sequence you can read off and explain.

1. Known phrase? Exact match in the lookup table from Step 3 → that label, no model involved.
2. Junk? Too short, or no letters → none.
3. Score — dot product against every prototype; keep the best per label.
4. Too close to call? Top two labels within min_margin → none.
5. Confidence — softmax over the per-label scores. For display only; it cannot change the winner.

const raw  = protos.map(row => dot(vec, row));
const best = new Array(labels.length).fill(-1);
protoLabel.forEach((l, i) => { if (raw[i] > best[l]) best[l] = raw[i]; });

const sorted = [...best].sort((a, b) => b - a);
if (sorted[0] - sorted[1] < minMargin) return { label: "none", source: "margin" };

const probs = softmax(best.map(x => x / tau));
const top   = argmax(probs);
return { label: labels[top], confidence: probs[top], margin: sorted[0] - sorted[1] };

One tunable dial, chosen on evidence, trading how often you answer against how often you are right.

Saying nothing is almost always better than a confident wrong answer. Two ways to decide, and they are not equivalent:

Text → the exact same IDs Python produces. Exactly, not approximately.

Look for a real library first. Hugging Face's tokenizers has Rust bindings and a WASM build; there are solid BERT tokenizer ports for most languages. If one exists for your target, use it and load tokenizer.json directly — that is strictly better than what follows. Hand-porting is the fallback, and it is only reasonable because BERT's WordPiece is small:

1. Normalise — strip control characters, unify whitespace, put spaces around CJK characters, decompose accents and drop the combining marks, lowercase.
2. Pre-split — on whitespace, then make every punctuation character its own piece.
3. WordPiece — for each word, repeatedly take the longest vocabulary match; every piece after the first carries ##. No match, or a word over 100 characters → [UNK].

function wordPiece(word: string, vocab: Map<string, number>): string[] {
  const out: string[] = [];
  let start = 0;
  while (start < word.length) {
    let end = word.length, piece = "";
    while (start < end) {
      const cand = (start > 0 ? "##" : "") + word.slice(start, end);
      if (vocab.has(cand)) { piece = cand; break; }
      end--;
    }
    if (!piece) return ["[UNK]"];
    out.push(piece);
    start = end;
  }
  return out;
}
// finally: prefix + [CLS] + ids + [SEP], truncated to max_len

A single call for the rest of the codebase, with the expensive setup done once.

const r = await classify("my card still hasn't arrived");
// { label, confidence, margin, source, top3 }

• Load once — build the model session on first call and share it. Concurrent callers must await the same in-flight promise, not start their own load.
• Do not cache failure — clear the cached promise on error, or one transient bad load breaks every subsequent call for the process lifetime.
• Preload — start loading when the input box opens, not when the user submits, so the result is not a spinner.
• Refuse mismatches — if the index's dims, pooling or model disagree with the loaded encoder, throw at startup. This is the Step 17 field earning its place.
• Never block the user — on any error, return none and let them carry on. A classifier is an enhancement; it should not be able to break the form it is attached to.

A separate, explicit table from label to whatever the product does about it.

The model predicts a label — what the person said. A separate map decides the action — what happens next. Keep them apart: several labels often share one action, some labels have no useful action at all, and the product will change its mind about actions far more often than the model changes its mind about labels.

export const LABEL_ACTION: Record<string, string | null> = {
  card_not_arrived: "card_tracking_flow",
  card_damaged:     "card_tracking_flow",   // several labels → one action
  not_english:      null,                    // understood, nothing to offer
  none:             null,
};
export const actionFor = (l: string) => LABEL_ACTION[l] ?? null;  // unknown → null, never throw

Identical decisions at every level, from your laptop to the real hardware.

Generate a fixture in Python: a list of inputs with the expected label, source, margin and confidence — deliberately covering known phrases, junk, ordinary cases, near-ties, text over the length cap, and non-English. Then check it at three levels, because each proves something the others cannot: