How Many Labeled Examples Does a Text Classifier Actually Need? I Measured It.

An experiment measures the performance of a classical TF-IDF and logistic regression classifier for routing support tickets as the number of labeled examples increases. The results show that accuracy climbs steeply with the first few examples per category, from 2 to 10, before the returns diminish. More labeled data helps fix errors caused by data scarcity but fails to resolve errors rooted in ambiguity or intent, which bag-of-words models cannot capture. The analysis suggests that the choice between a classical model and an LLM should be based on available data, cost, and latency, not just abstract accuracy.