# Can Classical ML Detect LLM-Generated Text?

> Published 2026-07-16 · https://www.promptzone.com/saoirse_pritchard/can-classical-ml-detect-llm-generated-text-1hid

A blog post by lyc8503 demonstrates that logistic regression and random forests using TF-IDF features reach **95.2% accuracy** on GPT-3.5 and GPT-4 outputs, matching or exceeding many transformer detectors on short-form text.

The discussion on [Hacker News](https://blog.lyc8503.net/en/post/llm-classifier/) received 54 points and 22 comments, with users noting the approach runs inference in under 5 ms on CPU.

## What It Is and How It Works

The method trains on paired human and LLM text using only bag-of-words statistics. No embeddings or fine-tuning are required. Features include character n-grams, word frequencies, and punctuation ratios.

A single scikit-learn pipeline fits in under two minutes on a 100k-sample corpus. Inference uses a pickled model under 50 MB.

## Benchmarks and Numbers

On the author's test set of 10k samples, results are:

| Model                  | Accuracy | F1 Score | Inference Time | Model Size |
|------------------------|----------|----------|----------------|------------|
| TF-IDF + Logistic Reg. | 95.2%    | 0.951    | 3 ms           | 48 MB      |
| TF-IDF + Random Forest | 93.8%    | 0.937    | 4 ms           | 112 MB     |
| RoBERTa-base detector  | 94.7%    | 0.945    | 28 ms          | 480 MB     |

Early HN commenters confirmed similar scores on their own GPT-4 outputs when training data matched the target model.

## How to Try It

Clone the repository and run the training script on any paired dataset. The author provides a ready CSV loader and evaluation notebook.

```bash
pip install scikit-learn pandas
python train_classifier.py --data gpt4_human_pairs.csv
```

The resulting model loads with joblib for immediate use in any Python pipeline.

## Pros and Cons

- **Pros**
  - Runs on CPU with no GPU requirement
  - Model size under 50 MB
  - Training completes in minutes on modest hardware
- **Cons**
  - Performance drops on heavily edited or paraphrased text
  - Requires fresh training data for each new LLM version
  - Less robust to adversarial prompt engineering than fine-tuned transformers

## Alternatives and Comparisons

Popular neural detectors such as GPTZero and Originality.ai rely on perplexity or fine-tuned transformers. The classical approach trades some robustness for speed and simplicity.

| Detector          | Accuracy (short text) | GPU needed | Cost per 1k calls | Open weights |
|-------------------|-----------------------|------------|-------------------|--------------|
| TF-IDF Logistic   | 95.2%                 | No         | Free              | Yes          |
| GPTZero           | 92-94%                | Yes        | Paid tier         | No           |
| OpenAI classifier | 89%                   | Yes        | Free (deprecated) | No           |

## Who Should Use This

Developers building low-latency content filters or academic researchers needing reproducible baselines benefit most. Teams already running transformer pipelines at scale should skip it unless inference cost is the primary constraint.

> **Bottom line:** Classical ML remains competitive for fast, cheap, and transparent LLM-text detection when training data can be refreshed regularly.

The approach shows that simple statistical signals still carry substantial information even as LLMs grow more sophisticated.