A blog post by lyc8503 demonstrates that logistic regression and random forests using TF-IDF features reach 95.2% accuracy on GPT-3.5 and GPT-4 outputs, matching or exceeding many transformer detectors on short-form text.
The discussion on Hacker News received 54 points and 22 comments, with users noting the approach runs inference in under 5 ms on CPU.
What It Is and How It Works
The method trains on paired human and LLM text using only bag-of-words statistics. No embeddings or fine-tuning are required. Features include character n-grams, word frequencies, and punctuation ratios.
A single scikit-learn pipeline fits in under two minutes on a 100k-sample corpus. Inference uses a pickled model under 50 MB.
Benchmarks and Numbers
On the author's test set of 10k samples, results are:
| Model | Accuracy | F1 Score | Inference Time | Model Size |
|---|---|---|---|---|
| TF-IDF + Logistic Reg. | 95.2% | 0.951 | 3 ms | 48 MB |
| TF-IDF + Random Forest | 93.8% | 0.937 | 4 ms | 112 MB |
| RoBERTa-base detector | 94.7% | 0.945 | 28 ms | 480 MB |
Early HN commenters confirmed similar scores on their own GPT-4 outputs when training data matched the target model.
How to Try It
Clone the repository and run the training script on any paired dataset. The author provides a ready CSV loader and evaluation notebook.
pip install scikit-learn pandas
python train_classifier.py --data gpt4_human_pairs.csv
The resulting model loads with joblib for immediate use in any Python pipeline.
Pros and Cons
-
Pros
- Runs on CPU with no GPU requirement
- Model size under 50 MB
- Training completes in minutes on modest hardware
-
Cons
- Performance drops on heavily edited or paraphrased text
- Requires fresh training data for each new LLM version
- Less robust to adversarial prompt engineering than fine-tuned transformers
Alternatives and Comparisons
Popular neural detectors such as GPTZero and Originality.ai rely on perplexity or fine-tuned transformers. The classical approach trades some robustness for speed and simplicity.
| Detector | Accuracy (short text) | GPU needed | Cost per 1k calls | Open weights |
|---|---|---|---|---|
| TF-IDF Logistic | 95.2% | No | Free | Yes |
| GPTZero | 92-94% | Yes | Paid tier | No |
| OpenAI classifier | 89% | Yes | Free (deprecated) | No |
Who Should Use This
Developers building low-latency content filters or academic researchers needing reproducible baselines benefit most. Teams already running transformer pipelines at scale should skip it unless inference cost is the primary constraint.
Bottom line: Classical ML remains competitive for fast, cheap, and transparent LLM-text detection when training data can be refreshed regularly.
The approach shows that simple statistical signals still carry substantial information even as LLMs grow more sophisticated.
Top comments (0)