Pulpie models surfaced on Hacker News with an 81-point Show HN thread and 19 comments. The release focuses on Pareto-optimal classifiers that remove low-quality or toxic content from web-scale datasets while preserving usable text volume.
What It Is and How It Works
Pulpie trains multiple models across quality, toxicity, and duplication axes. Each model outputs scores that let users select operating points on the Pareto front rather than a single fixed threshold.
The approach trains lightweight classifiers on curated subsets, then evaluates trade-offs between retained tokens and contamination rates. Users apply the models sequentially or in ensemble during Common Crawl processing.
Benchmarks and Key Numbers
Early results show retention rates between 38% and 72% of raw tokens depending on the chosen front point. Toxicity flagging reaches 94% recall at the strictest setting while keeping false-positive rates under 6%.
| Metric | Strict Front | Balanced Front | Lenient Front |
|---|---|---|---|
| Token retention | 38% | 55% | 72% |
| Toxicity recall | 94% | 87% | 71% |
| Duplicate removal | 82% | 74% | 61% |
| Model size (parameters) | 340M | 340M | 340M |
The thread notes these numbers come from a 100M-document subsample of recent Common Crawl.
How to Try It
The models are available via the project repository linked in the HN post. Users download weights, run inference with a provided Python script, and pipe scores into existing filtering pipelines.
Typical command flow:
python pulpie/score.py --input shards/ --output scores/ --front balanced
Integration requires under 50 lines of additional code for most Ray or Spark workflows.
Pros and Cons
-
Pros
- Explicit Pareto curves let teams choose exact quality-volume trade-offs.
- 340M parameter size runs on a single A100 in under 3 hours for 10B tokens.
- Open weights reduce reliance on proprietary filters.
-
Cons
- No built-in multilingual support beyond English and German.
- Requires separate handling of code and math content.
- Evaluation limited to one Common Crawl snapshot.
Alternatives and Comparisons
Pulpie differs from prior filters such as the original C4 quality classifier and the more recent FineWeb-edu pipeline.
| Feature | Pulpie Pareto | C4 Classifier | FineWeb-edu |
|---|---|---|---|
| Multiple fronts | Yes | No | No |
| Toxicity + quality | Combined | Quality only | Quality only |
| Model size | 340M | 1.5B | 1.5B |
| Open weights | Yes | Yes | Partial |
Teams that need only English quality filtering may still prefer the lighter C4 baseline.
Who Should Use This
Research labs building custom pre-training corpora benefit most. Production teams already satisfied with existing toxicity APIs can skip it. Organizations needing strict regulatory compliance should validate Pulpie scores against their own red-team datasets first.
Bottom line: Pulpie gives practitioners controllable trade-offs instead of one-size-fits-all web cleaning.
The release marks a practical step toward reproducible, tunable data pipelines rather than opaque proprietary filters.
Top comments (0)