The open-source Python library Deslop surfaced on Hacker News this week, offering a novel method for scoring the effectiveness of LLM prompts using principles from corpus linguistics and psycholinguistics. It provides a quantitative alternative to manual testing for prompt engineers and researchers.
Tool: Deslop | Language: Python | Core Metric: Desirability Score
License: MIT | Dependencies: Spacy, TextStat, Wordfreq
How Deslop Evaluates a Prompt
Deslop analyzes a text prompt and outputs a Desirability Score between 0 and 1. It calculates this score by evaluating four psycholinguistic properties derived from research on language cognition and acquisition.
The library checks for lexical diversity (variety of words), lexical sophistication (use of less common words), syntactic complexity (sentence structure variation), and referential cohesion (how connected ideas are within the text). It uses established libraries like Spacy for parsing and Wordfreq for lexical analysis, combining these metrics into the final composite score.
The Benchmark: What the Scores Mean
The developer’s initial testing on a small corpus of example prompts shows a score range. Simple, repetitive prompts score low (~0.3), while well-structured, varied, and specific prompts score higher (~0.7-0.8).
Early user feedback on the Hacker News thread (13 points, 4 comments) noted interest in the research-backed approach but questioned the correlation between these linguistic scores and actual LLM output quality. The library currently does not execute the prompt against an LLM; it analyzes the prompt text in isolation.
Bottom line: Deslop provides a fast, automated linguistic analysis of prompt structure, but its scores are not a direct measure of LLM performance.
How to Install and Run Deslop
You can install Deslop directly from its GitHub repository using pip. It requires Python 3.7+.
pip install git+https://github.com/etodanik/deslop.git
A basic implementation involves importing the Deslop class and calling the analyze method on a prompt string. The return is a dictionary containing the composite Desirability Score and the individual scores for each of the four metrics.
from deslop import Deslop
analyzer = Deslop()
results = analyzer.analyze("Your prompt text here.")
print(results['desirability_score'])
"Full analysis output"
The analyze method returns a dict with these keys:
desirability_score: The composite score (float, 0-1).lexical_diversity_score: Type-Token Ratio (float).lexical_sophistication_score: Based on word frequency (float).syntactic_complexity_score: Depth of parse trees (float).referential_cohesion_score: Entity overlap measure (float).processed_text: The cleaned text used for analysis.
Pros, Cons, and Practical Tradeoffs
Advantages:
- Speed: Analyzes a prompt locally in milliseconds, unlike LLM-based evaluation which incurs cost and latency.
- Transparency: Provides scores for four specific, research-backed linguistic dimensions.
- Reproducibility: Offers consistent, deterministic results for a given prompt.
Limitations:
- No LLM Feedback: The score reflects prompt form, not the quality of the generated output.
- Early Stage: The weighting of the four metrics and their correlation to practical results is not yet extensively validated.
- Language Support: Primarily designed for English text analysis.
Alternatives for Prompt Evaluation
Deslop occupies a unique niche between manual testing and full LLM-based evaluation platforms. The table below compares its approach to common alternatives.
| Feature | Deslop | Manual A/B Testing | LLM-Based Eval (e.g., G-Eval) |
|---|---|---|---|
| Cost | Free | Free (time cost) | API Cost per evaluation |
| Speed | ~0.1s | Minutes/Hours | ~2-30s + API latency |
| Metric | Linguistic Quality | Output Preference | Task-Specific Score (e.g., faithfulness) |
| Requires LLM Call | No | Yes | Yes |
| Objective | High | Low | Medium |
For teams needing to assess final output, platforms like Humanloop or Langfuse offer full lifecycle evaluation. For rapid, pre-flight checks on prompt structure, Deslop provides a first-pass filter.
Who Should Use Deslop (And Who Should Skip It)
Deslop is useful for:
- Prompt Engineers building large prompt libraries who need an automated first filter to flag overly simplistic or repetitive prompt drafts.
- Researchers in computational linguistics or prompt design studying correlations between prompt structure and model performance.
- Educators teaching prompt engineering concepts, using the scores to illustrate principles of clarity and specificity.
Skip Deslop if:
- You need to evaluate the factual accuracy, safety, or task-specific performance of LLM outputs. Use LLM-as-a-judge or human evaluation instead.
- You work primarily with non-English prompts, as its linguistic models are optimized for English.
- You require a production-grade, battle-tested evaluation system; this is a nascent open-source tool.
The tool’s value is as a linting or pre-screening stage in a larger evaluation pipeline. It can help reduce the number of poorly constructed prompts sent for more expensive LLM-based or human evaluation.
Verdict: A Research-Backed Prompt Linter
Deslop introduces a formal, linguistic methodology to a process often dominated by intuition and trial-and-error. Its core innovation is applying decades of psycholinguistics research to the new discipline of prompt engineering. While its Desirability Score is not a standalone measure of success, it provides a fast, free, and objective check on fundamental prompt composition.
The library’s future utility hinges on further validation studies linking its scores to measurable improvements in LLM output across diverse tasks. For now, it serves as a specialized, automated first opinion on the structural quality of your prompt text.
This approach represents a step toward more rigorous, analytical prompt engineering, moving beyond artisanal crafting toward an engineering discipline grounded in testable metrics.

Top comments (0)