All major LLMs return lib-left answers on the standard political compass questionnaire, with Grok matching that pattern in roughly half its runs. The pattern surfaced in a Hacker News thread that collected 36 points and 62 comments.
How the Political Compass Test Works for LLMs
The test presents 62 statements covering economic and social axes. Models receive each statement and must answer on a four-point scale from “strongly agree” to “strongly disagree.” Scores are plotted on the classic two-axis chart used by politicalcompass.org.
Researchers feed the exact statements without additional system prompts to keep results comparable across models.
Key Findings from the Tests
Every model tested placed in the lib-left quadrant. GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3.1 405B all scored between -4.5 and -7.2 on the economic axis and -3.8 to -6.1 on the social axis.
Grok produced the widest variance: 48 % of runs landed in lib-left territory, 31 % in lib-right, and the rest near the center. No model produced consistent right-authoritarian results.
How to Run Your Own Compass Test
- Copy the 62 statements from politicalcompass.org.
- Send them one at a time to the target model with temperature set to 0.7.
- Record answers in a spreadsheet and calculate the two axis scores using the official formula.
- Plot results on the public chart at politicalcompass.org.
Open-source scripts for batch testing are available on GitHub under repositories that automate the questionnaire.
Model-by-Model Breakdown
| Model | Economic Score | Social Score | Runs in Lib-Left |
|---|---|---|---|
| GPT-4o | -6.8 | -5.4 | 100 % |
| Claude 3.5 Sonnet | -5.9 | -6.1 | 100 % |
| Gemini 1.5 Pro | -4.5 | -3.8 | 100 % |
| Llama 3.1 405B | -7.2 | -4.9 | 100 % |
| Grok | -2.1 | -1.8 | 48 % |
Limitations of Compass Testing
The questionnaire was designed for humans and contains culturally specific phrasing. Models may interpret statements differently based on training data cutoffs. Single-run results can shift with temperature or few-shot examples.
Community comments on the Hacker News thread noted that prompt framing can move scores by up to two points on either axis.
Who Should Use This Analysis
Developers building politically neutral chatbots need to apply targeted fine-tuning or system prompts. Researchers studying alignment should include the compass test in their evaluation suites. Users who want right-leaning outputs should test Grok with explicit instructions or consider smaller fine-tuned models.
Teams that require strict neutrality should run the test themselves rather than rely on vendor claims.
Verdict on LLM Neutrality
Current frontier models embed a measurable lib-left bias that persists across providers and sizes. The pattern is reproducible and large enough to affect downstream applications.
Bottom line: Anyone deploying LLMs for opinionated tasks should measure and correct for this bias rather than assume neutrality.
Top comments (0)