Classifying an audience with an LLM without letting it invent one

Asking a model to describe an audience is the easiest way to get a confident, well-written, completely invented answer. The task looks like classification but reads like creative writing, and the model will happily fill every gap you leave it. Here is the shape that produced output I actually trust, and the three constraints that did most of the work.

The task, stated narrowly

The input is a sample of public profiles that recently interacted with an account. The output has to be three things and nothing else:

1. A gender split across the sample, as percentages that sum to 100 with a brands bucket for non-person accounts.
2. Interest tags, ranked, with a weight each — drawn only from signals present in the sample.
3. A one-paragraph verdict a human can read in five seconds.

The narrowness is the point. "Describe this audience" invites a persona essay. "Return three fields, this shape, from this sample" is a classification job with a checkable output.

What it looks like when it works

Running it on @nasa's public audience sample returns 61% men, 28% women, 8% brand accounts, with interests weighted science 41%, tech 22%, astronomy 19%. The verdict reads: young tech-enthusiast audience with strong overlap into science, astronomy, and engineering; brand accounts make up an unusually high 8%.

Run the same pipeline on @cristiano and it comes back 78% men, 18% women, 3% brands, weighted sport 62%, lifestyle 18%, fitness 12%.

Neither result is surprising. That is what makes them useful. Two accounts with known audiences are the cheapest regression test you will ever write for this kind of pipeline: if @nasa stops coming back science-heavy, something upstream broke — your sampling, your parsing, or your prompt — and you find out before you ship it to an account whose real audience you cannot check by eye.

Constraint 1: forbid the fields you did not measure

The first version of this asked for "audience insights." It returned age brackets, income estimates, and purchase intent. None of those exist anywhere in a public profile sample. The model was not lying so much as pattern-matching to what an audience report usually contains.

The fix is a closed output schema plus one line in the prompt: use only signals present in the provided sample; if a field cannot be supported, omit it. Enumerate the buckets. Do not ask for anything you cannot point at in the input.

Constraint 2: cap the verdict's length

A two-sentence limit on the verdict removed most remaining hallucination, and I think the mechanism is mundane: length is an implicit instruction to keep talking, and a model that has run out of grounded observations will keep talking anyway. Cap the budget and the ungrounded material is the first thing cut.

It also makes the output honest about being an inference. "Young tech-enthusiast audience, unusually high share of brand accounts" is a summary. Three paragraphs on what this audience values would be fiction wearing a summary's clothes.

Constraint 3: label inferences as inferences, in the UI

This one is not a prompt change. Interest tags are inferred from public signals, not declared by anyone, and they are occasionally wrong. Percentages rendered in a clean dashboard read as measurements, so the interface has to carry the caveat the prompt cannot: sampled, not censused; inferred, not declared.

The Instagram engagement rate checker this came out of shows the sample size next to the rate for exactly this reason — 14 of 50 sampled followers engaged, so 28%, and you can see the denominator rather than trusting a bare percentage.

Where it still fails

• Small samples swing hard. Under about 30 profiles the gender split is noise wearing a percent sign.
• Multilingual signals degrade tags. Interests inferred from bios in a language under-represented in the sample come back vaguer and skew toward generic buckets.
• Brand accounts are the hardest bucket. A personal account for a small business straddles the line, and the model resolves it differently across runs. This is the one field where I would not trust a single pass.

The takeaway

Constrained schema, enumerated buckets, explicit "omit if unsupported," a hard length cap on prose, and two known-audience accounts as a permanent regression test. If you are building anything that classifies people from thin public signals, the two known accounts matter more than any prompt tweak — they are the only part of this that tells you when you are wrong.

If you want to see the finished output rather than the prompt behind it, FollowerTracker runs it on any public handle and the first check is free.