Image models increasingly reach the public as a paid endpoint months before any weights are downloadable, if weights arrive at all. That changes what you can test and what your results mean. This is a method for evaluating a model you can only reach over HTTP: what a hosted endpoint takes away, how to design the test before you spend credits, and which conclusions actually transfer to the day the weights land.
API-first is a normal release shape
Stable Diffusion 3 is the clearest example. Stability AI opened it to everyone through their API in April 2024, with no downloadable checkpoint; SD3 Medium weights followed in June 2024. Between those two points, every public opinion about the model was formed through an endpoint, on someone else's hardware, with someone else's defaults.
That gap is now routine across vendors. It is not a conspiracy to hide anything — inference costs money, safety tooling is easier to enforce server-side, and a staged release buys time. But it does mean the thing you can touch is not the thing that will eventually run on your GPU.
What a hosted endpoint takes away
| Capability | Local weights | Hosted API |
|---|---|---|
| LoRAs and fine-tunes | Yours to load | Only if the vendor sells it |
| ControlNet, IP-Adapter, other conditioning | Full stack available | Rarely exposed |
| Inpainting with your own masks | Any mask, any denoise | Only via a separate endpoint, if offered |
| Sampler, steps, CFG | Every knob | Usually a preset or nothing |
| Seed reproducibility | Deterministic on a fixed setup | Seed accepted, but the backend can change underneath you |
| Negative prompt | Always available | Sometimes unsupported |
| Marginal cost per image | Electricity | Real money, every attempt |
| Where your images go | Your disk | Governed by the vendor's terms |
The last three rows are the ones that quietly ruin evaluations. Paying per image pushes people into small samples and cherry-picked winners. A backend that updates silently means the run you did last month may not reproduce. And if negative prompts are unsupported, half of your existing prompt library is testing something the model never received.
Design the test before you spend anything
- Write a fixed prompt set first. Twelve to twenty prompts, saved to a file, frozen before you see a single output. Improvising prompts after seeing results is how you end up measuring your own adaptation rather than the model.
- Cover the axes that actually separate models. Multi-subject scenes with spatial relationships ("a red cube to the left of a blue sphere"), text rendered inside the image, hands and limbs, non-photographic media, and one deliberately long compositional prompt. Aesthetic quality is the least informative axis, because it is the easiest thing for a fine-tune to fix later.
- Fix everything you can control. Same aspect ratio, same seed where the API accepts one, same negative prompt across models. Anything you leave floating becomes a confound.
- Save the request as well as the image. Store the full request body, the returned metadata, and the timestamp alongside each file. When the endpoint changes, that archive is the only evidence you had a different model.
- Judge in blind pairs. Two images side by side, source hidden, one question at a time — "which one followed the prompt?" is a different question from "which one is prettier", and you want them answered separately.
Compare like with like
The most common bad conclusion from an API test is that a new base model is worse than a checkpoint someone has been refining for a year. That comparison is not wrong so much as meaningless: you are measuring a raw base model against an ecosystem of fine-tunes, LoRAs, and prompt conventions built on top of an older architecture.
What survives that objection is prompt adherence. Whether the model puts the right number of objects in the right places, spells the word you asked for, and respects a negative instruction is a property of the base model and its text encoders. It is also the property least likely to be fixed by community fine-tuning later. If a base model is strong on adherence and mediocre on aesthetics, that combination usually improves. The reverse rarely does.
Use one control prompt across every model
Keep at least one long, heavily specified prompt that you run against everything, unchanged, forever. It becomes a rough visual ruler: you learn to read a new model's behavior from how it handles a prompt whose output you have seen a dozen times.
A cinematic template with a substitution slot works well for this, because you can vary the subject while keeping the entire style half constant:
Medieval Japanese film concept, {subject}, set in the Sengoku period, exploring themes of honor, loyalty, and the clash between tradition and change, style focusing on authentic historical settings, intricate character development, and epic storytelling. Shallow depth of field, vignette, highly detailed, high budget, bokeh, cinemascope, moody, epic, gorgeous, film grain, grainy
With the matching negative prompt, for any backend that accepts one:
illustration, cartoon, anime, 3d render, painting, crayon, sketch, graphite, impressionist, unreal engine
The negative here is doing something specific: it pushes hard away from every non-photographic medium, which is a good stress test. A model that still returns something illustrative has either ignored the negative prompt or does not support one, and that is worth knowing before you write any more prompts against it.
Practical cautions
- Endpoints drift. Vendors update models behind stable names. Re-run a small subset of your prompt set periodically; if the outputs have moved, your older conclusions are about a model that no longer exists.
- Seeds are a courtesy, not a contract. Unless the vendor documents deterministic seeding, treat repeated identical output as a happy accident.
- Read the terms before you build on it. Commercial use, retention of your prompts, and rate limits vary widely and change more often than the models do.
- Budget the test, not the model. Decide up front what the evaluation is allowed to cost. Per-image pricing is designed to feel small.
Takeaways
An API-only model can be evaluated honestly, but only if you fix the prompt set and the parameters before you start and archive every request you send. Weight your judgment toward prompt adherence and instruction-following, which belong to the base model, and discount aesthetics, which the community fixes later. Keep one control prompt permanently unchanged so each new model can be read against the same ruler. And treat any hosted result as a measurement of an endpoint on a given day, not of a model — the weights, if they ever arrive, will need testing of their own.
Top comments (0)