New open-weight image models arrive faster than anyone can properly test them, and the sample grids that accompany them are chosen by people who want you to be impressed. By the end of this you should have a repeatable way to judge one in an afternoon: what its licence permits, whether your hardware can run it, whether the tooling you depend on supports it, and how it behaves on prompts you care about rather than prompts it was demoed with.
The order matters. Most people test image quality first and discover the licence problem after they have built something.
Start with the licence
"Open source" gets applied loosely to image models. In practice releases fall into a few distinct buckets, and the difference is not cosmetic.
| Licence type | What it usually means | Examples |
|---|---|---|
| Permissive (MIT, Apache 2.0) | Commercial use, redistribution and derivatives allowed | HiDream-I1, released under MIT in April 2025; FLUX.1 [schnell] under Apache 2.0 |
| Non-commercial | Weights are downloadable, commercial output is not permitted | FLUX.1 [dev] |
| Community or tiered | Free below a revenue or usage threshold, paid licence above it | Several Stability AI releases |
| Open weights, closed terms | Downloadable but with use restrictions attached | Varies per release |
Read the actual licence file in the repository rather than the announcement post; the two do not always describe the same thing. Check specifically whether the restriction applies to the weights, to the outputs, or to both, and whether adapters you train on top inherit it. If you are building anything commercial, this single check saves more time than any quality comparison.
Can you actually run it
Parameter count is the headline; VRAM is the constraint. A model that fits comfortably on a data-centre card may need work to run on a consumer one. Three things determine whether it is realistic for you:
- Quantisation. Reduced-precision and GGUF builds trade some fidelity for a large drop in memory. If the community has not produced them yet, wait; if they exist, they are usually the practical way to run a large model locally.
- Offloading. Moving components between GPU and system RAM keeps a model runnable at the cost of speed. Fine for experimentation, painful for batches.
- Variant tiers. Larger releases often ship in several versions trading inference steps for speed and fidelity. Test the fast variant first — if its output is close enough for your work, the memory question mostly goes away.
Be suspicious of a benchmark run on hardware you do not have. Generation time on a rented card tells you nothing useful about your own workflow.
Does the ecosystem support it
A model you cannot integrate is a demo. Before committing, check whether:
- There is a working ComfyUI implementation, or a diffusers integration if you work in Python.
- A LoRA trainer supports it. Without trainability, you can never teach it your own concepts, which for most production use is the whole point of running weights locally.
- ControlNet-equivalent conditioning exists. Structural control is rarely available on day one and sometimes never arrives.
- Someone has published quantised builds and there is visible activity in the issue tracker.
A permissively licensed model with no trainer and no conditioning tools is worth less to a working pipeline than a restrictively licensed one with a mature ecosystem. Judge the surrounding tooling, not just the checkpoint.
Test it yourself, with a fixed prompt set
Build a small set of prompts you reuse for every model, keep the seed and resolution fixed, and compare outputs side by side. Six categories cover most of what distinguishes models:
- Compositional relations — several objects with specified positions relative to each other.
- Legible text — a sign, a label, a book cover with specific words on it.
- Anatomy under load — hands holding something, two people interacting.
- Material and lighting — named materials under a named light source.
- Style adherence — the same scene requested in three different styles.
- Restraint — a deliberately sparse prompt, to see what the model invents when you do not tell it.
A multi-attribute fantasy prompt makes a decent single-shot stress test, because it demands a subject, a skin treatment, layered materials, a foreground setting and a background all at once:
Fantasy-themed movie still: a detailed, large, steampunk ogre warrior with blue skin
and tribal tattoos. His light armor is made of leather and copper mechanisms. The
setting is a forest path with a steam-powered city landscape on the horizon.
That prompt was written for a hosted model with strong prompt adherence, which is exactly why it is useful as a comparison: run it unchanged on a new open-weight model and the gaps show up immediately. Count what it got right. The tattoos and the copper mechanisms are usually the first things to disappear.
Why leaderboards will mislead you
Public image-model rankings are useful for spotting that something exists and almost useless for deciding what to use.
Models sometimes appear near the top under unfamiliar names before anyone knows who built them or what the terms are — in April 2025 an entry called Mogao surfaced in the Artificial Analysis rankings with no public attribution. You cannot plan around a model whose licence and availability are unknown.
More fundamentally, these rankings aggregate aesthetic preference on short prompts from voters with no particular use case. That measures which model produces the most immediately pleasing image from a one-line request. It does not measure prompt adherence on long specifications, consistency across a series, editability, or how the model responds to a LoRA. If your work depends on any of those, the ranking is answering a different question from yours.
Use leaderboards as a discovery feed, then run your own prompt set.
Takeaway
Evaluate in this order: licence, hardware fit, ecosystem, then quality. Read the licence file itself and check whether restrictions cover the weights, the outputs or both. Confirm quantised builds and a trainer exist before you invest in a model you intend to customise. Keep a fixed prompt set with fixed seeds and run it on every candidate, and compare against a model you already know rather than in isolation. Treat rankings as a list of things to test, not as a recommendation, and never adopt a model whose provenance and terms are still unknown.
Top comments (0)