Instruction editing replaces mask-painting with a sentence: "put the subject in a leather jacket, leave the face and background alone." By the end of this you should know how that class of model differs from inpainting, how to phrase an edit so the untouched parts of the frame actually stay untouched, and which failures to expect before you build a pipeline on it.
What changes when an edit becomes a sentence
Classic Stable Diffusion editing is spatial. You paint a mask, you set a denoise strength, you re-run, you fight the seam. The model has no idea what it is editing. It only knows which pixels it is permitted to touch. Everything semantic lives in your head and in the mask you drew.
Instruction-following editors invert that. The source image and a natural-language instruction go in together, and the model works out which pixels the instruction implies. Black Forest Labs put this on open weights with FLUX.1 Kontext in 2025. Alibaba released Qwen-Image-Edit in August 2025. The editor that circulated anonymously under the nickname "Nano Banana" in mid-2025 was later confirmed as a Google model and folded into the Gemini image line. Different weights, different licences, same interaction contract: one image, one sentence, one image back.
The consequence that matters is referential editing. "Remove the person on the left" needs no mask. "Show the same room at night" needs no mask, no depth map, no separate relighting pass. The model resolves the reference itself.
Choosing between the three editing approaches
These are not really competitors. They are different levels of control.
| Approach | How you specify the edit | Good at | What it costs you |
|---|---|---|---|
| Inpaint / img2img | A mask plus a prompt | Surgical, pixel-exact local fixes | Mask work, seam blending, repeated passes |
| ControlNet / IP-Adapter | A conditioning image | Holding pose, layout or style | Extra models, more setup, per-case tuning |
| Instruction editing | One plain sentence | Referential and whole-scene changes | Fuzzier boundaries, identity drift |
A workable rule: if you can point at the region with a rectangle, inpainting is still faster and more exact. If describing the region takes a sentence, such as "the reflection in the window" or "everything except the subject", instruction editing wins.
Phrasing an edit that holds the rest of the frame
Three habits do most of the work.
Name the subject, then the change
"Change the jacket" is ambiguous when two people are in frame. "Change the seated man's jacket to brown leather" is not. These editors are grounded on the same kind of caption data as the text-to-image models behind them, so vocabulary that works in a caption works in an instruction.
Say explicitly what must not move
Preservation clauses are not filler. "Keep the face, pose and background unchanged" reduces drift across every editor in this family, because the instruction is the only signal the model has about scope. Leave it out and a clothing edit will cheerfully restyle the lighting as well.
One change per pass
Bundling three edits into one sentence gets you a weighted average of three attempts. Run them sequentially, inspect the output after each one, and you keep the ability to reject a single bad step instead of the whole chain.
Chaining edits without cumulative drift
Every pass re-encodes the image. Five passes later the skin texture has flattened, the grain is gone, and the subject looks slightly like a different person. Three things help:
- Order edits from largest to smallest. Scene and lighting first, then wardrobe, then small object removals. Big edits destroy small ones, not the other way round.
- Keep the original file around. When drift becomes visible, composite the untouched region back in by hand rather than asking the model for another correction pass.
- Upscale once, at the end. Upscaling mid-chain locks in artefacts that the next edit will amplify.
In ComfyUI this maps cleanly to a linear graph with the source image held on a separate branch for the final composite.
Start from a base image that is easy to edit
Instruction editors are only as good as what you feed them. A clean, well-lit frame with an obvious subject edits far more reliably than a cluttered one. This prompt was shared publicly by the creator Ludovic for Dreamina 3.1 and reportedly transfers well to Qwen-Image:
A low angle portrait of a confident African woman with braided hair standing in the savannah, with a herd of elephants blurred in the background, golden dust in the air, sunset glow highlighting her profile, shot with a Canon EOS R3, 135mm f/1.8 lens, earthy tones with warm highlights
It produces a single clearly separated subject against a blurred background under directional warm light, which is close to the ideal input for editing. The model can resolve "the woman", "the background" and "the light" as distinct things, and that is exactly what a preservation clause needs something to refer to. Generic prompts that return busy mid-shots give the editor nothing to grab onto.
The camera clauses (Canon EOS R3, 135mm f/1.8) are doing real work, not because the model simulates optics, but because that vocabulary is strongly correlated in caption data with shallow depth of field and portrait framing.
Where this still breaks
- Text inside images. Editing a sign or a label is the most common failure. Some models in this family handle short strings; none are dependable for paragraphs.
- Identity under repeated passes. Faces drift. If identity matters, cap the chain at two or three edits, or re-anchor with a reference-image method.
- Fine repeating structure. Hands, jewellery, guitar strings, teeth. These degrade faster than the surrounding image on every pass.
- Negative instructions. "Do not add a hat" works less consistently than describing the desired state. Say what should be there, not what should not.
- Licence terms. Open weights do not automatically mean permissive commercial use of the output. Read the model card before shipping anything client-facing.
Practical takeaway
Reach for instruction editing when the region you want to change is easier to describe than to mask, and for inpainting when it is easier to mask than to describe. Always include an explicit preservation clause. Run one change per pass, order passes from large to small, and upscale only at the end. And start from a base image with clean subject-background separation; that removes more editing problems than any prompt trick will.
Top comments (0)