# Structural Control for FLUX and Stable Diffusion 3.5

> Published 2026-09-01 · https://www.promptzone.com/riya_ahmadi/structural-control-for-flux-and-stable-diffusion-35-3jo6

Prompting alone gives you no way to say where things go. Re-roll a seed and the horizon moves, the subject turns, the layout you liked is gone. Structural conditioning fixes that: you supply a second input describing geometry, and the model fills it in with whatever the prompt asks for. This article covers what that input is, which variant fits which job, the parameters that decide the result, and the failure modes that make conditioned output look traced.

## The mechanism

The idea comes from ControlNet, introduced in early 2023 in [Adding Conditional Control to Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.05543). A copy of part of the network is trained to accept an extra image — an edge map, a depth map, a pose skeleton — and its output is injected into the main model at every denoising step, with the base weights frozen. The conditioning image never appears in the result; it constrains where the result may put structure.

Two things follow. You steer geometry, not content: a depth map of a person in a doorway becomes a robot in a doorway if the prompt says so. And the control signal competes with the prompt. Too hard and you get a coloured-in copy of the input, too soft and it does nothing. Most of the craft is in that balance.

![Architectural blueprints and drafting instruments spread across a work table](https://cdn.stocksnap.io/img-thumbs/960w/42H3JH8QI5.jpg)

## Two generations of implementation

The original ControlNets were side networks bolted onto Stable Diffusion 1.5 and later [SDXL](/jaroslav/how-to-install-and-run-sdxl-models-in-comfyui-a-complete-guide-2nk2), mostly community-published, one checkpoint per control type. That ecosystem is deep, still useful, and holds the widest variety of exotic control types.

The transformer-based models that followed took a different route. In November 2024 Black Forest Labs shipped FLUX.1 Tools, first-party variants for the FLUX family covering fill, depth, canny and a reference-image mode called Redux; Stability AI released ControlNets for Stable Diffusion 3.5 Large covering blur, canny and depth in the same period. Each control type is its own model release rather than one portable adapter, so switching types can mean loading gigabytes of different weights.

Check the model card licence before commercial work. Openly downloadable dev-tier weights often carry non-commercial terms while hosted API tiers do not.

## Which control map for which job

| Control | What it locks | Reach for it when |
| --- | --- | --- |
| Canny / edge | Hard outlines, every contour it detects | The source is a clean line drawing, logo, product silhouette or architectural shot |
| Depth | Relative distance from camera, volumes | You want the spatial layout but not the surface detail — the most forgiving option |
| Pose skeleton | Joint positions of a figure | A body needs to be in a specific position and nothing else should be constrained |
| Blur / tile | Low-frequency layout and colour distribution | Upscaling or re-rendering an existing image at higher detail |
| Fill / inpaint | Everything outside the mask | Changing one region, or extending the canvas outward |
| Reference image | Subject or style identity, not geometry | You want one look across a series without training anything |

Depth is the default answer for most work. Canny keeps too much: it transfers texture edges you did not intend, so a busy source produces an image welded to the original. Depth discards that and keeps only the arrangement of volumes, which is usually what you wanted to preserve.

![A partly finished clay figure on a sculptor's stand, structure visible before surface detail](https://cdn.stocksnap.io/img-thumbs/960w/66EE6E64A5.jpg)

## The settings that decide the outcome

1. **Match the aspect ratio of the control image to the output.** Nothing else here matters if the map is stretched. Crop before generating, not after.
2. **Start conditioning strength around 0.6 and move in small steps.** Below roughly 0.4 the control is advisory; near 1.0 the model traces. The usable band is narrow and it shifts per control type.
3. **Use the start and end step controls where the interface exposes them.** Applying control only over the early part of the denoising schedule fixes composition and then lets the model resolve texture freely. This is what separates results that look generated from results that look filled in.
4. **Tune the preprocessor, not just the strength.** Canny thresholds decide how many edges enter the map. Halving the detected edges beats lowering strength, because it removes the wrong constraints instead of weakening all of them.
5. **Write the prompt as though there were no control image.** A sparse prompt leaves the model leaning on the control signal to invent content, and that is when output starts looking like a colourised sketch.
6. **Stack a second control only for a specific gap.** Depth plus pose is a reasonable pair. Depth plus canny over-determines the frame.

### A worked case

Take this prompt, which describes an awkward physical arrangement:

```plaintext
candid photography of a Zen shaolin monk, meditating, sitting atop a huge pile of
computers, laptops and cellphones. Golden hour, orange sky.
```

It reads well on a sentence-following model such as FLUX, and unconditioned it produces a competent image — but the pile changes shape every seed and the monk sits at a different height and angle each time. If the composition has to match a layout, generate once, keep the frame you like, extract a depth map from it, and use that map for every variation after. The geometry holds while you change light, clothing, time of day or the entire style.

That is the working pattern in general: generate freely until the composition is right, then convert the winner into a control map and treat it as a fixed set.

![A robed figure sitting cross-legged in silhouette against an orange evening sky](https://images.rawpixel.com/editor_1024/czNmcy1wcml2YXRlL3Jhd3BpeGVsX2ltYWdlcy93ZWJzaXRlX2NvbnRlbnQvbHIvcHgxNTgwNDQzLWltYWdlLWt3eXEwY3d2LmpwZw.jpg)

## Failure modes

**Traced output.** Strength too high, or edge control applied over the full schedule. Lower it and end conditioning before the last third of the steps.

**Ghost structure.** Faint outlines bleeding through as artefacts, usually a resolution mismatch between map and generation. Regenerate the map at the output resolution.

**Control ignored entirely.** Normally a model mismatch: an SDXL ControlNet does nothing on a FLUX checkpoint, and control weights rarely transfer across base model versions.

**Flat depth results.** A depth map from an already flat source, a straight-on product shot or a portrait against a wall, carries almost no information. The control looks broken when it is simply empty.

**Visible seams around an inpaint.** Mask edge too tight against the subject. Grow and feather it; the model needs context to blend against.

## Practical takeaways

- Structural conditioning constrains geometry and leaves content to the prompt. Write them as separate levers.
- Depth is the safest default. Canny is for clean line sources. Reference-image modes carry identity, not layout.
- Conditioning strength around 0.6, applied to the early part of the schedule, covers most cases.
- Adjust the preprocessor before the strength: fewer, better constraints beat weaker ones.
- Control models are tied to a specific base model. Confirm the pairing before debugging anything else.
- Lock a composition once, extract it as a map, reuse it for the whole series.

## Related reading

- [A Practical Lighting Vocabulary for AI Image Prompts](/arlo_mensah/a-practical-lighting-vocabulary-for-ai-image-prompts-544f)
- [What Image Arena Leaderboards Measure, and What They Miss](/santiago_abbott/what-image-arena-leaderboards-measure-and-what-they-miss-19ni)
- [Stable Diffusion 3.5 Variants, Licensing and Prompting](/zuri_wang/stable-diffusion-35-variants-licensing-and-prompting-1e15)