Stable Diffusion 3 Architecture and How to Prompt It

Stable Diffusion 3, first announced by Stability AI in February 2024 with weights following later that year, was not an incremental upgrade of SDXL. It replaced the UNet backbone with a transformer, swapped the training objective, and added a third text encoder. That combination is why prompts carried over from SDXL often produce disappointing results, and why typography suddenly became something these models could attempt at all. This explains the mechanics and what they mean when you sit down to write a prompt.

From UNet to transformer

Every Stable Diffusion release up to and including SDXL used a UNet: a convolutional encoder-decoder with cross-attention layers where the text conditioning gets injected. Text influenced the image, but the two representations were never on equal footing.

SD3 uses what the research describes as a multimodal diffusion transformer. Image patches and text tokens both flow through transformer blocks, with separate weights per modality but a shared attention operation, so information moves in both directions. The practical payoff shows up in exactly the places the old architecture struggled: several distinct subjects in one scene, spatial relationships between them, and rendered text.

The accompanying paper is worth reading if you want the details of the scaling behaviour and the ablations.

Rectified flow, briefly

The second change is the training objective. Earlier models were trained to predict the noise added at a given timestep, and sampling followed a curved path from noise back to an image, which is part of why so many steps were needed.

Rectified flow instead trains the model to follow a straight path between the noise distribution and the data distribution. Straighter paths are easier to traverse in fewer, larger jumps. This is also the family of approach that later flow-based image models, including the FLUX.1 releases from Black Forest Labs in August 2024, are built on. If you have wondered why newer models behave differently at low step counts than SDXL did, this is the reason.

Three text encoders

SDXL used two CLIP encoders. SD3 adds a T5 text encoder alongside them, and T5 is a language model rather than an image-text contrastive model. It handles word order, clauses and negation far better than CLIP does.

This is the single change that most affects how you should write prompts. CLIP-era prompting was closer to keyword search: a long comma-separated list of tags, each contributing a little pull. With a T5 encoder in the mix, an actual sentence carries information that a tag list throws away.

T5 is also the expensive part. Most implementations let you drop it to save memory, and doing so cuts VRAM use substantially. The cost is exactly what T5 was buying: text rendering degrades and complex compositional prompts lose precision. Drop it for style exploration, keep it when the prompt has structure that matters.

What improved and what did not

That last row matters more than people expect. A mature library of SDXL LoRAs is a real reason to keep SDXL in the workflow rather than a sign of stubbornness.

Writing prompts for a transformer model

1. Write sentences. Describe the scene the way you would to a person. Keep tag lists for the parts that genuinely are tags, such as medium or film stock.
2. Put text in quotation marks. If you want a word rendered in the image, quote it and keep it short. Long strings still break down.
3. State spatial relationships explicitly. These models act on positional language, which was largely wasted effort under CLIP-only conditioning.
4. Lean less on the negative prompt. With better positive-prompt adherence, describing what you want beats stacking exclusions, and some flow-based samplers handle negatives differently or not at all.
5. Do not port your quality-word block. Strings like masterpiece, best quality, 8k, trending on artstation were artefacts of specific fine-tunes. On a newer base model they mostly add noise.
6. Change one clause at a time. Better prompt adherence means the model responds to small edits, which makes systematic iteration worth doing.

A prompt that transfers well

This one works across SDXL and newer models, which makes it a decent comparison test when you install something new:

Intricate latte art on the surface of a freshly brewed cup of coffee, the design a delicate and detailed Mexican skull floating on a smooth, creamy lake of milk foam, high dynamic range, cross-processed film look

The original used the pipe character to separate clauses and named a specific photo-filter preset. Both work as style shorthand, but on a T5-conditioned model plain commas and a described look are more reliable than symbols and product names the encoder may never have seen.

Run it on SDXL and on an SD3-family model with the same seed. The typical difference is not overall beauty, it is that the newer model keeps the skull as a coherent design inside the foam rather than letting it drift into a separate object floating above the cup.

Running it locally

Budget more memory than an SDXL setup needs, and check the model card for which encoder configuration you are loading. Quantised T5 variants exist and are the usual compromise on consumer cards. Also read the licence before commercial use: terms across the SD3 and SD3.5 releases have been revised more than once, and they are not the same as the older Stable Diffusion licences.

Takeaways

• The transformer backbone and rectified flow objective are what changed; everything else follows from them.
• The T5 encoder is why sentences beat tag soup, and why VRAM requirements went up.
• Quote short strings when you want legible text; do not expect paragraphs.
• Old quality-word blocks and heavy negative prompts are dead weight on these models.
• Keep SDXL around for its LoRA and ControlNet ecosystem, which does not transfer.

Related reading

• Generating European Comic Art with SDXL Checkpoints
• Running Stable Diffusion XL Locally with Automatic1111