# Reference-Image Personalization Without Training a LoRA

> Published 2026-09-20 · https://www.promptzone.com/carmen_salas/reference-image-personalization-without-training-a-lora-2b30

Keeping the same character, product, or visual style across a series of generated images used to mean training something. A newer class of open-weight models takes reference images at inference time instead. By the end of this you will know which of the two approaches fits a given job, how to pick reference images that actually work, and what the local setup costs.

## The training path, and what it costs

The established way to lock in a subject is to teach the model about it. Three techniques dominate:

- **[LoRA](/tara_suzuki/best-flux-loras-in-2026-for-realism-and-how-to-stack-them-1mck)** — a small set of low-rank weight deltas trained on your images and loaded alongside the base model. The default choice for characters and styles.
- **DreamBooth-style full fine-tuning** — heavier, higher fidelity, and it produces a whole model rather than an adapter.
- **Textual inversion** — learns a new token rather than new weights. Tiny files, limited capacity.

All three share the same shape of cost: a curated dataset of roughly 15 to 30 images with varied angles and lighting, captions, GPU time, and a training run you will probably repeat before the result stops looking either overcooked or vague. The artefact is also tied to one base model — a LoRA trained for an [SDXL](/jaroslav/how-to-install-and-run-sdxl-models-in-comfyui-a-complete-guide-2nk2) checkpoint will not transfer to a different architecture.

When the payoff is a recurring character you will use for months, that is a reasonable trade. When you need one consistent product shot by tomorrow, it is not.

![Contact sheet of small portrait frames showing the same face from several angles](https://cdn.stocksnap.io/img-thumbs/960w/V5T4HCFQKS.jpg)

## The inference path

Reference conditioning skips training entirely. You pass one or more images alongside the prompt, and the model uses them as a visual condition while generating. Nothing is learned and nothing is saved; the influence exists only for that generation.

The idea is not new — [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter), released in 2023, made image prompting practical on Stable Diffusion pipelines through a separate adapter module. What has changed is that reference conditioning is increasingly built into the base model rather than bolted on, which usually means better fidelity and fewer moving parts. HiDream-O1-Image, which gained native [ComfyUI](/tara_suzuki/best-comfyui-custom-nodes-in-2026-the-ones-actually-worth-installing-75d) support in May 2026, is one open-weight example.

## Which one to reach for

| | Reference images | LoRA training |
| --- | --- | --- |
| Setup time | Seconds | Hours, including retries |
| Input needed | One to three images | 15 to 30 curated, captioned images |
| Identity fidelity | Good, occasionally drifts | Higher, and more stable across poses |
| Style transfer | Strong — this is its best case | Strong, but needs a coherent dataset |
| Novel concepts the base model has never seen | Weak | The only option that works |
| Reuse across projects | Re-supply the images each time | Load one file |
| Portability across models | Depends on the model supporting it | None — tied to the base architecture |

A reasonable default: start with reference images. Move to a LoRA only when you have hit a concrete limitation — the face drifts across a long series, or the concept simply is not in the base model's vocabulary.

## Choosing reference images

The quality of the output tracks the quality of the references much more tightly than people expect.

1. **One subject per image.** Two people in the reference means the model may blend them.
2. **Neutral, uncluttered background.** Busy backgrounds leak into the output as texture and colour.
3. **Even, unremarkable lighting.** A dramatically lit reference exports its lighting along with the subject, which fights whatever lighting you asked for in the prompt.
4. **Sharp and reasonably large.** Upscaled or heavily compressed references transfer their artefacts.
5. **Vary the angle when supplying several.** Three near-identical front-on shots give the model less than one front, one three-quarter, one profile.
6. **For style rather than identity, choose by palette and mark-making.** A style reference works best when its colour and texture are consistent across the frame.

![Mood board of pinned photographs and colour swatches on a studio wall](https://cdn.stocksnap.io/img-thumbs/960w/HDXE91F0EA.jpg)

## Prompting alongside a reference

The reference handles appearance. The prompt should handle everything else — pose, setting, lighting, framing — and should avoid re-describing what the reference already provides. Repeating *brown hair, blue jacket* when the reference shows both wastes tokens and can make the model overweight those attributes.

Where reference conditioning genuinely shines is applying a heavy stylistic treatment to a described scene. This prompt is a good example of the kind of brief that pairs well with a style reference:

```plaintext
A masked vigilante in a black trench coat and fedora stands on a rain-soaked
New York rooftop in the 1930s. Entirely black and white, dramatic shadows,
film noir aesthetic, illuminated signs reflecting in the puddles below.
```

It is short on purpose. The era, the wardrobe, and the weather set the scene; *entirely black and white* and *dramatic shadows* set the treatment; the reflections give the light something to do. Feed a noir still as a style reference and the contrast curve and grain come along without you having to describe them. Feed a character reference instead and the same prompt puts your subject under the hat.

## Local setup notes

If you are running open weights in ComfyUI, two practical details decide how painful the first hour is.

The first is packaging. Older releases ship as several files that must land in different folders — the diffusion model, one or two text encoders, a VAE — and a mismatch between any of them produces confusing errors. Newer releases increasingly ship a single all-in-one checkpoint. HiDream-O1-Image is one of these, and having no separate VAE to place is a direct consequence of its pixel-space architecture.

The second is precision. Most open models are published in several formats — full precision, fp16 or bf16, fp8, and community GGUF quantizations. Pick by VRAM, not by ambition: a quantized model that fits in your card's memory beats a full-precision one that spills into system RAM.

![Empty city street at night in high-contrast black and white with wet reflective pavement](https://cdn.stocksnap.io/img-thumbs/960w/QXTFJRXTKR.jpg)

## Failure modes

- **Reference influence too strong.** The output copies the reference's pose and background instead of your prompt. Lower the conditioning weight if the model exposes one, or use a cleaner reference.
- **Influence too weak.** The subject drifts. Add a second reference from a different angle before you start increasing weights.
- **Style and identity fighting.** Using one image for both rarely works well. Separate them if the model allows multiple reference slots.
- **Consent and likeness.** Reference conditioning makes it trivial to reproduce a real person. Use images you have the right to use.

## Takeaways

- Try reference images first; train a LoRA only when you hit a specific wall.
- Reference quality dominates the result — clean background, even light, sharp source.
- Let the reference carry appearance and the prompt carry scene, pose, and lighting.
- Choose the model format that fits your VRAM, and verify quality at that precision first.


## Related reading

- [Pixel-Space vs Latent Diffusion and the VAE Detail Ceiling](/noor_suzuki/pixel-space-vs-latent-diffusion-and-the-vae-detail-ceiling-2pj5)
- [Prompting Deliberate Imperfection for Realistic AI Photos](/cillian_yoon/prompting-deliberate-imperfection-for-realistic-ai-photos-3129)
- [Prompting Image Models to Build Memes with Real Text](/aleksandr_nakamura/prompting-image-models-to-build-memes-with-real-text-41j0)