The first question I got after showing someone my photo-to-Minecraft-blocks converter was which model I was using. Second question, when I said none: was I planning to add one.
It's a fair instinct. The thing takes an image and returns a different image, which is the shape of about a thousand recent projects. But the more I sat with the question, the more I think this particular problem is a useful example of something worth naming: a task that looks generative and is actually a constrained search problem, where reaching for a model would make the output worse rather than better.
The output space is a vocabulary, not a canvas
Here's what the job actually is. You have a photo. You have a fixed list of Minecraft blocks, each with a known average color. For every cell in the output grid you have to name exactly one block from that list.
The critical word is exactly. The output isn't pixels — it's a sequence of identifiers that have to correspond to real blocks that exist in a real game version. There is no such thing as a block that's 40% of the way between white concrete and light gray concrete. The vocabulary is closed and it's small: a few hundred usable entries, not millions of colors.
Which means that even if you did generate the image with a model, you'd still have to project every pixel of its output back onto that closed vocabulary before anyone could build it. The projection step is the actual product. A generative pass in front of it doesn't remove that step — it just adds a lossy transformation before the step that has to happen anyway, and now your output is one degree further from the photo the person gave you.
That's the general lesson I'd pull out of this: when the output space is a small closed set and the mapping from input to output is well-defined, a model is architecture you're paying for and not using. Search does the job, exactly, every time.
Determinism is not a nice-to-have here
The second argument against a model in this pipeline is one I didn't anticipate until I'd used my own tool for a full build.
Converting the photo takes a moment. Building the result takes hours, usually across several sessions, and you don't do it with the tool open in front of you the whole time — you do it with a reference open on one screen and the game on the other, and you come back to it tomorrow.
So the conversion has to be reproducible. If you close the tab, reopen it, drop in the same photo with the same settings, and the tool hands you even slightly different blocks, your half-finished wall is now wrong in a way that's invisible until it isn't. Same input, same palette, same settings, same output — every time, forever. A deterministic nearest-color search gives you that as a property of the algorithm. Sampling-based generation gives you the opposite by design, and "set the temperature to zero" is a mitigation, not a guarantee.
There's a third practical reason, which is that a lookup table costs nothing to run. The whole pipeline lives in a Web Worker in the browser. No inference server, no model weights to download, and — the part I like most — no upload step, because there's nowhere to upload to. You can open DevTools, watch the Network panel while you convert a photo of your own kitchen, and confirm nothing leaves the machine. Verifiable, rather than promised.
Where the ground truth actually moves
The one thing that does change over time isn't the algorithm. It's the palette.
Minecraft blocks get added, and existing textures get retouched between versions, which means the average color of a given block is a version-dependent fact. My palette spans Java releases from 1.13 through 26.2, and Bedrock carries its own separate block and texture set on top of that. Older converters tend to freeze their palette at whatever version they were written against, and the symptom is unmistakable: the preview looks right, you load the build, and a whole region renders in a color the preview never showed you.
This is where the model-versus-table comparison gets concrete. Updating a lookup table when a game version ships is editing a data file. Updating a model trained on a palette snapshot is retraining, re-evaluating, and re-shipping weights — for a change that is, definitionally, just a few numbers moving. Choosing the boring representation is what makes the maintenance affordable for one person doing this on the side.
The evaluation change that actually improved the output
None of the above is where the real quality gains came from. Those came from throwing out my test set.
I had been testing on the kind of images that are easy to find: stock photography, clean product shots, high-contrast portraits with studio lighting. And every converter looks competent on those. Studio lighting gives you separated subjects, saturated colors, and smooth tonal ramps — you could match those with plain RGB distance and most people wouldn't complain.
Then I started running my own camera roll through it, and the failure modes showed up immediately, because real photos are backlit, phone-HDR'd, and noisy in the shadows.
The most visible one was faces. Skin occupies a narrow, perceptually dense band of color where small numerical shifts read as very different to a human eye — we're built to notice that. Naive RGB distance doesn't know that band is special, so it happily alternates between two blocks that are numerically close and perceptually mismatched, and you get a cheek that looks like it was run through a cheap dithering filter. Mid-range greens have a related problem for a different reason. Doing the matching in OKLab — a color space whose geometry comes from perceptual data, so that distance in it approximates perceived difference — is what fixed both. On stock photos that change is subtle. On a backlit photo of an actual person, it's the difference between "someone chose these colors" and "this is a filter."
The second failure mode was one I only noticed because of the material list. The tool tallies exactly what a conversion needs — you'll need 340 of this block, 118 of that one — and once you're reading that list before you build rather than after, the definition of a good conversion changes. A match that's marginally closer in color but spreads across a dozen blocks you'd have to go hunting for is a worse result, in practice, than a slightly looser match built from things you already have in a chest. Nothing about the color math tells you that. You only find it out by looking at what the conversion asks you to go and collect, and by being the person who has to go collect it.
That's the honest summary of the last few months of work on this: almost none of it was algorithmic. It was choosing a representation that stays correct when the game updates, refusing to introduce nondeterminism into something people execute over multiple days, and changing what I was measuring quality against. If you want to see what comes out the other end, the MC Pixel Art tool is free and runs entirely in the browser — bring one of your own photos rather than a clean one, since that's the case the whole thing got tuned for.
Top comments (0)