LeMario applies the Joint-Embedding Predictive Architecture (JEPA) to Super Mario Bros, training a world model that predicts future game states from pixel inputs without labels. The project surfaced in an Hacker News thread with 19 points and 2 comments.
What It Is / How It Works
JEPA learns representations by predicting embeddings of future frames rather than pixels. LeMario feeds Super Mario Bros gameplay sequences into an encoder-predictor setup that matches current and future embeddings. The model avoids pixel-level reconstruction, which reduces compute compared with generative world models.
Training uses standard video game rollouts collected from an emulator. No external rewards or action labels are required during pretraining.
Benchmarks / Specs / Numbers
The project reports training runs on a single consumer GPU. Early results show the model can predict Mario's position and basic obstacle layouts several frames ahead with lower error than a pixel-reconstruction baseline.
No public parameter count or exact FLOPs appear in the release. The HN discussion notes the approach runs with batch sizes feasible on 24 GB cards.
How to Try It
Clone the repository from the project page and install the listed dependencies. Run the data collection script against an NES emulator to gather 10k–50k frames. Launch the JEPA training loop with the provided config file for 50 epochs.
Inference scripts output predicted embeddings that can be decoded into approximate next-frame visuals for inspection.
Pros and Cons
- Lower memory footprint than generative alternatives because it skips pixel decoding.
- Works with unlabeled gameplay footage only.
- Limited to short-horizon prediction in the current release.
- No built-in action conditioning, so planning experiments require extra wrappers.
Alternatives and Comparisons
| Feature | LeMario (JEPA) | DreamerV3 | VideoGPT |
|---|---|---|---|
| Prediction target | Embeddings | Latent states | Pixels |
| Label requirement | None | Rewards optional | None |
| Typical VRAM | 16–24 GB | 24+ GB | 24+ GB |
| Game tested | Super Mario Bros | Atari suite | Custom videos |
LeMario trades pixel fidelity for faster training cycles. DreamerV3 adds reinforcement learning heads that LeMario currently lacks.
Who Should Use This
Researchers testing JEPA variants on low-dimensional control tasks will find the codebase useful. Teams already running Dreamer or MuZero pipelines can skip it unless they specifically need embedding-only prediction.
Bottom Line / Verdict
LeMario provides a compact, runnable example of JEPA applied to a classic game, useful for anyone exploring non-generative world models on modest hardware.
Top comments (0)