MIDREAL
Menu

Show HN: DeepSeek V3 in ~200 lines of pure Jax

https://github.com/Vaibhaav-Tiwari/deepseek-v3-jax (github.com) TOO EARLY

276 points | 7d ago | Discuss on Hacker News | Back to Radar

Comments

VaibhaavTiwari (author) 7d ago on HN
I wanted to understand what actually makes DeepSeek V3 different, so I rebuilt its three main ideas from scratch in pure JAX: Multi-head Latent Attention, mixture-of-experts with bias-based load balancing instead of an auxiliary loss, and multi-token prediction. The model is about 190 lines.

It’s a 36.2M-parameter version trained on a MacBook Air M5 (24 GB) with jax-metal, on 207 old detective novels. 12,000 steps took about 2.6 hours.

Full write-up with diagrams and results: https://vaibhaav.in/writing/deepseek-v3-jax/

The trained checkpoint and exact data are in the v1.0 release, so you can generate text without training.

Would love corrections if I got anything wrong :)

wg0 7d ago on HN
This is very illustrative. I'm sure a great learning experience for you but so much more for so many others.

Respect and thanks.

DoctorOetker 6d ago on HN
could you add a disclaimer about which facets are missing (ignoring the parameter count)? the way the title is phrased seems to promise a more compact but an otherwise equivalent code, but in reality there is the risk that the rewrite has unintentional functional regressions in case you oversaw the importance of certain parts of the original codebase.

What would such an overview look like? a fork of DS-V3 with "// reimplemented" markers so any non-reimplemented lines can be identified?

armcat 7d ago on HN
I love this, there is nothing like recreating a model from first principles, plus makes me want to revisit Jax again (I'll do it after I spend enough time with Zig). Thanks for this!
nico 7d ago on HN
> to asking a language model to create a 3D end-to-end working model for me to capture the nuances

Love this! Yes, having a visual of the system makes it a lot easier to reason about and understand

I wish all AI models came with interactive animated workflows of their architecture

> so I wanted it to be different just for the sake of it and used various books by…

That’s cool and great for your own project. At the same time it helps to use datasets that others have used, because then you can compare results more directly

Excellent write up, I really appreciate all the detail and the sample outputs at the end. Excited to see what I could apply for this to the small <10M models I’ve been training

VaibhaavTiwari (author) 7d ago on HN
that's true, I'll try to incorporate standard datasets when the goal would be comparison. the detective dataset was just for fun :)
joebarbere 7d ago on HN
Thanks for this! I've been trying to get local training working on Fedora with ROCm for a while now, and this is the first project that ran end-to-end on my RX 7600 XT 16GB.

The whole reason for me buying the card was to play around with deep learning and models, so it's awesome to have a stepping off point now!

No CUDA meant ROCm surprisingly just worked with JAX.

Gotchas for anyone with a 7600 XT: it needed two env vars, HSA_OVERRIDE_GFX_VERSION=11.0.0 because JAX's plugin rejects gfx1102, and XLA_FLAGS=--xla_gpu_enable_triton_gemm=false to stop a GPU hang.

I'm on Fedora 45 with the in-kernel amdgpu driver, rootless podman and AMD's rocm/jax image (ROCm 10.1, JAX 0.11.1). Nothing ROCm is installed on the host, which is nice.

Container setup is in a PR if it helps anyone: https://github.com/Vaibhaav-Tiwari/deepseek-v3-jax/pull/1

kouteiheika 7d ago on HN
> Gotchas for anyone with a 7600 XT: it needed two env vars, HSA_OVERRIDE_GFX_VERSION=11.0.0 because JAX's plugin rejects gfx1102, and XLA_FLAGS=--xla_gpu_enable_triton_gemm=false to stop a GPU hang.

It's sad that after so many years AMD's software support is still atrocious for non-datacenter hardware, and their short-sighted management still doesn't see it as a priority to fix. This is costing them sales, but I guess they only care about selling to hyperscalers.

genxy 6d ago on HN
Why they don't understand is that this materially impacts them because it blocks the dev funnel. It is absolutely confounding how AMD refuses to realize this for decades.
VaibhaavTiwari (author) 7d ago on HN
Thanks Joe, this is great. Reviewed and tested what I could without an AMD card, and merged the PR!
mattfrommars 7d ago on HN
Post like these always drives a motivation to understand how inference or what OP did.

For folks who are more familiar in this space, is there a good road map how to build and understand the intuition behind this? I’m comfortable in Java and Python and do vanilla SWE work.

rhdunn 7d ago on HN
Intuition: there are YouTube videos from 3blue1brown [1] and Welch Labs [2] on neural networks, LLMs, and related tech that are good.

Building: the videos/tutorials from Andrej Karpathy in Python [3] are considered to be very good at building and understanding the basics, from a small/toy neural network engine to transformers and tokenizers.

There was a post here a few weeks ago [4] on reasoning models using Python/PyTorch with links to an older series on non-reasoning transformers.

[1] https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_6700...

[2] https://www.youtube.com/@WelchLabs/videos

[3] https://karpathy.ai/zero-to-hero.html

[4] https://news.ycombinator.com/item?id=50245138

JBits 7d ago on HN
I'm not an expert but there's https://jaxformer.com, https://github.com/karpathy/nanochat and, of course, the original "Attention is all you need" paper at https://arxiv.org/abs/1706.03762.
VaibhaavTiwari (author) 7d ago on HN
you should definitely start with Stanford CS336, even im re-doing it
lp92 5d ago on HN
I highly recommend starting with Andrej Karpathy's videos on YouTube. Start here: https://youtu.be/7xTGNNLPyMI?is=dRtV-DjTyszYFGrA

Then go through the rest including the one where he builds GPT2.

itsthecourier 7d ago on HN
thank you so so much for writing this, I will recreate it for learning
VaibhaavTiwari (author) 7d ago on HN
thats great :)
try-working 7d ago on HN
Could you build and train a V4.1 the same way?
VaibhaavTiwari (author) 7d ago on HN
Surely that can be tried! of course at a small scale if you're short on GPUs. what I understand is that V4 replaces MLA with a hybrid of two compressed attention types and changes the residual connections, while keeping the mixture-of-experts design, but I haven't gone in-depth architecture :)
vivzkestrel 7d ago on HN
anyone got a c++ version of this?
VaibhaavTiwari (author) 7d ago on HN
I dont think one's available as of now but you can check out llama.cpp, though they run them for real weights rather than small build-from-scratch versions like this!
d_tr 7d ago on HN
I am just starting Raschka's book to write an llm from scratch in C++ with zero dependencies with avx-512 on a 9950x, and then I'd like to go through the literature for more advanced stuff.

I don't know what Jax is exactly but while I am aware that the algorithm implementation is not something terribly complicated, 200 lines still sounds impressive.

If anyone wants to share any tips, they're welcome.

VaibhaavTiwari (author) 7d ago on HN
you can think of JAX as numpy with automatic gradients and a compiler so no backward pass or going into kernels was needed. also, Raschka's book is excellent! you should also check out his model comparison primer that's awesome + you can go on to https://jax-ml.github.io/scaling-book/ next
d_tr 7d ago on HN
Thanks, I will definitely check these out!
steeve 6d ago on HN
fun nugget: if you grab our (zml) PJRT Metal plugin and point JAX to it, it should just work
VaibhaavTiwari (author) 6d ago on HN
Oh nice, didn't know that! Will give it a try, thanks!
madduci 6d ago on HN
Does this work on CPU only hardware (Intel Core Ultra 7)?
VaibhaavTiwari (author) 6d ago on HN
Yes of course! it’s plain JAX, so it runs on any CPU, you just gotta make a few changes cause this one is for Apple Metal
madduci 6d ago on HN
I mean, performance wise
akutlay 5d ago on HN
Don’t expect much

Comments are loaded live from Hacker News and are not stored by Mid or Real.