453 points | 15h ago | Discuss on Hacker News | Back to Radar
The GGUF route is in the repo (gguf_compiler.py emits plans over llama.cpp's own tensor types and the docs say the result runs in llama.cpp as-is), so porting these plans is clearly intended, but right now a 4090 gets you the Qwen files to read about rather than run.
Please [review the commenting guidelines](https://news.ycombinator.com/newsguidelines.html)
> The model is distributed as a single checkpoint of about 20GB, in mixed NVFP4 and MXFP8 precisions.
What's the point in quantizing a QAT model? Just for social media karma?
Quantization to a non-corresponding format promises none of those accommodations.
Afaik these are necessary steps to do the work and then get to the basic end state they want (model that will run on a Mac with 8GB or 16GB free ram). Going down to smaller models gives at least the option to run them on way more devices (in this case the majority of apple silicon MacBooks), even if performance is not as strong.
> Most of Inkling 2 Mini’s weights were trained and are deployed natively in NVFP4, with the last four layers in MXFP8.
I didn't find any non-sloppy factual report, but this channel has a good video on it: https://www.youtube.com/watch?v=OdMpNiByntM
There's some things like knowledge breadth that really need a huge model but those things can be compensated for with web lookups (which is preferable for source validation and currency anyway)
Personally I recommend ISTA-DASLab's version, pruned to half the number of experts to specialise in coding:
https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...
I didn't think my somewhat dated RX7700S would be supported in Strata, but it is and I'm getting 120tps+ prefill and 20tps+ generation with it, which is already practical for my purposes.
I tend to just use Claude for that because I can get a lot of it for the $20 a month, and because there's nothing privacy-sensitive in coding jobs anyway.
I'm not sure for. It might just be you can get these nice local models because of distillation from a much more powerful model.
But maybe the biggest issue right now is that the big western labs have to stay at the cutting edge (at least up until they IPO), no matter the cost. If the Chinese labs caught up to them before the IPO, their valuation would be liable to go up in smoke.
1) this takes away the advantage they AI labs had ("distilling humans" by stealing data, essentially) (or to put it bluntly: distilling humans by paying for data is a lot more expensive)
2) Because this is LLM reacting to humans who are reacting to LLMs this is no longer good data (data should be independent from training according to the math, or you will start to see, imho, what people are complaining about with ChatGPT 6.2 models: this data is no longer resulting in improvement across the board. It's getting better at some things at the cost of getting worse at others. This might even be a fundamental problem, rather than just one the labs are causing for themselves)
3) Is this really the game you want to play against China? Getting more training data through letting people generate it? I don't think US and EU combined can win that game. And ...
If that holds, I'll be able to do 99% of my work with local models around March 2027.
You can probably already do that now if you try. But it won't be Opus 5.5 obviously.
People often say that, but I don't believe it's actually true. There's a point of diminishing returns where, for the same task, you see no real difference between running it on Opus 5.5 and running it on a comparatively shitty model like Longcat 2.5. The number of tasks this applies to is growing almost every week.
In the last roughly five months, I went from using 100% US SOTA models to using 5% US SOTA models and doing the remaining 95% of the work with much cheaper Chinese models.
I was using Opus 4.8 until a couple months ago because the early Opus 5 models were really terrible at text.
I like it. It is very good indeed for a small MoE, it writes nicely and it is quite unusual in "personality", I suppose, but I find it difficult to believe it is that good, just based on my own interactive experiments.
I don't really trust benchmarks at this point, or videos with "insane" in the title for that matter. But I do not yet believe it is better than Muse Glimmer in a practical sense as a general coding assistant, and from that I would be a little surprised if it really really outperforms Qwen 3.8 27B in the way their claims suggest.
Maybe it is more obvious in long-context work, but I am not sure that most coding agent tasks are like long-context one-shot Godot game prompts where success is a bit more open-ended, where solutions can be more broadly creative.
Most of the time we have specific constraints and a set of particular requirements that we need to reach step-by-step. I am not convinced it is going to be as good there.
I also think the interactive stuff matters, and this model is not as attentive or knowledgeable as Muse Glimmer (or in my less-detailed experience, Qwen 3.8 27B, though it is quite quick by comparison)
TML themselves say: "Inkling 2 Mini is not the strongest small model for multi-turn chat or for instructions with many constraints".
They also say it relies on tool-calling to fill a knowledge gap (which my own minimal tests bear out)
It is enormously promising and it appears to be quite creative; perhaps looping gives it that. But I am wary of seeing what I want to see, when a ~30B dense model is, on average, going to plod through and do better on many tasks because of better basic knowledge.
(Edited multiple times for clarity)
I can totally believe that this model may even exceed Opus 4.6 in some areas (except knowledge), simply because 4.6 came along early enough in the RLVR pandemonium that it hadn’t yet had certain behaviours instilled into it that help smaller models succeed (such as relentlessly checking and verifying everything it does)
I am just re-running with Inkling 2 Mini a sequence of prompts I used with Muse Glimmer to write a content exporter in PHP (extract pages from a CMS, write into hierarchical folders, convert the HTML to markdown, add YAML front-matter, process any images so they are also extracted and renamed, etc.).
It is generally doing fine. Similar approaches and good tool calling.
(In medium reasoning mode using the hybrid Q8 model rather than the Unsloth Q4)
At some point I will dig in and get it to write something more serious for me, test out its Go understanding, see how it gets along in Pi.
And, yeah — it is doing all of this without any system prompt. Like Muse Glimmer, it does not need telling it is a senior engineer, or any advice about when to call tools, or really any of that.
But I have seen some behaviours that put it behind Muse Glimmer: it really doesn't take correction as well (dug its heels in before finally reaching for a search tool, has actually misunderstood a correction, confused itself by thinking I said something it said, etc.).
I guess as you can see I am trying not to read too much into all this because local LLMs definitely have their own often disappointing hype cycle. But it's undeniable that this is huge progress from this lab in a really short time.
My goal is not to be 100% local but to have all my personal data on local models only. Claude I just use to build stuff so it doesn't contain much data about me.
Looking to replace whisper with something better though. Especially at the end of a recording it tends to hallucinate weirdly.
What a cynical way to look at things. I prefer to think of it as a mentoring relationship, sharing and growing knowledge, like a parent would a child or a teacher a student. This relationship is core to our culture and is how we promote progress while retaining capability. In this case the generational efficiency gain takes the form of going from giving Anthropic $200 a month to paying $5 more on my utility bill. Now that's progress ;)
/igotthatreference.jpg
----
Gave it a quick browse to verify what my local clanker told me, it seemed to check out
I believe it is Dynamically allocating weights based on fisher information of conversation traces & For some K-quant look up table stuff it can store in an importance matrix to get better compression.
The results are complete nonsense and really don't back up any of the README claims. For Inkling-2-Mini; at 5.4 GB its KL is worse than Unsloth's UD-IQ1_S: 0.0895 vs 0.0808 on English/chat/code, and 0.287 vs 0.124 across 13 languages (e.g.: horrible). It then achieved better results by post-training the qaunt'd model on sample inputs (AFAIT).
According to anthropomorphic-clud their results for GSM8K & HumanEval are within the noise regime and vastly over stated.
Also, although not necessarily for this case, there are some inference engines made for just one model/model family, like antirez’ DS4.
(Though, it hasn’t been updated for Inkling-2-Mini yet, there is an open merge request)
As an aside, for the gaming side of things, this card will potentially get even more usable next year since AMD has said they're going to add support for FSR4 to that generation, which is also pretty crazy.
The 18 tokens per second number smells suspiciously like the model is being processed as if it were a dense model somehow: 400 GB/s / 22 GB/token ~= 18 tokens/second (the model is ~22GB on disk at Q8).
I have not tested the sanity of the model though. But I think a carefully quantized version of the model will give good results but most likely that is a couple of weeks away.
(numbers at low depth, it obviously falls with depth)
Have not pushed the latest changes yet, doing some benching, but here is my repo: https://forge.dmz.skyfritt.net/ruben/llama-inkling-2-mini
This repo's README is a mess of different histories as the author bounces from one popular model to the next. It refers to vague things like "the 1.2GB file" that probably made sense in an early vibecoding pass, but are now lost after the repo has been pushed around to different popular models.
The truth is that random GitHub repos with messy READMEs almost never hold a real secret that the mainstream quantizations have missed. I would recommend sticking to mainstream quantizations like the official GGML release, Unsloth's quants, or the more well known community quantizers like bartowski or AesSedai. They typically follow an optimized size-quality tradeoff so you can find the one that fits your needs.
When you see a repo like this where everything is so obviously vibecoded that the README is a rambling mix of projects and claims, you should probably go looking for a more authoritative source. The local LLM communities get spammed with 10 of these types of projects per day but I have yet to see a real breakthrough come from any of the vibecoded ones. The real work is coming from people who study this and have compute to test at scale or work on implementing advanced techniques which require inference engine support, like trellis quants.
Could you elaborate on this? Intuitively how can the distribution over logits be similar and the results different?
Quantization processes are somewhat complex and are more involved than uniformly truncating bits. Look up imatrix quants for a sampling of some of the techniques used. The short story is that the quantization is performed in a way that prioritizes weights which matter to a certain corpus. The corpus used as the input to that process therefore biases where the quantization losses go.
It's common to use a corpus like WikiText-2 from Wikipedia articles, but you can imagine a random sampling of wikipedia articles may not be favorable to, for example, programming work. A lot of quantizers keep their corpus as a secret sauce, so you may not even be able to see what they're prioritizing or even how some of them measure KL divergence.
You can get amazing KL divergence numbers if you basically train the quantization on some text corpus and then measure KL divergence on it.
This is also why you can't compare KL divergence numbers from different sources unless you've verified that they're computed in the same way.
There are a lot of quants out there that claim amazing KL divergence numbers but are really bad for programming work, for example. If you want to do continuations of Wikipedia articles they might be great.
Was a respectable effort but a 27GB model cut down to a couple gigs tends to perform about as well as models in that size class not the "27B" it says on the label
Esp on world knowledge.
For the size of the model, it has some impressive abilities. It performs really great (rtx 3090) at coding tasks, but i have not tried for longer, more complex tasks with my current harness.
I also had some refusal issues - but i guess it will be fixed by abliteration from the community.
Check out r/LocalLlama - recent and hot topics are filled with this model and opinions
Edit: At medium context (80k+) in my Pi harness, it seems to get confused about line endings and the edit tool. It will call sed a lot and wants to write Python scripts to edit files vs using the write tool.
Edit again: My AGENTS.md had an instruction that was making it obsess over line endings.
Still...I'm glad that they're getting hype and hn frontpage attention at all. The prior inkling release didn't really seem to catch on while /r/localllama seems much more stoked about this one
The best I could do with established techniques and staying at nvfp4 levels of quant, is around 40-45 toks/s on 5070 Ti with 800 toks/s prefill around 128k kv cache... with the kernels matching output byte for byte and basic manual verification of the patched code.
3060 sits around 20-25 toks/s with prefill around 350-400 toks/s i think maybe it can be improved but my 3060 system was ddr4 so not the greatest.
Either way at below 6gb you cant run beyond q4 quants at above 10-20 tok/s with pretty small context size, afaik that gets awfully close to the roofline depending on expert cache hit rate... unless you quant the logits into the ground.
Comments are loaded live from Hacker News and are not stored by Mid or Real.