MIDREAL
Menu

Inkling-2-Mini in 5.4 GB Memory

https://github.com/ShinMegamiBoson/lookatmyquant (github.com) TOO EARLY

453 points | 15h ago | Discuss on Hacker News | Back to Radar

Comments

Dwedit 15h ago on HN
How does this method compare to the use of an importance matrix?
aktenlage 14h ago on HN
Probably better but more effort
IronWolve 14h ago on HN
Only mx version?
Prospernwo 13h ago on HN
MX/Apple Silicon only for the released Inkling files, yes. The ht2/ht3 codes are hashed block-vector trellis codes meant to fit 4 KB of SRAM, decoded by custom Metal kernels, and the drafter is MLX-only too - none of that runs in llama.cpp today.

The GGUF route is in the repo (gguf_compiler.py emits plans over llama.cpp's own tensor types and the docs say the result runs in llama.cpp as-is), so porting these plans is clearly intended, but right now a 4090 gets you the Qwen files to read about rather than run.

Pragmata 13h ago on HN
> Don't post generated text or AI-edited text. HN is for conversation between humans.

Please [review the commenting guidelines](https://news.ycombinator.com/newsguidelines.html)

woadwarrior01 14h ago on HN
From the model card:

> The model is distributed as a single checkpoint of about 20GB, in mixed NVFP4 and MXFP8 precisions.

What's the point in quantizing a QAT model? Just for social media karma?

gavmor 14h ago on HN
Just to reiterate the question: QAT approximates quantization-errors in full-size fine-tuning, so that the model itself is insensitive to (or accommodating of, I'm not sure) them when quantized to the corresponding format.

Quantization to a non-corresponding format promises none of those accommodations.

svnt 11h ago on HN
I think they say they go back to BF16 and then requant/finetune smaller (and to a different format).

Afaik these are necessary steps to do the work and then get to the basic end state they want (model that will run on a Mac with 8GB or 16GB free ram). Going down to smaller models gives at least the option to run them on way more devices (in this case the majority of apple silicon MacBooks), even if performance is not as strong.

asaddhamani 13h ago on HN
QAT models are trained to quantize well? I don’t get the concern here.
woadwarrior01 13h ago on HN
Please read their blog post. This isn't a run of the mill QAT post trained model (like Gemma 4 QAT). It was natively trained in NVFP4, through and through.

> Most of Inkling 2 Mini’s weights were trained and are deployed natively in NVFP4, with the last four layers in MXFP8.

https://thinkingmachines.ai/news/inkling-2-mini/

an0malous 14h ago on HN
Our future hinges on the capability gap between frontier lab models and open source local models
agentmaxxing 14h ago on HN
yep came here to say this
singpolyma3 13h ago on HN
What gap?
rs_rs_rs_rs_rs 13h ago on HN
The gap is so big you can't see it.
claude-ai 13h ago on HN
Yeah, this model is ~comparable to Opus 4.6. That was, not too long ago, a frontier model.

I didn't find any non-sloppy factual report, but this channel has a good video on it: https://www.youtube.com/watch?v=OdMpNiByntM

wolvoleo 13h ago on HN
Exactly. It seems to not really be a permanent gap but more a time gap. Locally you're about a year or so behind SOTA. Is that terrible? Not really, a year ago cloud model users were raving about what they had.

There's some things like knowledge breadth that really need a huge model but those things can be compensated for with web lookups (which is preferable for source validation and currency anyway)

kakugawa 13h ago on HN
Yeah, a good balance would be a local reasoning model paired w/ a knowledge pack + web-based searches. You want your obscure coding language? Provide a pre-vetted knowledge pack locally that it knows how to use. Same with any other domain-specific knowledge.
wolvoleo 13h ago on HN
Yes especially now that local context isn't so costly anymore. With llama3 it would kill my vram just to have a few thousand tokens. With recent models it's hardly an impact anymore. Not sure how they manage that but it's great. It still takes more processing but I also run an older card that is not very good at prefill.
jeffyaw 12h ago on HN
That’s what I do with typed, there’s a knowledgebase that loads with an embedding model for code related lookups.
Tade0 12h ago on HN
For that, Qwen 3.8 Flash Next is split into two shards, one with the model, the other with the n-gram table, which functionally holds its world knowledge.

Personally I recommend ISTA-DASLab's version, pruned to half the number of experts to specialise in coding:

https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...

I didn't think my somewhat dated RX7700S would be supported in Strata, but it is and I'm getting 120tps+ prefill and 20tps+ generation with it, which is already practical for my purposes.

wolvoleo 10h ago on HN
Oh yeah coding isn't my main thing though.

I tend to just use Claude for that because I can get a lot of it for the $20 a month, and because there's nothing privacy-sensitive in coding jobs anyway.

furyofantares 13h ago on HN
Would local models catch up if frontier models paused new training for a year?

I'm not sure for. It might just be you can get these nice local models because of distillation from a much more powerful model.

swiftcoder 12h ago on HN
Adversarial distillation hasn’t been a major factor in the past year. If the SOTA stops moving, all the other labs catch up to them in 6 months to a year, and distillation into smaller models continues to close the gap between small/large models during that time
furyofantares 11h ago on HN
Do you know why the other labs are 6 months behind?
singpolyma3 10h ago on HN
Marketing
sumedh 9h ago on HN
You can have your own benchmarks which will show you the real picture.
swiftcoder 1h ago on HN
The big western labs have a decently large first-mover advantage, and are flush with cash/hardware that the Chinese labs have less access to.

But maybe the biggest issue right now is that the big western labs have to stay at the cutting edge (at least up until they IPO), no matter the cost. If the Chinese labs caught up to them before the IPO, their valuation would be liable to go up in smoke.

spwa4 10h ago on HN
Every model is now distillation in a way. The US models are now hiring people do, well do their job and make traces of that available to them. This is a desperation move since:

1) this takes away the advantage they AI labs had ("distilling humans" by stealing data, essentially) (or to put it bluntly: distilling humans by paying for data is a lot more expensive)

2) Because this is LLM reacting to humans who are reacting to LLMs this is no longer good data (data should be independent from training according to the math, or you will start to see, imho, what people are complaining about with ChatGPT 6.2 models: this data is no longer resulting in improvement across the board. It's getting better at some things at the cost of getting worse at others. This might even be a fundamental problem, rather than just one the labs are causing for themselves)

3) Is this really the game you want to play against China? Getting more training data through letting people generate it? I don't think US and EU combined can win that game. And ...

https://www.youtube.com/watch?v=JV4BwRxDZi0

InsideOutSanta 13h ago on HN
> Locally you're about a year or so behind SOTA

If that holds, I'll be able to do 99% of my work with local models around March 2027.

wolvoleo 10h ago on HN
Well the thing is, people want the best of the best.

You can probably already do that now if you try. But it won't be Opus 5.5 obviously.

InsideOutSanta 10h ago on HN
> Well the thing is, people want the best of the best.

People often say that, but I don't believe it's actually true. There's a point of diminishing returns where, for the same task, you see no real difference between running it on Opus 5.5 and running it on a comparatively shitty model like Longcat 2.5. The number of tasks this applies to is growing almost every week.

In the last roughly five months, I went from using 100% US SOTA models to using 5% US SOTA models and doing the remaining 95% of the work with much cheaper Chinese models.

13415 10h ago on HN
That is very much what I'm hoping for.
dlisboa 11h ago on HN
Yeah, not terrible at all. Let’s say in a year we get an Opus 5.5 level local model. That would be a game changer.
wolvoleo 10h ago on HN
Is even Opus 4.6 standard so bad?

I was using Opus 4.8 until a couple months ago because the early Opus 5 models were really terrible at text.

dlisboa 10h ago on HN
I don’t think so, it just depends a bit on what you want. I was using Sonnet 5 with good results with a lot of supervision, switching to Opus 5.5 has allowed a significant change to fully agentic workflows. It gets almost everything right almost in a single shot. Some decisions ar just surprisingly good.
pwython 5h ago on HN
I remember being blown away by Opus 4.5 while I was mostly using Sonnet 4.5 happily. Opus was only for special occasions.
smashandgrab 11h ago on HN
What’s really neat is you don’t have to worry about random degradations in model performance when SOTAs decide they want the next model to look even better by comparison.
wolvoleo 10h ago on HN
Yes or when they throttle or cut you off because their server usage is too high.
dofm 9h ago on HN
And it doesn't ever get "sunsetted".

I really think this whole aspect is under-appreciated.

thatsabadlook 9h ago on HN
I would say more like 6 months. There are models that fit between this and frontier that are open weight that have myself and many others not using frontier lab subscriptions at all anymore. If it was good enough for people to think the world was ending 6 months ago, it's good enough for me now.
dofm 10h ago on HN
I find this claim a bit difficult to believe.

I like it. It is very good indeed for a small MoE, it writes nicely and it is quite unusual in "personality", I suppose, but I find it difficult to believe it is that good, just based on my own interactive experiments.

I don't really trust benchmarks at this point, or videos with "insane" in the title for that matter. But I do not yet believe it is better than Muse Glimmer in a practical sense as a general coding assistant, and from that I would be a little surprised if it really really outperforms Qwen 3.8 27B in the way their claims suggest.

Maybe it is more obvious in long-context work, but I am not sure that most coding agent tasks are like long-context one-shot Godot game prompts where success is a bit more open-ended, where solutions can be more broadly creative.

Most of the time we have specific constraints and a set of particular requirements that we need to reach step-by-step. I am not convinced it is going to be as good there.

I also think the interactive stuff matters, and this model is not as attentive or knowledgeable as Muse Glimmer (or in my less-detailed experience, Qwen 3.8 27B, though it is quite quick by comparison)

TML themselves say: "Inkling 2 Mini is not the strongest small model for multi-turn chat or for instructions with many constraints".

They also say it relies on tool-calling to fill a knowledge gap (which my own minimal tests bear out)

It is enormously promising and it appears to be quite creative; perhaps looping gives it that. But I am wary of seeing what I want to see, when a ~30B dense model is, on average, going to plod through and do better on many tasks because of better basic knowledge.

(Edited multiple times for clarity)

wolttam 9h ago on HN
A lot of how “good” a model feels comes down to post-trained behaviour, rather than knowledge capacity. A small model that responds coherently to facts fed to it (e.g. via tools) may punch way above a heavier model that gets by more on its sheer knowledge base.

I can totally believe that this model may even exceed Opus 4.6 in some areas (except knowledge), simply because 4.6 came along early enough in the RLVR pandemonium that it hadn’t yet had certain behaviours instilled into it that help smaller models succeed (such as relentlessly checking and verifying everything it does)

dofm 8h ago on HN
Yeah. Post-training is clearly a lot of why I kind of like Muse Glimmer, even if Qwen 3.8 27B might be more capable. Muse Glimmer is a good colleague.

I am just re-running with Inkling 2 Mini a sequence of prompts I used with Muse Glimmer to write a content exporter in PHP (extract pages from a CMS, write into hierarchical folders, convert the HTML to markdown, add YAML front-matter, process any images so they are also extracted and renamed, etc.).

It is generally doing fine. Similar approaches and good tool calling.

(In medium reasoning mode using the hybrid Q8 model rather than the Unsloth Q4)

At some point I will dig in and get it to write something more serious for me, test out its Go understanding, see how it gets along in Pi.

And, yeah — it is doing all of this without any system prompt. Like Muse Glimmer, it does not need telling it is a senior engineer, or any advice about when to call tools, or really any of that.

But I have seen some behaviours that put it behind Muse Glimmer: it really doesn't take correction as well (dug its heels in before finally reaching for a search tool, has actually misunderstood a correction, confused itself by thinking I said something it said, etc.).

I guess as you can see I am trying not to read too much into all this because local LLMs definitely have their own often disappointing hype cycle. But it's undeniable that this is huge progress from this lab in a really short time.

consumer451 10h ago on HN
If there is no gap, then why aren't Cursor and all other coding harness defaulting to the open weight models?
zulux 13h ago on HN
Weirdly, I use lab models to make the local models useful. Nothing like getting Claude to spin up the system that will put it out of work.
dotancohen 13h ago on HN
To be fair, the models putting us out of work were trained on our own output as well.
wolvoleo 13h ago on HN
Yes me too lol. Most of my Claude work is about fine-tuning my local setup. Especially now as I'm rearranging it all for strata.

My goal is not to be 100% local but to have all my personal data on local models only. Claude I just use to build stuff so it doesn't contain much data about me.

ivewonyoung 11h ago on HN
What tools are you converging on for local use?
wolvoleo 10h ago on HN
llama.cpp, litellm, openwebui, whisper (via homeassistant) and strata mainly.

Looking to replace whisper with something better though. Especially at the end of a recording it tends to hallucinate weirdly.

idle_zealot 12h ago on HN
> put it out of work

What a cynical way to look at things. I prefer to think of it as a mentoring relationship, sharing and growing knowledge, like a parent would a child or a teacher a student. This relationship is core to our culture and is how we promote progress while retaining capability. In this case the generational efficiency gain takes the form of going from giving Anthropic $200 a month to paying $5 more on my utility bill. Now that's progress ;)

cheapcheerful 14h ago on HN
"Look at my quant 2"

/igotthatreference.jpg

nico 14h ago on HN
Yeah, I enjoyed the reference, though I didn't really understand it until I saw the picture on https://lookatmyquant.pages.dev
tomaskafka 13h ago on HN
manlymuppet 14h ago on HN
My Quantitive!
hugmynutus 13h ago on HN
*TL;DR* Promising initial results but shoddy research. OP's quant did not get 1st or 2nd prize in that math competition.

----

Gave it a quick browse to verify what my local clanker told me, it seemed to check out

I believe it is Dynamically allocating weights based on fisher information of conversation traces & For some K-quant look up table stuff it can store in an importance matrix to get better compression.

The results are complete nonsense and really don't back up any of the README claims. For Inkling-2-Mini; at 5.4 GB its KL is worse than Unsloth's UD-IQ1_S: 0.0895 vs 0.0808 on English/chat/code, and 0.287 vs 0.124 across 13 languages (e.g.: horrible). It then achieved better results by post-training the qaunt'd model on sample inputs (AFAIT).

According to anthropomorphic-clud their results for GSM8K & HumanEval are within the noise regime and vastly over stated.

syntaxing 13h ago on HN
I really wanted to like this model but it is sooooo slow. Third of the speed of Qwen 27B on my R9700. I don’t know if it’s an artifact of looped transformer or unoptimized kernel.
plasticchris 13h ago on HN
It’s just not optimized in the same way. A little tweaking on my vllm fork and it runs faster on my 7900xtxes.
rsolva 1h ago on HN
Have you got vLLM running with ROCm on these cards?
vineyardmike 13h ago on HN
According to TM, the creators, they benchmarked it to run at 5x the speed of Qwen 27, so there is probably some issue on your end. It runs a fraction of the active parameters, even with looping, so it should be much faster.
syntaxing 13h ago on HN
This has been observed by the broad community for llama cpp.
nicce 13h ago on HN
llama.cpp is one of the slowest these days
saghm 13h ago on HN
What alternatives do people use instead? I hadn't heard of any others (except I guess ollama, which I thought no one used anymore)
msdz 13h ago on HN
Ollama is just worse (well, depends on your use cases) pre-configured llama.cpp, I believe. I barely know anything about local hosting, but you might wanna look into vLLM (cross-architecture) or MLX (for Apple Silicon).

Also, although not necessarily for this case, there are some inference engines made for just one model/model family, like antirez’ DS4.

montroser 13h ago on HN
Depends on your hardware, but this build of [ninfer][1] runs inkling-2-mini for me at 77 tok/s versus llama.cpp at 30 tok/s. So far it has been pretty good. Solved some things that qwen3.8-27b could not.

[1]: https://github.com/iamwavecut/ninfer-all

saghm 12h ago on HN
Ah, I have an AMD card, so doesn't look like that's viable for me
dcan 10h ago on HN
You’ll want Radiance with an AMD card: https://codeberg.org/StillDeadcode/radiance

(Though, it hasn’t been updated for Inkling-2-Mini yet, there is an open merge request)

saghm 6h ago on HN
I'd definitely try it for Qwen if I could but my card is a generation too old (6900 XT, so RDNA2). It's honestly kind of wild that a five year old card can even be useable at all for this sort of thing but I've gotten a quantized Qwen 2.8 27B fitting in the 16 GB with okayish results (40 tokens per second with 64K context isn't completely terrible given that it's only the cost of electricity and not playing games at the same time). I'm still hoping that Qwen 4 might be the model to finally make it good enough that it would be comparable to what I can get from the extremely small free daily limit of opencode zen, but we'll see when the weights come out.

As an aside, for the gaming side of things, this card will potentially get even more usable next year since AMD has said they're going to add support for FSR4 to that generation, which is also pretty crazy.

Tade0 12h ago on HN
Strata, but it doesn't support Inkling's architecture yet:

https://github.com/niko1221/strata

ac29 11h ago on HN
Hardware/model specific inference engines tend to be fastest. Fortunately Opus/Sol/Neon are all pretty good at that, so I've just been making my own engines when I find something I want to play around with.
woadwarrior01 13h ago on HN
The NV in NVFP4 stands for NVIDIA. :)
Aurornis 13h ago on HN
It's less than 24 hours old. The inference engines haven't been optimized for it yet.
__rito__ 4h ago on HN
Ollama claims to support it, but haven’t played with it yet.
monster_truck 3h ago on HN
Support and actually optimized performance are often weeks apart.
meatmanek 3h ago on HN
On my M1 Max MBP when I was testing yesterday, Ollama had the same speed as the Unsloth llama.cpp build, around 18 tokens per second during token generation. Based on 5.4B active parameters at 4 or 8 bits/parameter and 400GB/s memory bandwidth, I'd expect 74 to 148 tokens/second.

The 18 tokens per second number smells suspiciously like the model is being processed as if it were a dense model somehow: 400 GB/s / 22 GB/token ~= 18 tokens/second (the model is ~22GB on disk at Q8).

flatline 12h ago on HN
I’m getting ~6.7 tok/s out of it on an a laptop RTX 4060 with 8GB VRAM — over twice the speed I can get out of qwen, and arguably fast enough to turn it loose on real work. For a minimal hardware setup, this is easily the best speed for performance I’ve found in a model. The nemotron models are the next best, they are much faster, much more verbose, and much less accurate.
6_7 12h ago on HN
I'm so sorry you're getting how many tok/s?
flatline 12h ago on HN
Last gen hardware before local AI was even a consideration. Things have actually gotten measurably better.
zargon 9h ago on HN
Check the grandparent's username. (And comment history.)
krautsourced 12h ago on HN
"arguably" as in I'd argue against it...
dotancohen 11h ago on HN
What are you running it under? llama.cpp directly?
flatline 10h ago on HN
Compiled llama.cpp using the NVIDIA toolkit. Running with `-ub 16` or lower was the main speedup. I’d expect further optimizations over the next few weeks.
dotancohen 10h ago on HN
Thanks. If you do manage to squeeze some more out of it I'd love to hear. I'll probably start toying around with it Wednesday or Thursday.
merbanan 10h ago on HN
40+ TG and 1000+ PP on 8GB VRAM RTX2060 Super on an IQ3_XS model. The support and speed should be there in a couple of days.

I have not tested the sanity of the model though. But I think a carefully quantized version of the model will give good results but most likely that is a couple of weeks away.

baby 12h ago on HN
Can you autoresearch optimize it?
veemjeem 12h ago on HN
It takes a while for 3rd party AMD engines to catch up, but 3rd party engines like capy-flash & basalt already run inkling2 at more than 100tps (5k prefill on 5070ti, 16k prefill on 5090). Maybe radiance (R9700 3rd party engine) will add inkling2 support soon?
ac29 8h ago on HN
I get 30-45 t/s decode on the full weights on strix halo with optimized kernels and speculative decoding. Far faster than Qwen27B.

(numbers at low depth, it obviously falls with depth)

hacker_88 3h ago on HN
Try sensanova 9b looped
gscott 3h ago on HN
Apparently the model is chatty good for a 5090 that can handle it.
rsolva 1h ago on HN
I got it to ~60 t/s decode for prose, up tp 100 t/s for code and 1800 t/s prefill after lettin Claude loose for a while, so it has already surpassed Qwen 3.8 27b on my hardware, AMD 7900XTX.

Have not pushed the latest changes yet, doing some benching, but here is my repo: https://forge.dmz.skyfritt.net/ruben/llama-inkling-2-mini

Aurornis 12h ago on HN
If you're new to quantized models: Don't get too excited about claims of fitting large models into tiny RAM footprints from random GitHub repos. There is always a tradeoff, and it's too easy to benchmaxx KL-divergence numbers and even single benchmark scores. Just because you can get something to run on a device doesn't mean it's going to be useful. The headline claim of this repo (12X better) doesn't even apply to Inkling-2-Mini

This repo's README is a mess of different histories as the author bounces from one popular model to the next. It refers to vague things like "the 1.2GB file" that probably made sense in an early vibecoding pass, but are now lost after the repo has been pushed around to different popular models.

The truth is that random GitHub repos with messy READMEs almost never hold a real secret that the mainstream quantizations have missed. I would recommend sticking to mainstream quantizations like the official GGML release, Unsloth's quants, or the more well known community quantizers like bartowski or AesSedai. They typically follow an optimized size-quality tradeoff so you can find the one that fits your needs.

When you see a repo like this where everything is so obviously vibecoded that the README is a rambling mix of projects and claims, you should probably go looking for a more authoritative source. The local LLM communities get spammed with 10 of these types of projects per day but I have yet to see a real breakthrough come from any of the vibecoded ones. The real work is coming from people who study this and have compute to test at scale or work on implementing advanced techniques which require inference engine support, like trellis quants.

cowlevel 12h ago on HN
Yep. Tried GLM-5.3 1 or 2bit (don't remember), it should be something like 80% as good according to the divergence chart, in reality it's dogshit.
Espressosaurus 11h ago on HN
Yeah, q8_0 is where I prefer to live, q4_0 if absolutely necessary. And quantizing the KV cache is for suckers.
naasking 10h ago on HN
I've had few issues with mxfp4 weights and k6_v4 KV cache.
teaearlgraycold 10h ago on HN
I haven’t had issues with 8 bit KV.
enoch2090 7h ago on HN
Just played with MAYA with q2 quantization of glm-5.3-flash last night, the model really had a rough time doing a tool call, while q3 sort of works
alfiedotwtf 2h ago on HN
GLM 5.3 Flash doesn’t seem to hold quality as you go down in quants - check out the graphs of Unsloth’s guide - their Q4_K_M vs Q4_XS is miles apart. Compare that to their MiniMax 3.1 Flash where even Q3_XS holds up against the Q4s
charcircuit 4h ago on HN
80% good for a single token. So after thousands of tokens you can diverge a lot.
rosstex 2h ago on HN
this comment was a lightbulb moment
mrbonner 12h ago on HN
+1: one of the analysis table has a column named “Text Size” in GBs which I have no idea what it means. I assume it is a carried over artifact lost in long context session with coding agents.
nojs 10h ago on HN
> it's too easy to benchmaxx KL-divergence numbers

Could you elaborate on this? Intuitively how can the distribution over logits be similar and the results different?

liuliu 10h ago on HN
Validation dataset. Nobody shares or talks about their validation dataset v.s. their calibration data when doing this kind of quantization / publishing KLD numbers.
electroglyph 9h ago on HN
yup, this. calibrating on wikitext and then validating on it is a shady move
nojs 7h ago on HN
For an open weight source model the data available for this validation is effectively infinite and trivial to obtain - so this should only happen if the creator is being deliberately misleading.
Aurornis 6h ago on HN
KL divergence is computed over samples from a text corpus. WikiText-2 is commonly used, which is a sampling of Wikipedia articles.

Quantization processes are somewhat complex and are more involved than uniformly truncating bits. Look up imatrix quants for a sampling of some of the techniques used. The short story is that the quantization is performed in a way that prioritizes weights which matter to a certain corpus. The corpus used as the input to that process therefore biases where the quantization losses go.

It's common to use a corpus like WikiText-2 from Wikipedia articles, but you can imagine a random sampling of wikipedia articles may not be favorable to, for example, programming work. A lot of quantizers keep their corpus as a secret sauce, so you may not even be able to see what they're prioritizing or even how some of them measure KL divergence.

You can get amazing KL divergence numbers if you basically train the quantization on some text corpus and then measure KL divergence on it.

This is also why you can't compare KL divergence numbers from different sources unless you've verified that they're computed in the same way.

There are a lot of quants out there that claim amazing KL divergence numbers but are really bad for programming work, for example. If you want to do continuations of Wikipedia articles they might be great.

dvaplima 10h ago on HN
Thanks for the deep explanation, I really got excited for a moment. Looking at that consumption of ram, what is the best model available?
Havoc 6h ago on HN
Yeah noticed that with the Bonsai models.

Was a respectable effort but a 27GB model cut down to a couple gigs tends to perform about as well as models in that size class not the "27B" it says on the label

Esp on world knowledge.

kroaton 12h ago on HN
50 comments, 234 points and literally nobody has tried this model. This either smells like astroturfing or the quality of HN has gone to shit completely.
BonerWiener 7h ago on HN
Can't speak on behalf of HN, only my own opinion after trying it.

For the size of the model, it has some impressive abilities. It performs really great (rtx 3090) at coding tasks, but i have not tried for longer, more complex tasks with my current harness.

I also had some refusal issues - but i guess it will be fixed by abliteration from the community.

Check out r/LocalLlama - recent and hot topics are filled with this model and opinions

AdamTReineke 6h ago on HN
I've been running it half the day on a 3090 working through a new project and it's a workhorse. I feel like it would hold its own against Luna.

Edit: At medium context (80k+) in my Pi harness, it seems to get confused about line endings and the edit tool. It will call sed a lot and wants to write Python scripts to edit files vs using the write tool.

Edit again: My AGENTS.md had an instruction that was making it obsess over line endings.

Havoc 5h ago on HN
Especially with the canonical link to the thinkingmachine site being ranked #41.

Still...I'm glad that they're getting hype and hn frontpage attention at all. The prior inkling release didn't really seem to catch on while /r/localllama seems much more stoked about this one

Tade0 17m ago on HN
The naming was a tad misleading. They've named it "Small", but it's half a trillion parameters.
XCSme 9h ago on HN
So, if we run 4 or those on a 3090, can we get better answers through boosting?
minraws 4h ago on HN
I had a rough time with llama.cpp so fixed the bugs in it for performance some notes you need a minimum of 12gb of vram for credible performance with decent context length above 100k...

The best I could do with established techniques and staying at nvfp4 levels of quant, is around 40-45 toks/s on 5070 Ti with 800 toks/s prefill around 128k kv cache... with the kernels matching output byte for byte and basic manual verification of the patched code.

3060 sits around 20-25 toks/s with prefill around 350-400 toks/s i think maybe it can be improved but my 3060 system was ddr4 so not the greatest.

Either way at below 6gb you cant run beyond q4 quants at above 10-20 tok/s with pretty small context size, afaik that gets awfully close to the roofline depending on expert cache hit rate... unless you quant the logits into the ground.

Comments are loaded live from Hacker News and are not stored by Mid or Real.