848 points | 6d ago | Discuss on Hacker News | Back to Radar
Looking forward to the "Flash" and 27B versions.
Hopefully the Qwen 4 Flash model will still fit within 96GB without requiring excessive quantisation as, verbose thinking aside, 3.8 is a very solid workhorse model.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
For many of our use cases it's currently far into the "good enough" range because external factors (such as process design) are now blockers.
Maybe I'm mixing up model variants, but Qwen 3.8 27B that I run locally totally does handle images. Surely the bigger one would as well?
Instructing it to ask any and all questions as necessary sends it into an existential crisis, because the question tool (oh-my-pi harness) suggested limiting the number of questions, and it couldn't figure out how to do both.
I believe that's all reasoning levels do, but most inference engines should have thinking budget.
The issue you’re describing could be a small model problem
There was also a bug in which the reasoning tokens were “encrypted” for opus but you could ask Haiku to regurgitate them and it would.
Anthropic have been doing anything and everything to prevent distillation “usage” and it’s caused me at least once to cancel my subscription because it’s hard to trust a company that secretly alters its responses based on something I can’t see and never get feedback on.
Asking it to think/reason too strongly would hit classifiers. Quoting reasoning summaries in my messages would hit classifiers. It thinking about how post-training effects another model's would hit classifiers.
I thought it might be overzealous Fable 5.2 protection, but at one point I went through a chain of Fable -> Opus -> Sonnet. Eventually even Sonnet triggered a classifier no matter how the conversation went... I had to rewind and have Claude prompt Codex going forward for most of the work.
Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Models often get great scores despite being a lot worse than the frontier in practice.
Edit: I do agree their harness methodology is pretty bad, but I think that's more directly attributable to the fact that harnesses appeared after their initial methodological design, and they are too lazy or resource constrained to improve their methods. Still dishonest, but a simpler motive.
AA has always been meaningless, but people keep referring to it anyway just because the synthetic view it provides is too convenient (even if meaningless).
they keep harnesses the same across models other than in the coding agent index (https://artificialanalysis.ai/agents/coding-agents), which uses native harnesses for all models too
Ah yes, the reverse-benchmaxers of OpenAI models.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number.
Maybe somebody should archive the scores and see if they in the future ever backtrack particular models to favour others (like last week)
I wish there were more ~60b dense options like ripple too!
Do you run Flash Next or 27b on yours?
I'm actually running Gemma4 for the most part - for my specific needs it was the best out of the several I evaled
By comparison opus 5.5 (max) is $10.01.
If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
They compare different reasoning levels.
https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...
Scroll down to the cost per task pareto front chart which includes more difficult levels by default: https://artificialanalysis.ai/models/qwen4-max#intelligence-...
Qwen4-Max is not in a good spot on that chart. Astra xhigh and Fable 5.2 medium had a similar cost per task with higher intelligence ratings.
If it's actually there, what is the cost at medium thinking? It would show Qwen 4 Max (medium) if it's there.
Everywhere that I can see on the page it just shows ”Qwen 4 Max" while for 6.1 Sol, Opus 5.5 there's always a thinking level in brackets after: (low), (medium), (max) etc.
It's possible that they will run Qwen 4 Max at other reasoning levels in the future, or it's possible that they never will. Regardless, it's unlikely to be better at any reasoning level on a price/performance basis than the GPT 6.1 Sol/GPT 6 Astra pareto frontier. It's close enough to Claude Opus 5.5 line at lower reasoning levels that it could potentially cross that if it doesn't loose too much performance at lower reasoning levels. But it's far enough from the global pareto frontier that it's unlikely that it would cross that at any resoning level.
You can see for example that the cost per task for 6.1 sol is about 4x higher at max compared to medium thinking level.
Which effort levels are cost-equivalent is debatable and depends on whether you are paying API prices or on a subsidised plan or are able to run the models yourself on your own hardware. But I think AA uses public API pricing for everyone as a sensible baseline.
I'm not saying this model would land on the frontier - from the single data point it looks unlikely. However, that single data point might be making it look much further from the frontier than it should be.
EDIT: I will also take back my point that we should be comparing models at the same thinking level - max on one model won't correspond to max on another. What we need is a range for each model. What we have is a range for the most popular models, showing that the same model with different thinking levels can have vastly different results. Then for less popular models we have just one data point, which isn't enough to draw the kind of conclusions people are drawing in this thread.
It does not (universally) hold for long running workloads -- which is what people are increasingly doing.
Leave a Sonnet on medium, it will complete the task, spend more time and possibly significantly more token money, vs. Opus 5.5 on xhigh.
I always use Max effort in both Opus and Fable. It's actually faster overall because tasks get done faster with fewer "corrections" from my end.
On GPT models, I'd sometimes dabble in xhigh/high but that's because OpenAI usage quota are too limited.
Do you have proof of that or is it just vibes?
This is wrong.
To make a vacuum analogy, one brand's Low setting could be another's Medium setting.
To compare you either give them the same resources ( Watts or $ ) and see how much they can do, or you give them a task and then check how many resources they required to finish it.
The label on the knob is irrelevant, except that either model might have a setting higher than the other which is sometimes useful if the resources aren't a constraint or lower when they are.
It's very clear part of America's diesease is the idea that everything can equate to a dollar cost.
Which is why you need to test a range of settings.
Suppose you were testing car mileage. For the big name popular cars, you do 10km drive at 20km/hr, 40km/hr, 60km/hr and then publish the results.
For all the less popular cars you only do one test at 60km/hr and then you publish all the results for all cars on one big "Pareto frontier" graph claiming to show which car will allow you to drive home most efficiently.
It happens that lots of the less popular cars are not very efficient at 60km/hr so naturally the graph makes them look bad. Would they also be bad if tested at 20km/hr or 40km/hr?
Nobody knows. Because this is where the analogy collapses - unlike cars, hardly anyone is doing these kind of tests for LLMs.
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
If we have data only for “Chinese max” (let’s say it costs $1) and “non-Chinese low-to-max” (spanning $0.10 to $3), comparing them in full at least tells us what we can expect from each model if we have $1 to spend + the maximum performance we can expect with unlimited budget. What’s the point of restricting to “max vs max”, which answers only the second question?
Another analogy for this are review sites overly focusing on reviewing all the SKUs of GPUs that cost $1k+ for an audience that, according to Steam's Hardware Survey, predominantly use midrange X060s-equivalent cards. Sure, it's fun to know whether or not you can enable PT on a 5090 but for most of us, covering 4070 TIs and below in-depth (meaning actually taking the card apart, seeing how good/bad the past jobs are, possibly buying multiples of the same SKUs to evaluate QC, etc) is much more useful because of the value prop.
Sure, that’s not in question. The discussion here is not about what to measure (we agree that benchmarking only max is suboptimal) but how to compare models if you have an already fixed set of measurements, outside of your control, which includes only “max” for model A but a wider range for model B.
The person I was responding to was claiming (in their previous comment) that the sensible thing to do, then, was to artificially truncate the data for model B to “only max” as well.
Nah, there are interesting conclusions to be had in comparing various model/effort level combinations, esp. when it comes to cost and time per task.
I have my own private benchmark suite; it involves algorithmic problems, the need for good visual understanding, and the ability to run long workloads unattended (as the agents need to run a few iterations, self-evaluate the results, and hand over the assignments when they feel they have succeeded). Chinese models do poorly on such problems, and running them at anything less than max efforts is pointless; they can't even hand over the assignments graded against a mechanical set of tests they themselves can execute. Hand that same task over to, say, Sonnet medium and you have a weird outcome: excellent result, but way more time and tokens burned than, say, Opus 5.5 or Sol 6.1 running at high. I do those benchmarks just to be able to spot such patterns and then better choose the models for my real work - but i wouldn't have caught it if I wasnt running various model/effort combos.
I'd love a comparison on for example monthly quota usage per task, between all ~$20/month subscriptions.
It's so difficult to compare how much "productive" usage you get out of each subscription, without signing up to each one at least.
It's like arguing windows 11 just needs more bloatware.
These models don't have to all in one every task in the world.
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
They have altered their benchmark multiple times in the last 4 weeks (v5 in https://artificialanalysis.ai/articles/artificial-analysis-i...) and the last 5.0.1 change https://artificialanalysis.ai/methodology/intelligence-bench... pushed up Anthropic's models a bit, you can see that in v5.0.0 at the first link, Opus 5.5 has a lower score than in it gets in v5.0.1, and v5.0.1 was released just before Fable dropped.
Some part of me believes that the benchmark mix was put together to favor Anthropic's models.
But admitting their model is weaker is a no go. So conspiracy theory here we go
It's mainly just a composite of existing benchmarks. It would be very easy to benchmax if, say, AA shared with a preferred lab which benchmarks it plans to upweight/downweight in the next version. Just speaking hypothetically, of course.
Right now the big labs are buying benchmark tasks from "benchmark labs" (or whatever they call themselves) for absurd amounts of money, and making sure they perform well on them.
The incentives this generates are terrible.
The big AI labs will only buy benchmark tasks that the frontier models perform poorly on (because they have trouble finding any public ones). So the tasks are being mass-written now directly based on the models they're supposed to train. Obviously the math says you shouldn't do this. If there is a causal connection between the benchmark and the model, you lose the normal guarantee that improving benchmark scores will improve model.
In fact improving such benchmarks may make the model worse. To understand that let's just give a ridiculous example. I create tasks like "1 3 5 11 13 17, give next number". The next number is 301. These are the house numbers on some random street in Brussels, Belgium (I'm guessing the big jump is because they "merged" 2 streets, and just added 300 to the numbers of the second part).
If the model trains on this, performance will actually decrease because obviously this is not how you should reason.
BUT making tasks under this "impossible to reason towards" part is of course how you make tasks that might even be reasonable, and of course frontier models are going to suck on tasks like this. So they sell!
BUT they suck! Finding houses is actually going to work a lot better if you ignore this fact until it becomes unavoidable in one out of 1000+ streets.
And yet this is what benchmark improvement has moved to. If you look at the terminal-bench 4.0 tasks you will see what I mean. Doing better on those tasks ... does not actually make a better model BUT you have to actually go read those tasks to realize that. A lot of the new tasks largely demand you make weird jumps in your reasoning not based on understanding the underlying theory. They rely just being able to predict what goes wrong in certain cases when theory would suggest it should work, and mostly it even does work but for some reason not acceptably. They reward not doing the full work, realizing what goes wrong and correcting, without understanding why the normal reasoning won't work and correcting without knowing why. I find it very hard to believe this makes a model more intelligent. It rewards knowing the encyclopedia by hard, so to speak.
But it certainly makes it look better. Whereas the obvious, coding improvements, do not make models look better anymore.
Rarely seems to make them look good whenever I drop in to the site. Far more expensive for little gain. Even within their own effort levels it looks bad, Fable xhigh is 50% more in cost than high for zero improvement across a range of tasks.
A tiny bump in intelligence score hardly seems worth it for cost, and honestly most people's tasks are not cutting edge work.
Current benches on there seem to make Sol look the best, at a certain point the subscription subsidy tap will turn off and people will be in for a rough time paying full price.
That might change later, and then the benchmark can be adjusted, but currently this is the target
Also, a generic benchmark might not translate well into what everyone needs. I keep building my own test cases that test models for my own work specifics. Only that benchmark is useful to me. I don't need it to win a PhD math quiz.
Neither option is perfect.
A third option would be updating the benchmark every 6-12 months, but that has its challenges as well as you need to make sure that the benchmark covers things the model hasn't seen or is harder for the models to game.
Interestingly, Artificial Analysis does have a score that uses IRT, it's their Progress Index [2].
[1]: https://en.wikipedia.org/wiki/Item_response_theory
[2]: https://artificialanalysis.ai/methodology/progress-index
(not to say there is no value in "zero-trust" environments, but often pure-benchmaxxing is a deadend; some benchmaxxing is always involved and much less risky options than "train on the ZDR test env" usually exist)
You could run the benchmarks using downstream providers like AWS if they are available.
You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.
But ultimately it's another arms race like with viruses and anti-virus software.
Or is it just the usual, better conditions credits-based pricing for entities beyond 200?
Cause if it's the latter, that's very different to a multi-year lock-in with a single lab, easy to switch away from and kinda why one could have both Anthropic and OpenAI enterprise agreements alongside each other, basically what I described with slightly better conditions. Bedrock was just an example, they also have more favourable conditions on inference for some (though Anthropic really limits discounts), but whether you go usage based from the labs or a cloud provider, is the same lack of long term lock-in. A e.g. $ 20,- seat per employee for LLM access that still needs tokens allowance on top does not create a long term commitment, financially or otherwise, the friction here is none-existent.
If however someone at this stage is fully locked into a single lab exclusively for the next two years or longer (multi-year), they have gambled, pure and simple. Could have picked a winner or a lab that is going to be uncompetitive starting this Thursday and for all eternity after. Imagine having picked your companies one and only lab or even worse model in 2024...
Otherwise you have to rely completely on trust that AA is really measuring intelligence.
This is a permanent problem in benchmarking really. Transparency is required for trust but invites over-tuning.
I have been using Terra a lot and I'd say that most of the models (even small ones) are are roughly equivalent when it comes to working in a consistent, established codebase.
But in practice, Terra is not very good at staying on task compared to say Sol 6.1 or Sonnet 5.5. Terra 6.1 will often just pause, thinking the objective has been met or the turn has ended.
I kind of agree that closed benchmarks are bad. With a closed benchmark you still send the questions to the model provider and they may well train on it. Might as well publish it.
For most things I'd use Kimi is probably better so the ranking is wrong for me.
However it also seems like in those benchmarks where Kimi underperformed it did do quite badly, whereas when Astra underperformed it did so less badly. How you reflect that into a single number I'm not sure - it's certainly difficult to get right and it doesn't seem obviously biased to me.
Certain labs have been shown to and will continue to train on public evals, whether unintentional or on purpose, any respectable ranking across multiple evals also needs private data to counteract this.
Do you know a better site?
You can easily come up with a difficult private benchmark and test them yourself to validate the intelligence gap. It is real.
China lost. There is no catching up. Ever. That ship sailed.
What’s the evidence for this? Are there plausible alternate explanations?
At the very minimum, AA changes what they define as "intelligence" rather frequently and it always benefits OpenAI and Anthropic. To me, it feels like they use Astra/Opus as their baseline and tweak weights of benchmarks until the intelligence index matches their preconceived opinions on model intelligence (as opposed to crafting the index in a vacuum without any context on what different models are good at).
If nothing else, it points to these aggregate benchmarks having no idea how to measure actual "intelligence" and instead tweaking weights of benchmarks until it matches what they feel it should be. In practice, I think a better system would be to have many benchmarks within a particular space (coding, agentic work, science, etc.) and scoring each category separately and without weights applied to benchmarks (and simply removing anything once it becomes saturated). Then, each user can decide which benchmark suite matches their usage and use that to make an informed decision instead of having to trust someone else's opinion of what "intelligence" is.
With that system, you can still have benchmaxxing, but I'd imagine it would mirror actual usage better and so be less of a departure from reality.
The index you're referring to has 30% of its score from "Engineering knowledge". Functionally, a third of that index is just trivia which will reward large models. A small model with web search for knowledge does just fine in practice, but will be penalized heavily in this index.
Then you have science-specific as another 20%. I don't do that stuff, so now half of that index is useless to me.
Reasoning and agentic is the remaining 50% and is all I functionally care about. That index is the closest to my usage, and yet half of its weighting is irrelevant to me. The other option is the "Coding agent" index, but that couples models to agents and they don't test my preferred agent.
I'd much prefer a benchmark site without these arbitrary weights and that let's you pick and choose which benchmarks to include in the summary. Heck, it could even allow you to set your own weights to each benchmark/category.
Specifically, they've made knowledge recall the largest part of their indices. That rewards larger models, but has disproportionately small real-world gains as a small model in a harness with web search will perform just as well.
Take DeepSeek Flash 1017. It's half the score of Fable on the Engineering index. Fable is not twice as good at any engineering work I do.
As the frequency of model releases increases, we will also be moving faster [1] to ensure our Index stays up to date to reflect both (1) how people are currently using models and (2) what are the strongest signal benchmarks which show these capabilities.
[1] Index release history: https://artificialanalysis.ai/methodology/intelligence-bench...
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
I would assume it will not be on Bedrock for some time, AWS has not kept up with the latest non-American models.
This forces ant/oai to lower their prices but how much that is subsidized nobody knows. It might very well just be they are in deep trouble, they see their clients leaving, they have to lower their prices.
But let's not forget openai is riding on the edge of bankrupcy, their last investment round apparently gave them few months of runway, they are leaking money. I don't think they know how to turn this around with all the open models being competitive.
I would just enjoy cheaper model costs while they are like this.
I feel open-weight was able to keep up the past year using efficiency improvements (hybrid attention, improved MTP drafters, improved optimizers) to offset having less compute. But OpenAI/Anthropic started cutting token costs just a few months after these, which to me suggests they copied these improvements. Indeed it would be hard to work in LLM training and not know about DSpark, Muon etc.
So what do you do if closed source uses your learnings, gives no knowledge back, and has 10x more compute and funding on top of it? The situation feels quite dim right now, esp. with the Nvidia thing.
Chinese AI also has a clear business mode: making money from its (i.e., chinese AI) applications.
I'll argue that the latter is closer to the end-goal here than the former.
These days the way China subsidizes these companies is more from state-backed buildouts of data centers, renewable energy, and electrical infrastructure. It's a 'make the field as green as possible for the cows to come' kind of strategy. It's very different from America's hyper-individualism which feels very reactive with specific greasing-of-the-wheels + DARPA/CIA VC + DoD spending while relying on companies to be the ones to push for better infra and capacity.
If we extrapolate over the next several years, prices for VRAM/RAM will start to come down as manufacturing capacity comes online, models will continue to become more token efficient, and intelligence will improve at every quant size. As this happens, more and more people will be running LLMs locally on hardware they already own, and there's going to be a point where local models can do everything you need to do unless you're literally trying to do breakthrough mathematics research.
> and there's going to be a point where local models can do everything you need to do unless you're literally trying to do breakthrough mathematics research
Maybe. But competitive markets pay exponentially more for incrementally superior services. Professional athletes are a great example of this. If you're building a product with competitors and those competitors have frontier models and you stick to local then you're probably going to lose both today and in the near-ish future.
If you're bitter lesson pilled then the alpha is entirely in having a better model, more data, and more compute. Your vision is that human ingenuity will provide more alpha. Oof. I wish. But my tastebuds are very very bitter.
Just this week, qwen 3.8-27b got two new interesting quantizations, one of which came from community. Llama.cpp also improves continuously, and novel alternatives like Strata pop up.
Also improvements to llama.cpp stay, and many small improvements stack up nicely.
i highly doubt this premise. the success of a product has very little to do with the amount of the intelligence used in producing it. you could throw millions of dollars using AI to develop a movie and it would still flop in comparison to an indie-made film.
I want to use "more intelligence" but not in terms of single higher IQ.
I want to use more intelligence in terms of a sum quantity of knowledge work being performed. A team of 10 instead of a team of 5.
This is when a local model will shine. A fixed cost and I don't have to worry about how much I am using it. I already can't tell any difference at all between Opus and Fable. It won't be that long that I won't be able to tell the difference between a local model and Opus.
There will be an infinite demand for "more" intelligence but I think many people are making a category error on the dimension in question.
the lesson is that training a model on all input ever output will never be as good as training a model on arbitrary outputtable input.
Once I have a model locally, I know it isn't going to change on me. Compare that to using an API from Anthropic or OpenAI (and yes, the only thing you can say is you're using an API, not one model), they can discontinue a model you're relying on (as every model could have peculiarities you've designed around) or modulate their speed/quality day-to-day to save on compute costs.
And trying to claim that buying your own hardware doesn't pay off compared to API or subscription costs doesn't hold much water either, at least not for me. For starters, I already had the GPUs and machines I started running models with, whether it was for gaming or for other reasons. And in the case of the new GPUs I bought recently (2x AMD Radeon Pro R9700; no you do not need CUDA and you haven't for a long time now), I can hold a reasonable expectation that they will hold at least some of their value years from now, compared to paying for API/subscription usage for years in which case you're left with nothing.
So, if you have a local model that is "good enough" (and they already exist for many use cases), it's just kind of the obvious choice to make, IMO. And in just one person doing it as a hobby while I'm still looking for a job 16+ months after a layoff. A small business and certainly a large enterprise has even less reason to open themselves up to all the downsides that come with using Anthropic or OpenAI.
Open source doesn’t have those motivations. Once something works well there’s no reason to degrade it.
why more like hardware? takes very long time to develop, expensive to manufacture (run), valuable.
.. if exponential, they’ll never catch up but if logarithmic, closed models will plateau and so catching up is inevitable.
My money is on a plateau
I hope it will plateau
Following that logic the gap widens a little bit here but then will steady again. Before this there has been a general catch-up trend too which probably also persists leaving us in a slow closing of the gap as the net effect?
This doesn't take other factors into account. Compute capability, infrastructure capacity (the chips can't work without electricity!) etc.
Those factors translate into a ceiling for this hypothetical exponential speedup.
So, with new models coming out on a weekly basis, I think trying to figure out a future trajectory is just going to be a wild guess. For all we know, Anthropic and OpenAI models may also be plateauing in the near future; or there may be a breakthrough and they may improve beyond our wildest expectations. I honestly don't know.
That said, I suspect available compute and model sizes that major Western vs. Chinese labs can handle will still keep the gap open at least in the short term.
On the other hand, I am also not sure how much the gap will matter past a certain point. I am specifically wondering at what point people will become interested in smaller models fine-tuned for specific application areas (e.g. cheminformatics) rather than massive generalist models.
There are a number of issues with relying on a frontier LLM for these kinds of tasks.
One, there are only 3-5 companies in the world that have the capability, and you depend on whether they're interested enough in your use case (do you think you can interest Meta in putting support behind optimizing support for Swedish law, for example?). These companies measure their investments by ROI, and they focus on things that have large markets. Heck, Google in particular is kind of infamous for suddenly killing off projects that don't work for them.
Second, all of these companies are American, which may be a no-go for certain use cases involving strict data privacy.
Third, these capabilities may need to be maintained and updated, and you'd depend on the goodwill of frontier providers to keep doing that.
Fourth, the frontier model is likely to be more expensive to deploy and run than a smaller specialized model.
You may have a point in a world with infinite resources, but the reality is that this situation is influenced by plenty of market constrains.
Their open weights models are the the smaller ones, which aren't released yet.
Lots of research goes into open-weights, optimizing cache re-use, storage, inference costs, etc., and all these benefits are easily rolled into closed AI models. Meanwhile any advances made by OpenAI, Anthropic, etc. are kept hidden away to give their models an edge.
It'd already be a miracle if open-weights were gaining on-par with closed-weight models. That they've been closing the gap for most of the past year is wild!
I don't know how frontier western models fare in these aspects, but I haven't heard much from them yet.
But yeah the big players are totally going to try to clamp down on distilling.
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
American history is open-weight. American trade is closed-weight.
Anecdotal though, N = 1.
Naughty Naughty. AMPK pathway is too "dangerous" to talk about.
At least it doesn't call the cops on you if you try, unlike Anthropic.
For comparison, here's one I got from Qwen3.8 Max (at default reasoning effort): https://simonwillison.net/2026/Jul/20/afraid-of-chinese-mode...
This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.
Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.
Here are those 3.8 Max 0902 pelicans: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... - they're mostly not as good as 4, but I prefer the 3.8 xhigh one.
https://stealthgoat.com/fnfl/pelicans3.html
https://stealthgoat.com/fnfl/pelicans8.html
My favorite, and shows this is not all benchmaxxing, this model is just good at SVG: https://stealthgoat.com/fnfl/pelicans6.html
Oh, and, quantization matters: https://stealthgoat.com/fnfl/pelicans2.html
Bird knees are also quite weird though. Don't some bend backwards ?
You are not the first one to think of this. And Simon has thought of it as well, and commented on it several times.
It clearly hasn't; if it was quality of the pelicans and bikes would be static over time. Instead the quality has been improving along with the models.
Like... we get it. Simon likes pelicans. If I want to find his boondoggle robo pelican drawings, I know where to find them by now.
No need to spam every single model release thread.
So, again... What should I be "paying close attention" to?
Here are a bunch of Pelicans I made in my basement. Side by side quant comparisons and something for fun :)
https://stealthgoat.com/fnfl/pelicans3.html https://stealthgoat.com/fnfl/pelicans8.html My favorite, and shows this is not all benchmaxxing, this model is just good at SVG: https://stealthgoat.com/fnfl/pelicans6.html Oh, and, quantization matters: https://stealthgoat.com/fnfl/pelicans2.html
Heres some examples.
I'm curious if an increase in ability to generate this parlays into the model, generally, being better overall or in other adjacent areas?
Thanks again.
For example the IQ3 XXS model was quantized to maintain knowledge in some areas more strongly than others via certain data that the optimization algorithm would then use to decide which parameters could get quantized and by how much while meeting some overall target weight goal.
So, you get to just sorta see weird or lower quality pelicans. It still is /mostly/ for fun but it is not a meaningless meme as far as it goes. And you can even see evidence of that, look at Qwen 4 max pelican vs. Qwen Flash 3.8 at Q4... IDK.
So, I think the best thing you could say is roughly what I said, how well it abstractly connects two specific things, e.g. knowledge of pelican and bicycle physical configurations and then how it expresses that into SVG. For example, you can read thinking traces on some of those pelicans I sent (well you can't, but I did) and it would specifically say "The pelicans legs must be rendered correctly, I must not put one leg on the wrong side." and it would then still do it wrong meaning it just didn't understand the SVG well enough. But it might be able to render an image. Also read thinking traces some time on ascii generation and you can see they have a vivid concept of lots of things, but their ability to translate that to ascii is hilariously bad on basic prompts.
So they need some baseline level of understanding about the world and physical object relationships which even qwen flash 3.8 has, they need to be able to recall it, and then translate it on top of some problem domain in some meaningful way and a lot can break down on any of those steps. IDK, maybe I am just too tired and spent too long with LLMs in the last month.
The original concept of this "benchmark" was to see how the model performs on a reasonably complex task that probably isn't in the training data.
Even if they don't fine-tune for this specifically, there are now surely tons of examples of people asking for a pelican on a bike in the training data, and making an SVG isn't that complex of a task for current gen models.
Even with the same model at the same thinking level, you might get different quality out of the same prompt on different runs. If you want to use this as an eval, you would really want to generate 100 of them, and see how often they're anatomically/mechanically accurate.
Im not sure how you get the pelican to sit still let alone pedal, but there is a reason ai salaries are so high.
They’re not banned, but must be discouraged. It’s like someone being loud at a museum or using their bright phone at a movie.
If you think it breaks the rules, then flag or downvote. If it just annoys you, collapse it. I'm not sure why this needs to be a whole discussion every time Simon posts a pelican.
If you think HN is transforming into Reddit, I think you may not have visited reddit recently, nor have read the HN guidelines (people have been saying this since 2007) or earlier
> Please don't post comments saying that HN is turning into Reddit. It's a semi-noob illusion, as old as the hills.
I also dont't know why anyone would downvote a pelican post when it's not offtopic? We're not supposed to downvote stuff we don't like or disagree with, just stuff we don't find appropriate or relevant to the discussion.
Although if this keeps up, soon there'll be another camp of "comments complaining about Simon's comments being annoying are annoying" :)
You stop one, you'd stop the other.
You are right, I didn't used reddit recently and don't use it much these days. I like community here.
Because you keep complaining.
Would love to see an SVG of an Elephant on a skateboard...
“there’s little evidence that AI labs are pelicanmaxxing. Or at least they’re not doing it in a plainly obvious manner. Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict.”
As far as I've seen, among the popular models today, there are still significant gaps in the quality of bike-riding pelikan svgs, making the benchmark still useful. Maybe not as a differentiator of frontier models, but certainly still informative for lesser competitors.
Harder to train off all them.
Aka, let him have his fun, others are making benchmarks too
[^1] https://twitter.com/sunjiao123sun_/status/202455551655137292
"We need to make the pelicans look good, but not so good that it looks like we're using them as an internal benchmark."
2. I think they still are _somewhat_ informative. Like, the same model may produce very different pelicans at different reasoning efforts, even though the training data is the same. So looking at pelicans may give you _some_ information (like "what reasoning effort I want to use that model with?".
https://dylancastillo.co/posts/pelicanmaxxing.html
What I found interesting is that for the most part, the labs really aren't... except if you look at Otter Riding A Plane. That is close to Ethan Mollick's benchmark for diffusion image generation, Otter Using WiFi On A Plane. The results for DeepSeek and GLM look very much like Ethan's typical benchmark image output, which either means they're benchmaxxing... or they're the only two models that understand you ride inside a plane, not on top of it.
EDIT: Actually Qwen's result for Heron Riding A Plane was fascinating too. Because obviously what you were really asking for was a heron riding a cartesian plane...
A while ago, somebody demonstrated that labs are not benchmaxing on it. It may change but it doesn’t appear so.
So either perspective is properly hard or the moonshine is strong in this one.
https://chat.qwen.ai/s/deploy/t_5fb28dd7-a61f-4946-8e59-8dd1...
Not a dig at you Simon, you've done good stuff, more just in general. Why is this always upvoted, it's like reading The Sun or something.
And why always one image. Should it not be say a 100 images then averaged or something? A prompt with temperature is a slot machine in Vegas, wtf does that have to do with model quality.
The “real” benchmarks are widely available, and frankly kinda sketchy anyway, so why not have a little fun?
I say this because, even just based on my own product related to cad design/3d modeling, the SOTA models have gotten very good at 3d and I don't think the 2d SVG generation tells us much anymore. Maybe I can whip this up myself on my own site if anyone is interested tbh.
Sorry, this post has been removed by the moderators of r/LocalLLaMA.
Has been removed, do you happen to have a copy?It also applies to languages, I use Qwen 27b a lot and it's a lot smarter in english than in french.
But I agree. If the model writes good code and makes good decisions, as long as I’m not building a video game about recent Chinese political history, I don’t care about its politics.
But the same models will tell me about the -arr stack and offer to set it up for me. You just have to frame the prompt the right way, at least for me. It can’t think it’s assisting with immediate piracy. Future theoretical piracy or for educational purposes, you’ll have better luck.
Why do you ask? About half the time a china model gets released someone askes this. Almost as sure as a pelican.
Hoping it’s a great local model though!
Showing 400 of 464. Read the rest on Hacker News
Comments are loaded live from Hacker News and are not stored by Mid or Real.