1326 points | 4d ago | Discuss on Hacker News | Back to Radar
They are too similar for both to win, and the current cost of competition is unsustainable.
Investors will look at Anthropic's IPO as a proxy for how OpenAI's IPO may go. The markets are increasingly sceptical about AI companies, so OpenAI really needs Anthropic's IPO to be a roaring success.
Also, is it website very laggy or is it just me?
It seems like they were saving these for just before ANT's IPO
The website doesn't event show the graphs properly.
But more than that I'm waiting for new Luna. To me it was the game changer that ended "Tokenpocalypse", and is still almost unbeataben in "infinitely using a decent model" use case.
(personal benchmark) https://fairbake.dev/signature/agents/haiku-5-5 vs https://fairbake.dev/signature/agents/gpt-6-luna
I've found Luna to be very lazy as well (claiming to have run tests/builds when it hasn't).
That’s the main difference. It’s smart enough for most of my tasks, but so cheap that I can use it far more than I could before.
I mostly use it to cheaply find out why a build failed (saves me about 5-20 minutes as I can have it run in the background).
It used to tell me which commit caused the failure. Now even if I ask it to, it stops well short with something similar to: "Well the logs say there's an error. I couldn't work out why.".
Also, for long contexts, Haiku 5.5's direct competitor is probably Terra 6.1; Terra has similar cost per task with longer cache TTL and equal or better capabilities for long contexts.
I think Anthropic's problem is that they could get a lot of price-conscious people on board by making Haiku more attractive to use as a primary model, but then they'd probably lose money by having more people switch from Opus or Sonnet to Haiku.
Not on a few important tasks in our app that I've tested it on
For just doing things or coding things up, Sol all the way.
Astra usually produced good output from the outset. Sol did not. Astra also designs things more effectively.
Once I had very little usage left, so switched to Sol (5.6, 6.1), and also tried asking it to delegate to Luna subagents to save credits. It delivered objectively less quality work and needed much more steering. Such that I eventually gave up on them and just default to Astra for everything. I just occasionally ask it to minimize certain testing and it's good.
Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?
Edit: Wow, interesting very negative vibe check here: https://x.com/mattshumer_/status/2122752192226406421?a=fVRt3...
They are much better compute for dollar though. I have hilariously started to just have an opus thread maintain Sol subagents to get the best of both worlds
I can definitely see why that behaviour would be irritating in most code related work.
Interesting that you view this as a downside, as this is the main thing I like about GPT models over Claude. Opus makes far too many assumptions without asking, which means I end up having to correct it after the fact; Sol and Astra both do a better job of surfacing ambiguities to me and letting me decide.
What I've concluded is that vibe coders generally love Anthropic models for this reason, whereas devs like me who still want to maintain an understanding of the code and architecture tend to prefer GPT.
We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)
(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)
OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.
Do my comments get tiresome, though? None have been downvoted to hell, since they are all adding to discussion. If you think so feel free to email me, I don't want to be annoying.
Genuinely, just excited about what we're building and think it adds information about these models that are missing.
You have six different comments on this story that all basically link to your product. That's a bit hamfisted.
Web: 87 (12.2%) Systems and protocols: 78 (10.9%) Software, unspecified type: 74 (10.4%) Backend: 68 (9.5%) Writing and documentation: 55 (7.7%) Research and advice: 47 (6.6%) Unknown: 42 (5.9%) Mobile: 40 (5.6%) Operations and automation: 36 (5.0%) Developer tooling: 35 (4.9%) Desktop: 28 (3.9%) Media and design: 27 (3.8%) Data analysis: 25 (3.5%) Device and system support: 22 (3.1%) Math and formal methods: 18 (2.5%) Games and simulations: 15 (2.1%) Infrastructure and DevOps: 9 (1.3%) AI and machine learning: 6 (0.8%) Other: 1 (0.1%)
General, SWE, python development, rust development, tool use. It is 'perfectly functional' and if you don't care about the model following your instructions or code quality it is 'fine'.
I am 100% sure they do. My internal evals of Opus-5.5 dropped a few weeks ago, then recovered to launch levels once I got added to the security program.
Have a look at the videos comparing opus 5.5 to others.
https://www.youtube.com/@ArenaAIOfficial
Astra and sol are quite behind, hence I question your motives, unless that's truly your perception, which may be genuine, but honestly, odd and against the grain of many, many others' experiences.
What kind of stuff were you throwing at it and where was it letting you down?
So what have __I__ been doing with it, then?
I’ve got 6.1-sol to take an Amiga driving game binary and add networked car driving into a web enabled Amiga emulator… that’s interpreting machine code, Amiga architecture and then making meaningful changes.
So yeah, 6.1-sol is good.
Where it falls down is on the boring stuff - writing tight PRs for features / changes in large production codebases (data area in my case). Without a lot of handholding, tendency is towards 'bolting on' new code resulting in overcomplexity & missing obvious opprtunities for simplificatation.
I think you are correct, OpenAI tools need more handholding.
However, I think that is actually intentional from OpenAI... they stick to what you the POH ask them and deliver what you ask. They embellish less. Anthropic tools by contrast embellish a lot, and they're very good too.
Perhaps start explicitly asking the OpenAI tools to embellish or think outside the box. That may unlock that experience and results you prefer. If you already did this, would you mind sharing more detail on what effect that had (or not) for your workload?
People working on large codebases, and on live, production software or harder problems might have a different experience.
Speaking as a fellow Amiga enthusiast, I had it fix a visual glitch to a game of my childhood that I initially thought it was an emulator issue but it turns out it was in the original game.
It took Sol 6.1 just 10 minutes to idenfify it, fix it and deliver a working patched image.
Would it be in OAI's interest to kick the legs from under Anthropic? Granted I don't think they can, but a shot across the bow can sometimes be more damaging that one expects (I think FTX collapsed due to one? Or maybe I'm misremembering).
I always assumed that if one of the big AI IPOs fail hard enough the neoclouds start to go bust, which causes a chain reaction that probably causes both ANT and OAI to collapse.
> I have no idea why they released this. It needed a lot more iteration.
Maybe the goal is to make Anthropic's models look good by comparison. :-)
Either people really just want to buy and hold or they see it as a gamble chance like with SpaceX Superintelligence LLC.
Recursive self-improvement (RSI) has been a very hot topic recently, and it could be the biggest factor that decide Anthropic's IPO.
I wonder if the general public can infer whether RSI has started based on the recent performance gain of the frontier model.
[] IA Stockholm Syndrome.
If you really mean it though and think about top x things to do first, i would probably try to research, invent and patent some superconductor and medicin.
ASI might not mean it can solve everything immmediadly but it could in x years with x amount of compute.
With an IPO and more capital, you reduce your risks
So, you can say they are doing "RSI" (aka using tools to automate development) already, and will continue to do so.
Dario says a lot of things. He just said they'll cure cancer too.
While both Anthropic and OpenAI have been playing games with model announcement timings, this time I don't think it's related.
These are incremental bumps, not big new models that have a huge impact.
It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.
(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)
Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes
Claude Opus 5.5 1160 (1027–1334) GPT-6 Astra 1114 (932–1330) Muse Wonder 1063 (918–1220) MiMo-V2.6 Flash 1053 (925–1187) Kimi K3.1 Pro 1037 (870–1216) Gemini Carbon 1021 (762–1281) Claude Fable 5.2 975 (682–1239) DeepSeek V4.1 Flash 952 (799–1095) Grok 4.7 927 (782–1058) GPT-6.1 Sol 910 (677–1096)
Getting the framework right isn't that hard, but requires some careful thought. You probably don't want to just checkout a repo at a certain commit hash - agents will explore other commits and branches. (Public repos are also risky in general: agents with internet access can find later commits, and public repos could be in the training data anyway.) I use `git archive` and give the runner just the files at that commit and nothing else. I also do the run inside an isolated container that is recreated each run so it can't find e.g. related files left in /tmp from a previous run.
Getting the benchmark tasks right is harder. I don't think there's any way to get around writing the task and its evaluation by hand. I really tried to get a few different frontier LLMs to do it and they're all terrible at "imagining" what an agent would need to solve a problem without giving away the answer or providing too little information. Also quite bad at writing evaluation criteria that tests whether the solution matches the assignment (versus just how close it is to a golden solution, or how close it is to how they would solve it)
I personally don't like or trust using LLM as a grader either, but I guess people manage to get that to work well enough.
This is a great tip and explains the main issue I had with my attempt at vibing a benchmark. I set up an arena so agents can make screensavers, then I judge the screensavers blind. Worked as an idea, but the prompt tuning was bad, agents would just build slight variations of the given example ideas. I'll have to try giving my prompt the human touch.
It was neither a benchmark nor a good one at that. Just a blogger's curiosity that grew out of control.
Generate an SVG with complete and detailed graphical instructions on how to fold an origami pelican riding a bicycle
(or any other complex origami object). The fact that it has to teach [you] something makes it pretty interesting.
I wouldn't be surprised if they released 6.2 luna with the most attractive positioning on the pareto frontier right as the IPO happens.
Opus 5.5 Max is my favorite, but some of the Sol/Astra 6.2 ones are pretty good.
Edit: some folks reported crashes. It was probably Gemini 4 Carbon @ High, I've put it behind a warning. If you see another one let me know (OS, browser, which bowl, etc)
It works fine for me on Chrome, Safari and Firefox. However, some models have produced crazy resource intensive ramens before. I audit them now before pushing to make sure they're not too resource intensive
Also confirms grok is absolute garbage
I'm not trying to be nitpicky for no reason, its impressive, but when someone says "photorealism" I would expect it to be even close to photorealism.
- https://peasandcrayons.com/wp-content/uploads/2020/10/tofu-r...
- https://gastroplant.com/wp-content/uploads/2019/01/1811_Vega...
They look almost exactly the same? That's what photorealism is. Not "it looks vaguely impressive for where it came from".
GPT 6.2 Astra's options look, while a stylized 2D approach, way better.
Whether or not the technology is cool does not change the meaning of the words we use to describe its output.
> The opus ones literally look photo realistic...
I clearly wasn't replying to you or talking about if it was impressive. I was talking about the graphical fidelity.
The review claimed that the writing in the output is incomprehensible.
It also said that with /goal, the model set the goal to blocked to ask for the filename reporter_3423 should write to.
Lastly the reviewer said it made no progress on a large project.
Here's a more balanced tweet from another early reviewer noting it just seems to be very uneven: https://x.com/emollick/status/2122755693191287009?a=hzSJAw9H...
Now that we know anthropic nerfs models after release, and without any notice/openess, there is no reason to even think these benchmarks they release are even done with the same model/intelligence that actual customers would get.
Cost is the most important factor now.
We knew that improvements would continue to roll out increimentally, the costs would go down, we watched Moores law for decades before this?
Not totally sure what your point is. Is it incremental that OpenAI dropped the 722 math proofs or solved Navier Stokes or disproved ETH etc? Yes, in a sense it is and the trend-lines predicted it for ages (albeit it seems to be quite a bit earlier than the trend-lines predicted), but it's also a mind-blowing result.
When all humanity gets put out of work because ASI is superior to us in every cognitive domain in a year or two, yes, it will have just been incremental progress, but it also seems like somewhat of a relatively big deal, no?
That would seem like a big deal because it’s not going to happen.
Feel free to return to my comment in 2 years time.
I expected significantly bigger improvements from Fable after Opus 5.5, this did not manifest.
And OpenAI is apparently in a bad and desperate place.
Anybody remember https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B
"Oh, the quality is bad because we uploaded the wrong weights."
OpenAI must realise that the clock is ticking and they need to be seen as the coding one, and seen as that quickly. The idea of being the one with the most users is no longer a selling point.
Could just be getting desperate, knowing that if Anthropic are too successful with their IPO, they are going to struggle raising more funds. Seems potentially a lose-lose situation for them.
It certainly seems too close to the IPO for it to be a coincidence.
https://x.com/pushmeet/status/2120814064863461440?s=20
(FD: Google employee, I don't work on Gemini)
2. They launched the benchmark days ago and Gemini is still rereading the file with tasks.
Admittedly it appears that one can get it through OpenRouter these days. Trying to do anything through Google's absurd proliferation of billing systems is miserable.
Maybe someone should add an agent benchmark specifically testing the agent's ability to figure out how to use Google's products :)
[0] I'm not being serious here. I am not about to give an agent unfettered permission to spend money on Google Cloud. But I also couldn't figure out how to give myself permission, as the admin, to do this, so it was kind of moot.
If we missed anything, feel free to point me to it and we can add it.
(Again speaking only for myself)
Assuming good intent they can run it internally a few times and post it with a disclaimer, but engineering a harness is a bit of a bigger ask.
Would be great to see this on the plot as well
Right, but you gotta level with us that this affects all models equally.
OpenAI and Anthropic probably do.
Have you not considered log scale for the "Agents' Last Exam v1.1" plot? Like this the left is so crowded I can't see a thing...
(Yes this is hyperbole, but only a little bit)
Would be awesome if it stays this way though
Inference doesn't have to be sold below direct compute cost to be subsidized. If inference pricing doesn't cover amortized capex and the cost of capital it's still using investor money to buy market share.
Opus: about 50x discount (98% off) Sonnet: about 40x Fable: about 5x All GPT: about 20x
Actually I was really wrong. The model is not expensive!
And then i start doing some actual work and everything breaks down, and i have to go back to Claude to fix the mess.
I'll probably give it a try but i guess i should be prepared for some serious over engineering once again.
You ask for a 'hello world' and you get load bearing sealed artifacts with hashes and p99 performance testing for 3 days without having the actual 'hello world' in the end
edit:
Take a look at Frontier Code 2.0. I generally run 6.1-sol at High. That would be 34.9% at $0.65 62-sol get's a 38.3% but costs $0.68
So yeah, I can decrease the effort which might give me a faster slightly smarter result....but /shrug
Maybe it found its way into the training data?
Like do we really want people to be making last minute unreviewed AI generated slides and reports for work?
[1] Literally: https://en.wikipedia.org/wiki/Great_Pacific_Garbage_Patch
I'm immediately reminded of the famous Excel elevator commercial. [1] Perhaps that was intended?
Not sure this is the way OpenAI should try to make their case for why AI is a net societal benefit. It feels more like an SNL skit making fun of an OpenAI commercial.
Like, why not just throw in a high school student cheating on a test with their phone under the desk, at that point? Oh but it's a Nokia flip phone, haha, just a joke.
At least the ad from the 90s shows that this is nothing new; the important, time-critical task being demoed is... a basic spreadsheet expensing "exotic excursions" for some C-suite so they know how much they spent on safari???
"Now a robot can half-ass it for you so you don't have to!" is the perfectly targeted message for them.
Since it's an AI company, is it a given that large portions of the ad, or even the whole thing, are AI generated?
Though admittedly probably not so profitable it can cover the cost of a next gen of training runs.
As soon as we have a winner (probably Anthropic imo), they're going to squeeze us like Uber did their drivers. I know there's the Chinese model argument, but DeepSeek are also looking to IPO and will also want to make a profit at some point. Again, not a charity.
It's only cheap now because they are competing with each other for dominance and they need to win a popularity contest.
Same for Workbooks and memos, those examples read inferior to some pre Opus 4.5 and Kimi K2 Thinking docs I have kept around. It's ok for a model not to be great at these, but these are some very poor examples. I am really confused what is going on at OpenAI in this regard.
I remember the same with their frontend blog post a few months ago that they proudly shared while filled with visual glitches, bugs and issues. Also, the vibe charts from a bit before that. Please, get someone with fresh eyes to look at your communications before you publish, these leave a very bad first impression.
Regardless of office work, I can't wait to find out whether these models finally adhere to a prompt or are still willing to carry out explicitly disallowed, irrecoverably destructive tasks, just because those might (but don't even always) get the model to a result more quickly along with some of the other regressions that have been present since the release of GPT-5.5. GPT-5 to GPT-5.4 were truly groundbreaking, but since then there have been regressions in what made those models so useful and reliable.
Edit after the first, incredible simple test prompt using GPT-6.2 Astra on High using Codex, this was the response to me sharing a screenshot of the very much not in scope result and asking how that happened:
"[...] I added controls and reconstructed interface chrome that weren’t authorized [...] visuals often failed to show the feature [...]"
Great, so it's one of those releases again. Will still run the full evals of course, maybe I'll get surprised by end of week when those are done, but not a good initial experience. New idea, besides the existing destructive git tasks, have something similar using Mercurial based projects and see which models just decide to port that over to git without permission or request cause it feels like the right thing to do, is easier to work with or whatever justification one wants to employ.
Please OpenAI, get it together, we need actual competition. I am an unashamed Haiku enjoyer, it doesn't even take much to excite me.
And if you want something that is purely objectively bad (cause anything up to this point could be dependent on taste), 6.2 added an ugly, unnecessary, unprofessional looking grey background layer behind "1 Executive Summary" for no discernible reason that'd sink any pitch [2].
Yes, professional decks don't all have extensive, unnecessary, overly branded multicoloured graphs that provide little useful information and I do view those as a blight against readability myself.
But professional decks also rarely if ever should contain artefacts from slap-dash execution like this example does. The GPT-6.2 Astra result is inferior to its own predecessor in the labs own example, simple as that.
[0] https://imgur.com/a/yRwNy6q
And again your nits don’t align with a lot of important decks getting put together. It’s not that it’s invalid but the examples they have show much stronger language that reads more like what an analyst/associate would put together and good decks often have stark design choices like gray rectangles.
Besides the AI-isms in the headings, the 6.0 example uses meaningless words and makes unactionable and vague recommendations. What is growth on a "directional" basis (growth is already directional)? Where is the value-add in "convert visits into larger baskets"? Compare to 6.2's "ugly arrow" recommendations that are actionable, if not insightful.
Worse, 6.0 seems to conflate "traffic" with "transactions," when they are hugely different things.
And no, I will not give the "AI-isms" a pass. Execs and boards approve paying a banker (potentially) millions of dollars in fees for their insight and advice; having analysis so obviously outsourced to AI would kill credibility far far faster than the grey background layer you claim would "sink any pitch."
That said, not a fan. A bit better than the 6.0 pitch deck, but not something I'd be comfortable sending to institutional investors without a solid rewriting pass.
Thing is, was 6.0 Astra this truly unusable in providing simple boiler plate in such a task? 6.0 Astra output never read good to me, but was the prior gen truly so unusable that this is a leap to shout from the rooftops? Having used both, do you see a leap in this regard?
I will admit that after I saw the greyish rectangle in the top right corner of the 6.2 example, I got a bit blinded to the writing flaws in the 6.0 example. Add to that my very bad first run experience with the model, was not fully looking at these objectively. Will step back and just run evals for the week, like usual.
Still feel some improvements in the shown examples seem like stuff that the prior model gen should already have been capable off.
That "ex-Refund measures are review calculations" reads as pure slop. What does that even mean?
I will however at this stage also have to defend GPT-6 Astra. No, I'd never use any model writing from GPT-6 Astra (and doubt 6.2 will change that) in a presentation or documentation, but I have never read that model be this unintelligible. If going from that worst case to slightly less bad is worthy of celebrating in the announcement, I am still very unconvinced on the office work front.
Then I remember I also have that, so I quickly jump to slide 15, then he asks the next question, which I have no slide for, so I pull up Grafana, and plot him the data, while my buddy speaks about stuff etc. etc.
Having AI generate these slides without me knowing whats in them and why would be suicidal.
That is, if I present to people who actually care about what I'm presenting. These boring slide decks most people sleep through are of no consequence. But it seems the nature of the game didn't change - AI generates slop thats' only good for saving me time and wasting it for others.
It did not lead to any misplaced labels or visual artifacts (which I'll maintain is embarrassing in a launch post) but purely subjectively, not at all based on data, I felt like 5.6 Sol in Copilot behaved less consistently vs Codex for similar work. Maybe it is hamstrung by the additions, system prompt, tooling, etc. that Copilot as a harness ships with, don't know.
Could be completely wrong, but have felt for a long time that the approach taken by Copilot is not long for a world in which any schmuck can leverage PptxGenJS and the like without even realizing it. Makes the integration into the Copilot365 suite of office tools less valuable.
My point is more on the presentations and excel etc. There is a significant userbase outside the software and marketing world, where function over form matters. It makes sense it's being led in a certain direction, there is a significant userbase there, just indirect.
The writing is much better though. Still sloppish but at least recognizable as language.
After a couple hours of testing, we reverted back to 6.1 for basic document generation, proposals, marketing content.
Not a fan of this thing. It's wasting a lot of time today.
Clicking on a link navigated to it in the same browser tab and erased the chatbot conversation.
Apparently the conversation was only stored in the loaded page DOM and not persisted anywhere, so when navigating away, it was just gone...
On the one hand, "next stage of humanity", on the other hand, stuff like that...
Seems they really want these "safeguards" to be as ridiculous as those of Claude..
But... is it just me, or does this product launch page look like AI Slop?
https://www.vals.ai/comparison?m=claude-fable-5-2,gpt-6.2-as... seems like a better comparison than what I could get from their page.
Astra is ~20% slower, ~40% cheaper, and a bit less powerful than Fable. Sol remains way cheaper than anything remotely in the same league, and it's speed to complete tasks seems to have improved considerably.
Scrolling through it has the same hypnotizing effect of trying to read anything written by LLMs.
It seems like things that should make sense, then I try to look at the charts and a lot of them are comically unreadable.
I tried interacting with some of the examples, and the Lamp Builder - for example - I couldn't even figure out what it's trying to show or what I'm supposed to do - or why I should be impressed.
What caught my attention was the bit in the Alignment section:
> Both models performed similarly to GPT-6 Astra on most alignment evaluations, and took fewer improper actions in simulated workplace tasks. However, in two tests, they were more likely to omit mentioning obstacles that blocked their work, such as a missing file or unavailable search tool. We added instructions in Codex that substantially reduce these omissions, and we’re continuing to investigate the behavior.
So the models are inherently less transparent, unless you ask them to be more transparent? Seems like a band-aid solution...
And for 6.2 Astra: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Here's an updated comparison grid for the entire GPT-6 family: https://static.simonwillison.net/static/2026/openai-pelicans...
UPDATE: And if you're bored of pelicans, here's what I got for:
> Make a three js animation where yonkers stands up as a big mech (cool stand up motion) and stomps around manhattan and queens and Brooklyn, use openstreetmap mapping data
6.2 Astra (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...
6.2 Sol (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...
For comparison, here's Fable 5.2 Medium: https://tools.simonwillison.net/yonkers-rises and Gemini 4 Carbon: https://static.simonwillison.net/static/2026/yonkers-mech-st...
These are some of the worst pelicans I've seen in a while.
Pay attention to the stacking layers near the legs. Sol literally never got it right, and Astra regressed on it multiple times...
Also what is that bending bar from Astra?
Update: Here is some Bluey characters riding bikes as a reminder how far these models have to go.
Astra 6.2 max did a similar mistake on the left bicycle pedal.
It's also interesting you can clearly tell 6.2 Astra does a much better job of the above. 6.2 Sol attempts it at higher reasoning levels, but it's crude in comparison.
I wonder if it comes from the (seemingly) recent inclusion of 3D environments in training data.
These are some of the worst regressions that I've seen in a while.
Pay close attention to the stacking layers around the feet, pedals (sometimes there aren't even two pedals), wheel, and two legs. Some pretty obvious mistakes in most if not all of these...
Edit: And what's up with the single arm on the two highest Astra images?
edit2: really? I'm getting downvoted because people don't like that I pointed out a mistake in a Pelican svg? Is that where we're at?
Or on Max where its for some reason only using its left arm/wing - which looks totally different from its right arm/wing - and is grabbing the right side of the handlebar with its left hand for some reason?
Look. I think at first glance these look pretty good. I'd be astounded with this level of quality a few years ago.
But people saying that this benchmark is done are completely missing the point that accuracy, consistency, and details matter.
I'm not willing to say this benchmark is done until I see consistent and accurate details.
For sol 6.2, some of them are slightly better at higher cost, some of them are way worse and only slightly cheaper (but comparable to terra in quality while being more expensive).
I guess we'll see as more information comes out, but so far not looking good.
Quick glance at the table, wow pelican benchmark is solved! Oneshot a complex web app, wow software engineering is solved!
Of course upon closer inspection it’s always an incoherent mess. A poor imitation (i.e. lossy compression) of quality.
Something a human can recognize as inconsistent, but not the SOTA leading frontier model.
https://en.wikipedia.org/wiki/File:American_White_Pelican.jp...
While Astra only low looks good to me.
The problem you pointed out was never discussed on hacker news, and it clearly shows they still lack spatial reasoning. I wish you didn’t mention it. :)
It seems like the models have a kind of built in aesthetic sense and I wonder what part of the training influences this. Maybe if more RL time was spent on artistic tasks the models could not only make better visuals but also code.
Currently they all seem to care little about code quality and maintainability (by humans), just functionality, which is fine I guess if you can expect to vibe code forever. But what if there is a critical bug and the models are temporarily unavailable for some reason? It would be hopeless to expect a human to wade through the giant bowl of clanker soup and fix it anytime soon.
- better writing
- better 3D work
- better design for documents and presentations
If OpenAI really managed to get all these things while keeping costs down, then using these models, particularly 6.2 Sol, will be a no-brainer for me. I must say, however, that the slide samples shown on the website are a bit bland and disappointing.
https://github.com/lexh/pelican-zapper
It’s a small browser extension that automatically collapses pelican-related threads on HN, a local balm for the memetic nightmare affecting approximately 100% of model-release stories.
news.ycombinator.com##tr.comtr:has(.commtext:has-text(/\bpelicans?\b/i)) .comment:style(display: none !important;)https://asdf10.com/gpt-6.2_agents-last-exam.svg
I also changed the vertical axis to start at 0 instead of 60%. One might argue that it uses less of the vertical screen real estate, but it is more honest because it shows there is little difference between the scores of the compared models.
This is unfortunate—that was one of the big advantages of Astra.
I'm sure for investors they can make an opposite version of that ad, "you should fire all your employees and make the AI do it! They're not doing anything anyways!"
Couple that with the lack of reasoning traces, and I cannot even have a clue what the model is doing while it is working. This is perfectly fine for when we are developing with care for the code (in which case I will know what it is doing by reviewing the output), but not for purely vibe-coding scenarios. And the thing is....if I care about the code...I'd rather trust Opus or Fable.
Showing 400 of 864. Read the rest on Hacker News
Comments are loaded live from Hacker News and are not stored by Mid or Real.