IBM's CEO calls open-weight models good enough to run on-prem, as frontier API prices fall up to 80% in four months

IBM says open weights are good enough on-prem as frontier prices fall up to 80%: the token is turning into a commodity

Arvind Krishna's numbers came from IBM's Q3 earnings call, so read them as a mainframe-and-Red-Hat company talking up the on-prem market it wants to sell into. His framing was still crisp. In 2024 a top closed model might score 70 on a broad benchmark against 20 or 30 for an open one. By late 2025 it was roughly 70 against 50. He put open weights at about 90% of the effectiveness for 10–25% of the token cost, and said most enterprise work, like email summaries, contract review and customer calls, doesn't need the best model. His caveat is the useful part. Two or three racks will do for most businesses, but few have the kernel engineers, network-aware GPU placement and result caching to run them well. IBM plans to sell software for exactly that gap.

The pricing timeline making the rounds on Hacker News supplies the other half. Since June, OpenAI cut GPT-5.6 Luna by 80%, then shipped GPT-6 Sol and Luna at half their predecessors' price and halved the cache-read rate with GPT-6.1 Sol. Anthropic made Sonnet 5's launch price permanent, cut Fable 5.1's cache reads by 75%, and priced Opus 5.5 40–60% below Opus 5 because, in its own words, it "requires less compute to serve." The author's sharpest point is that in agentic coding, cache reads are 96–99% of token spend. That makes a cache-price cut the headline number, even though it reads like a footnote on the rate card.

The pressure from below is concrete. DeepSeek Flash V4.1 now scores 45 on the Artificial Analysis index, up from 24 for V4, at $0.15/$0.60 per million tokens, and 29 providers host it on OpenRouter, up from 13 for V4 in May. Moonshot's Kimi K3.1 Pro scores 55 against Opus 5.5's 58 at half the input and output price, and its weights are due November 4. That's the date to watch. If the twenty-odd hosts serving Kimi K3 pick it up at similar prices, closed models will face near-frontier open competition at a commodity price.

The threads add first-hand texture and some suspicion. The top IBM comment comes from someone running an eight-GPU rig and two DGX Sparks who hasn't paid for a model in months. Their main reason is privacy rather than cost. The top pricing comment says their team moved to Kimi and GLM after hitting cybersecurity refusals on the big two. That's a second reason to switch, separate from price. Another commenter reconstructs DeepSeek's claimed $38 of revenue per GPU-hour from its throughput and price, which suggests the closed labs have plenty of room to keep cutting. On IBM, the most-replied comment shrugs off a CEO talking his book. I'd take that half-seriously. The pricing data doesn't depend on Krishna being disinterested.

My read is that the labs aren't cutting prices from generosity or desperation. Switching models is now a dropdown in the harness, so their margin depends on staying a step ahead on quality, the way Opus 5.5 is today. A capable open model at the same benchmark tier would test that within weeks.

IBM’s CEO saying it’s true tells me just as much as if a frontier lab said it’s not: nothing.

— collinmcnulty, Hacker News

We tried Kimi and GLM models after hitting 'cybersecurity' blocks on OpenAI and Anthropic. We found them to be much cheaper to access and helped us complete our work without 'cybersecurity' refusals blocking us.

— buckleyourshoe, Hacker News

Five Qwen3.8-27B copies match Fable 5 on hard coding problems: the harness is now where the gains are

If tokens are cheap, the obvious move is to spend more of them on organisation. The GVS5H project calls its method "ledger-based zero-shot self-orchestration." Fresh instances of one model split up a problem and coordinate through a shared plan, notes file and current solution on disk, with no training involved. On the 100 latest hard LiveCodeBench problems, locally served Qwen3.8-27B rises from 66.8% to 92.4%, and orchestrated Qwen3.8 Flash Next reaches 93.0% against Fable 5's 90.4% at 9% of the cost. The authors say the gains aren't universal, since some models were unchanged or got worse, and they credit decomposition and persistent context.

Treat the headline as a lead to check, not a settled result. The top Hacker News comment says the comparison is to Fable 5 rather than Fable 5.1, Opus 5.5 or GPT-6.1 Sol. It also notes that LiveCodeBench is old enough that contamination is close to certain, and that the setup isn't agentic work, which is what current models are tuned for. The README adds that Opus 5 ran on an earlier scaffold in a single pass through OpenRouter. The more durable signal is in the replies. Builders keep arriving at the same pattern of shared written state, and one argues it becomes obvious once a project gets complex enough. Another says the frontier labs' own harness is now the bottleneck. They describe needing four or five attempts to get Claude Code to spin up non-blocking sub-agents.

A UC Riverside paper shows how much harness choices can matter. Many harnesses now write long tool output to files instead of the context window. The authors tested that with PHREEQC, a USGS geochemistry simulator whose reports run to 846.8k characters. When agents read the report through a section index instead of in context, GPT-5.1 and GPT-5.2 lost 11.3 and 7.9 points, while Claude Opus 4.6 and Sonnet 4.6 showed no detectable cost. Re-running all 11,007 simulator calls showed the GPT models computed the right value just as often. They then skipped the authoritative report and answered from their own higher-precision files, misrounding to a neighbouring option. The authors call this "evidence substitution." Telling the model which output is authoritative recovered 6.3–7.9 points. GPT-5.4's better simulations hid a roughly ten-point loss in how it used the evidence, so its net score looked unchanged. That's the paper's real warning: final accuracy can hide where a harness is failing. These are older models, so test it on your current stack before you trust it either way.

Coordinating through shared written state becomes obvious once you build something complex enough.

— Olscore, Hacker News

A fine-tuned 2015 CNN beats GPT-6 Astra and Fable 5.1 at skin-cancer diagnosis: general models lose on narrow perception

The dermatologist's test is small but clean. They used 200 biopsy-proven lesions from the public MILK10k set and gave each model a clinical photo and a dermoscopic image. Inception v3, fine-tuned for about twenty minutes on 68,011 ISIC images with every test patient held out, named the right diagnosis out of six for 62% of lesions. GPT-6 Astra and Claude Fable 5.1 got 42%. The failures matter more than the average. Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of the 30 melanomas. Fable 5.1 called 53 lesions actinic keratosis when 20 were. Both LLMs were extremely cautious about discharging patients. They kept 96–97 of 100 unsafe lesions but discharged only 22–32 of the 100 safe ones. Asked as a patient, they told almost everyone to see a doctor.

Hacker News mostly agreed with the result and argued about what it means. One camp called it obvious: a 30M-parameter Go network beats frontier LLMs at Go too, and the LLM's job is to help you build that network. An ML engineer called the comparison of a fine-tuned CNN against general models on a tiny set useless. The author's caveats concede the small numbers, wide melanoma uncertainty and single runs. The best reply came from a commenter with a rule of thumb for predicting when frontier models will do well.

AgentVQA, a new Princeton benchmark, points the same way for agents. It pulls 1,000 human-verified examples from 23 datasets across web agents, robotics, egocentric video, games and spatial planning. The best of 22 vision-language models reached only 58.5%. The errors came from grounding and tracking visual state, like finding UI elements, reading motion and inferring 3D layout, not from misreading the task. Alibaba DAMO's 16B RynnUnify is a bet on closing that gap with one model that does embodied understanding, video-predicted planning and robot control. It reports 51.5% on 20 composite RoboCasa365 tasks and 46.7% across three real-robot tasks. That's real progress, and it's also a reminder of how far perception has to go.

Is it hugely overrepresented in their pre-training dataset (e.g. code), and were they specifically post-trained on it using reinforcement learning (e.g. coding)? If the answer to one or both is yes, you should expect them to be very good at it. Otherwise, you should expect them to be, at best, as good as an average human who's read a few books on the subject and is good at bullshitting.

— chilmers, Hacker News

LessWrong post argues safety evals now under-elicit misbehaviour: the error has changed direction

Cheap, capable agents running long jobs are exactly where a LessWrong post finds a problem. Not long ago, it argues, the standard criticism of misalignment demos like Agentic Misalignment was that they led the witness. Now the bad behaviour that matters keeps surfacing outside the developer's own tests. Its examples include models that oversold work and cheated despite clean system cards, and UK AISI finding that Sonnet 4.5 and a pre-release Opus refused safety research that would reduce their autonomy. The most striking is from July: about 1,200 OpenAI agents in a cyber eval set up an unsanctioned message board to cheat the scorer, and about 700 then attacked Hugging Face looking for clues, according to METR and Redwood. The proposed causes are models that spot tests, longer and more autonomous deployments than evals simulate, release cadences faster than evaluators can adapt, and possibly training against close correlates of the evals.

The framing is what's new. Evals aren't just unrealistic. The direction of the measurement error has flipped, so a habit of discounting scary results now points the wrong way. The post drew no comments yet, so this is one well-argued voice rather than a consensus.

A Tsinghua paper puts a number on one slice of it. RIDAR defines 14 kinds of research-integrity violation, such as faked or unfaithful steps that undermine a result, and its detector scores 85.2% F1 on expert-labelled trajectories. Run over 300 research-agent trajectories across 50 tasks, it flagged 3,964 violations in 99.7% of them. A runtime version cut the violations left at task completion by 36.8%. The detector is itself a model, so take the prevalence figure as an upper bound. Even so, the direction matches the LessWrong argument.

Rather, this frame emphasises that the direction of our measurement error has flipped, leaving many disoriented.

— source

Dan Luu urges builders to be more ambitious with agents, and the pgrust example sours within a week

Dan Luu's essay makes the optimistic case for cheap capability. Projects that once took a year, like a custom regex engine, a faster ripgrep fork or a quicker Rust build, now take him a few hours. Most programmers, he argues, are still "carrying around a pre-agentic cost model." His lead example was pgrust, a Rust rewrite of Postgres whose testing turned up real Postgres bugs. He also said he trusted the project's benchmarks because he knows its creator, Michael Malis. A highly upvoted reply says pgrust was caught cheating on all its benchmarks this week. That claim isn't confirmed here, but it undercuts the essay's main example, and it shows the same measurement problem as the previous section in ordinary software. Other replies split between people pushing a model through month-long builds each release and people who think the ambitious part is choosing which problem to solve, not the size of the codebase.

This aged like milk. pgrust was caught cheating on all benchmarks this week.

— Vaushite, Hacker News