A colleague of mine has a strategy game to compare language models, 4.1 scores pretty high in this:

https://clankerbattle.com/

It's more common than you think. I work in a startup and we pay API prices too. And we cut a lot of money by switching from Anthropic models to Kimi K3.

All of these flash models have this. You have to build your harness so that it deals with it. Infinite loops are solved by having an error message that says what to do differently on failure, invalid tool calls are solved by making the tool schema less strict and detect things in the runtime etc.

Hallucinations you can't fix. Gemini is a bit worse there than DeepSeek, but there's not much research on how to fix that. The only one is the CaMeL paper by Google, where you tag every prompt and result and then for every assistant response or tool call you first check where it got that data and error if you notice fabrication. This one is really annoying to implement.

With larger models the fabrication starts when the context grows or if you have too many tools, for flash models it's much earlier. We use the flash models for repetitive agentic tasks, where the prompt defines clearly what to do and how. The whole run is about 4-5 steps typically, and context size stays in the comfort zone.

Can you share what tools and processes you're using to do this?

I've been using Pi to build custom extensions and wrapping workflows in shell processes to make it more deterministic and enforce certain validations, all guided by Fable. This isn't production work, though, just playing llm factorio at home.

What you want is a bunch of sessions to replay. Something anonymized if it's not yours, and something that's not depending on state.

You replay all your sessions against your harness, and then store all logs all output, everything to a safe place.

Finally use a blind judge to check everything, and score the output.

Then fix your harness, iterate again until better until you are in a point where it's just the model's weakness. If you get to that, use a bigger model.

Yes. I'm working in the agent industry and my god are we excited on new versions of Chinese flash models. The direct competition is Gemini Flash, and these models are much better on agentic tasks with fraction of the task price compared to Gemini. Things like oh here's a set of simple instructions for you to follow, call these tools, return this report. 20-30% of the price per task. And especially Deepseek Flash produces better quality than Gemini does.

Where Gemini still wins is non-text input what Deepseek cannot do, yet, and Deepseek Flash has this thing of cheaper models where a failing tool call can derail your agent to a retry loop if you're not careful on instructions in the error message.

If they fix and make the tool calls to work better in non-optimal situations, it's much easier to switch from Gemini without a few weeks of evals and bugfixing.

Which versions of flash and at what thinking levels? Which chinese flash models and at what thinking levels? What tasks? What completion rates? How was quality evaluated?

- Which versions: 3.6 vs 3.7 vs. 3.8 for Gemini Flash, and v4 0731 for Deepseek v4 Flash, and GLM 5.3 Flash

- Medium for Gemini, high for Deepseek.

- Things like find information, then understand something about it, then send a slack message or email etc.

- Completion rates somewhere in 80-90%, Deepseek a bit better than Gemini

- Quality evaluated by Fable 5.1 and Astra 6.0 acting as a rubric judge.

Gemini quality would probably be better with high thinking level, but that would be 40% more expensive. And Deepseek is already third the price of Gemini.

Thanks... I have been trying to figure out some things. Been doing my own evals. Flash 3.8 does burn a lot more tokens on high. Interesting how smart and not smart it is. For personal use almost impossible to justify the cost of 3.8 Flash cost.

Deepseek also burns a lot of tokens, its output on high is 2x of Gemini on medium. But it's dirt-cheap so it still can be 60-70% cheaper.

From the large models Kimi K3 is definitely the one burning the smallest amount of tokens. Even if you pay for the fast version in Fireworks it's third of the price of Opus 5 for the same task.

All this really needs evals, the token prices tell nothing.

I am surprised at how well DSv4 flash does in the real world vs many benchmarks. You look at Flash 3.8 and it supposedly beats opus 5 and deepseek is far below.. but they were measuring efficiency, whatever that is… Something doesn’t add up for me on the published benches

Well, it's much more than that. In general everybody's building agents now. You see these things that can help you to do things like adding things like OCR an appointment from a picture of a hand-written paper and add it to your calendar, search things from the internet, find that email with a PDF and add it to your local paperless instance.

Building an agent like this by yourself is really easy. Now, we have Gemini's subscription, OpenAI's ChatGPT subscription and all those, 20 bucks a month right?

What if you can spend that 20 bucks in tokens to do your own. And you pay 15 bucks _a year_ in tokens to run that? And you own the data, you own your code and integrations. It's really easy to do, and these flash models are _more than enough_ for simple agentic tasks.

And 180k in Germany, even in Berlin is pretty good. You are living a very good life with that salary.

That's like top 1% salary for Germany. Maybe 3% if you count family units, or 0.5% for individual taxpayers. I'd say that's pretty good.

Yeah, and buying a house is still kind of out of reach with this income if you don't start paying your mortgage whey you're 20something...

Yep, at least in my case (Spain), given the price of the houses and that banks are giving mortgages for 70-80% of the value, tops, you better have 100K in the bank for the down payment if you want to live in a relatively big city. So even with a good salary, is difficult to buy a house.

They try to build them, but for example in Finland where there's cheap electricity and lot of interest to build them, the people started protesting on rising electricity prices and now the politicians are noticing this.

Same in Denmark.

Maybe then using an open weights model is a good way to hide your tracks...

Or at least use an obscurer model from an obscure company, so they don't know who to subpoena.

Everyone is aware that multiple services offer zero data retention, right?

Super happy subscriber for years... One of those magazines that I open on a Sunday morning with a good cup of coffee and sleeping cats next to me before the family wakes up.

If you enjoy well-written technical content, please subscribe and support them. I hope they continue making the great articles.

Just checking, are they print magazines? Or are you referring to the online articles instead?

It's interesting that Deepseek models were missing in the comparison. I see Deepseek v4 Flash a direct competitor to Gemini Flash for text-based agentic work.

ELI5 always works