Docker has an ARR of ~$200mm. They did that by pivoting to a strong focus on developer tooling in 2019 after they nearly went under due to focusing too much on their enterprise infrastructure product.
I would be fine being a business “failure” by that definition.
As someone who grew up around professional chefs you might not understand the true baseline level of cooking knowledge that many have going into adulthood; following a recipe is a challenge and they have no idea why they do any of the steps.
Kenji Lopez-Alt (weird that you intentionally chose to write his name in a way that he doesn’t) is by definition not an amateur (he worked professionally in kitchens), but more importantly, he is a really good communicator. His work is about explaining the why of cooking instead of the how to people who did not have the benefit of being raised by professional chefs. He may not have pushed food science forward, but he has pushed understanding of food science forward. His writing is for people that have no clue that adding pasta water to a sauce is a thing, let alone why. Of course professional chefs know to do this, and some of them even know why.
Has he had some dud recipes? Sure. That’s a good thing though, it means that he is willing to both try new things, as well as make mistakes.
Apple has a real knack for not being first mover but being “best” mover. They didn’t invent tablets or mp3 players or thin laptops or ARM laptops, but once they entered those markets they absolutely became the definitional product for the entire category.
Agree with this. As soon as things get in range for motivated amateurs, progress skyrockets.
Has also been the case for things like chess computing; a lot of the progress we made over the last decades there (even before involving neural networks!) happened thanks to software improvements because the problem got so accessible, not just faster hardware.
I expect similar trends with AI; I'd expect to get decent, human comparable capability with <200GB/s of memory bandwidth and under 60GB of RAM long term (SSDs with very high read bandwidth looks also promising, but we'll see).
I hope that in a decade or two, training will also be somewhat feasible for "pro-sumers".
>As soon as things get in range for motivated amateurs, progress skyrockets.
So true. The reverse is also true - when greedy companies overprice their initial release so that it is out of range of the enthusiastic hobbyist they stall progress and adoption.
This is true for hardware (eg failed Intel Optane, Knights Bridge) and software that does not have a cheap or free basic plan.
Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk.
In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of tok/s will far outweight any single user use case.
Those two combined means that unless LLMs reach the state of 'good enough' (TBD), I fully expect the economics and people's interest to align with 99%+ of LLM usage would be in centralized servers. (not dismissing the fact that there are use cases for local LLMs, and that the progress matters).
> Those two combined means that unless LLMs reach the state of 'good enough'
Once you are able to not just go on-premise, but on-device, there a boatload of other advantages than efficiency (network independence, privacy, etc.) that make it attractive.
Tiny language models are already plenty "good enough" for a lot of on-device use-cases. E.g. there are a lot of things, that can be reformulated as ranking problems, where you can apply pairwise ranking, which even small language models are good at.
I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens. But we could say the same thing about a lot of the computation that iPhones do locally. They could have been much thinner clients, but instead they now have more compute power than desktops had when iPhones launched.
> I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens.
The economics of real-time, low-latency inference of very large near-SOTA models will heavily favor a centralized setup. But if you can afford to wait for your answer - be it a day, a week, or even more at the extreme low end (or if you just stick to leaner models for your relatively quick replies) the economics start to shift in a very clear way. A slow-going local inference setup relying on cheap SSD offload does not need the high power input of a datacenter rack, and the cooling load is outright trivial - even when working on many requests in parallel, which (in a SSD offload context) is what maximizes throughput even for local inference. These are serious problems for centralized inference that will probably limit the scale at which it can be applied.
iPhone are in practice more reliant on central servers than ever before. Except for some games, if you take a random person's iPhone it becomes almost useless without internet connection. Which is exactly my point, it's not about compute, it's about people's expecations and priorities. The main driver of the need for higher performance chips on iPhones is its camera and smooth 120fps UI animations, not local processing for apps for almost all people.
However, the compute situation just skew it even further when it comes to LLMs.
Phones are communication devices. So I would always expect connectivity to be central to their use. But that's a different issue than the "inefficiency" of edge computation and whether powerful edge devices can sell (they obviously can).
A lot of the compute that iPhones do either has high bandwidth requirements (photo editing etc) or low latency requirements (games etc). LLMs are an interesting case where unless you're feeding in images, it's just text
> Most people are already used to rely on the internet on basically everything
FAANG has trained them well. "Normal" people also use chatGPT & co. to ask mundane questions that qwen27b could easily answer (random trivia, writing emails, correcting translations etc), why do these simple requests need to be processed on a B300 on server on the other side of the planet? There is no reason.
And this being hackernews, I would expect people here to go against the grain in this regard...maybe even promote local llms to "normies".
1. Better hardware optimized for matrix operations and other ML/NN specific workflows (NVIDIA hardware, Apple M4/M5 chips, Intel AI Boost, other NPU/TPU chips, etc.).
2. Bigger RAM/VRAM/unified for loading models in memory -- with 3090/4090 you can run 22-24B models in VRAM; with 5090 you can run 30-32B models; with Apple hardware with 128GB unified RAM you can run 70-128B models.
3. Better quantization (smaller, better value representations/lower errors/loss, optimized CUDA/etc. kernels) will lead to larger models being able to be run on the lower VRAM hardware at (with optimized kernels) decent tokens/s.
4. Better optimizations in general taking advantage of the hardware (coupled with the hardware and driver advances) and the NN architectures (NOTE: I saw a huge performance boost a while back after upgrading my llama.cpp version).
5. Better model architectures optimized/designed for running on lower-spec hardware, like Mixture-of-Experts which make it easier to run on a single GPU or distribute the weights more efficiently across multiple GPUs (i.e. you know that specific weights will be used together so you can keep them resident on a specific GPU and not have to swap the weights in/out of the various GPUs or transfer intermediate results of a single layer, only the result from that block/expert).
5. Improved NVME or similar SSD technology which a) has a large amount of storage, and b) has access efficiency on par with RAM/VRAM, especially if/when coupled with efficient DMA access to the GPU/NPU.
Unfortunately you are wrong. Time marches on and progress in silicon does too. You may not like the timeline, but it will happen.
While not exactly an “SSD”, High Bandwidth Flash will make it to market and make big sparse MOE models even more accessible. Everyone knows bandwidth is the game right now and you think hardware vendors are going to sit on their heels and not take part of the market? Absolutely silly to discount progress both on the software and hardware sides.
I don't understand how anyone with any degree of technological or scientific literacy or knowledge of advances in the past 100-200 years can say something won't happen in the future.
I am absolutely certain it will - the only uncertainty is how long, but given the rate of progress, I'm guessing it won't be far away.
it’s equivalent take to laugh at first transformers 9 years ago, because they were shit and hardware requirements were immense.
IMHO it’s a matter of time until we (consumers) will get the hardware (maybe coupled maybe even more novel techniques). Though I expect it will take another 10 years or more.
The frontier labs will do their best to prevent this from happening. Their financial model won't work if people start running open-weight models on their local hardware.
This is why banning Chinese open-weight AI models is a major policy debate in Washington. The labs can't survive log-term without subsidies, and a ban can act as a subsidy.
I concur, that this could be a possiblity and it definitely seems now.
Though my bet would be, if USA will go ultra protectionist in this regard - in 10-20 years most world will run Chinese LLMs and hardware for this purpose.
"It will never be possible to have a computer in your home. You will always have to buy computer time and access via dumb terminals. Technology will never again advance."
Seems like what Apple's going for with afm3. Their latest model that will be embedded in macOS 27 is a quantized dense 20B that only select between 1 to 4B at inference, based on the prompt, not token by token. If only they could make a 100B or 400B dense that selects ~5 to 15B...
Nothing to understand. Straight up hallucination. I could have sworn I read that they used a novel architecture where the model is dense but you could select specific layers or something at inference. reread the announcement: just said MoE. Corrected my brain's weights so thanks.
>when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
I am 100% sure this won't happen in 10 years time. At least not on a $200 SSD. But I wouldn't be surprised if it ran on $1K to $2K HBF SSD. It will still be better than a $50K Rack.
There are certain physical limits. Calculations need to be done. Either less calculations are necessary for the intelligence, or u accept less intelligence. But there is a limit in what u can do with specific hardware.
By that time the same Mythos class run on specialized hardware 5 years earlier will have fixed the same bugs that the Mythos class on consumer devices could have exploited.
No, CVEs are on average still actively being exploited for far longer than 5 years. The oldest CVEs on CISA’s KEV (Known Exploited Vulnerabilities) catalog are from early to mid 2000s.
That's status right now, and that's because we just got models than can speed up the mitigation or find new CVEs. Plenty of reports of how many of them get patched now, most recently with Chrome.
8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS.
1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS.
5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today.
Theoretically around 104.8 TFLOPS.
In the same timeframe mobile processor CPU's went from 0.001 TFLOPS, to today's Apple's A19 Pro chip which delivers 2.074 TFLOPS.
That's _without_ getting into ASIC's, or purpose-built hardware like Taalas's model on silicon HC1, or generic AI dies like what they're planning with HC2 or Cerebras, which will massively compress the timeline.
Yeah but here you describing the opposite phenomenon. You're saying that the hardware is going to become cheaper and more powerful with the years, to the point a current State of the Art model from today will run on a normal consumer hardware in ten years. What people are trying to do now is the opposite, optimize the software as much as possible so that it does not need the best hardware but the normal one we currently have. As if we were trying to make a current AAA game to run smoothly on the 1080 GTX of your example.
(2) Are you saying that you think we're at the limits of computing in general, or that specific technology?
We know, for example, that a human brain level intelligence is possible to run on a human brain. We are nowhere near that. And actually that's not even a physical limit necessarily.
Leaving aside the discussion on LLMs intelligence vs human intelligence, on a purely energy consumption level we are definitely and without any possible questioning nowhere near that indeed.
Here you have shown yourself that progress slows down and doesnt speed up. 8.9/0.35 = ~25x more performance in 10 years from 2006 to 2016. 104.8/8.9 = ~12x more performance in 10 years from 2016 to 2026. Growth has dropped 50%.
That isn't deceleration, you've just chosen a very selective way to compare. If you use time as a denominator, which is kind of intrinsic when talking about rates of acceleration, you get a very different result. If you graphed .3, 8.9, and 104.4 on the y axis, with years on the x axis, it would be pretty clear that there was in increase in the rate of progress.
We went from adding 8 teraflops in a decade, to adding almost 100 the next decade. If we add "only" 400 more teraflops in the next decade the graph will make that initial growth look flat in comparison, even though your math would show that we are basically stalled out.
It’s like claiming that a company that goes from making $1 to $1k to $100k to $1mm in a 4 year period has decelerating growth.
We will see such power and price now only when AI market crashes or China reaches node parity and goes after market share as currently the way they are buying out most of the latest node production the consumer prices will only be palatable to the very rich or we will need to be happy with older slower nodes
Speaking of ASICs - how likely is it that as models get better we'll see someone baking a whole model directly into the silicon? It's like having l0 cache.
This is definitely being done with private models by HFT/quant firms, data processing agencies/orgs (large intelligence agencies, _every_ data analytics org, etc).
Yes, right now it would be obsolete in six months, but I also must add that this never stopped crypto miners from making new ASICs.
However, with how useful Kimi is right now - at some point if someone makes a dedicated hardware board with "good enough" model for daily tasks - that would be a very sought after commodity.
They're important everywhere of course, but especially on mobile. If AI researchers figure out how to offload knowledge and expertise from reasoning weights, then a core reasoning ASIC linked to the knowledge would totally rock.
Just a note that I think the direction most people are paying attention to is memory bandwidth; thats the real bottleneck and “number go up” but also constraint people are designing around
Problem is prefill on these, right? Initial prompt processing takes forever? I suppose you’re right. Cost is not a thing on its own. It’s a performance-cost frontier and one can do CPU inference in the worst case.
Not sure if you're thinking of the same story, but this reminds me of Cliff Stoll finding KGB hackers poking around military and university networks in the 80s because of a time billing discrepancy.
The power grid has the potential to be insanely dangerous. We largely have eliminated that danger because of tight regulations and codes.
Look up backfeeding, which is a huge concern with this stuff. It is entirely possible for anyone sharing a transformer with you to send power back up the lines to your house in a dangerous way. That’s why you shouldn’t just plug a generator in with a suicide plug when the power’s out.
A grid tied battery without the right protections is electrically identical to a generator with a suicide plug. We want some standards here.
He takes a scientific approach to cooking, and could accurately be described as coming up with techniques that are in the spirit of finding non-traditional ways of doing things.
He hacks food in other words.
Someone doing a technical write up about a new technique in their discipline is pretty hacker news to me.
I would be fine being a business “failure” by that definition.
reply