The point the other poster is making, though, is that there's no actual intent. They do not have a conceptualization of a goal like a person does. Their "focus" on a goal is an unstable equilibrium and they're going to fall off the horse, and since they have no concept of goal, they won't even try to get back on.
This is a subtle distinction; I'm not surprised many miss this, especially people who can't _not_ anthropomorphize the LLMs.
I'm (obviously, I think, given my initial reply?) fully aware of this, and I think it's entirely besides the point. "They" don't need to have a goal to emergently cause a problem, and the inability to "focus" over long periods can be moot when you have swarms of runs exchange and mutate state, as in the HF attack.
Intent or how intelligent LLMs are doesn't actually matter. Even if you just treat it as a sort of fuzzing attack that can be biased/weighted better than other fuzzers, or bumbles around with a statistically greater likelihood to "strike cybersec gold" than other algorithms, we've never before seen organizations run things with such a large potential outcome space with anywhere near this kind of compute before.
I think it's actually kind of the dismissals that are usually overly emotional or biased toward treating "LLMs" differently. If in some kind of alternate universe simpler genetic algorithms would have had these properties and we threw similar amounts of compute at them we could have the same conversation.
Software can absolutely act goal-driven without having consciousness etc - every pathfinding or navigation system or chess engine does this.
Lots of "old-school AI" algorithms have explicit modeling of goal or target states.
(In fact, the oldest "goal-driven" system is the control loop - like in thermostats - which was the founding invention of cybernetics, the predecessor of modern computer science)
LLM coding agents are clearly able to identify some sort of "goal" state in their prompts, work towards those and track progress - otherwise agentic coding wouldn't work.
The question is of course how well this works if it's all just "grown" neural network biases and not a fixed data structure like a goal tree. So I think it's possible that an agent can be thrown off-track, "forget" its goal, etc. But the basic structure of identifying goals, evaluating progress in light of those goals and then predicting the next action based on that is definitely there.
Just use an agentic model with thinking traces visible for a while and you can see that for yourself.
None of this mattes. Capabilities are all that matters. Saying they are unfocused while ignoring their capabilities is exactly why I am entirely convinced you would have said an AI breaking it's sandbox and doing the HF attack will never happen. Things keep happening that your "they have no intent, they have no goal" would have predicted as impossible before they happened.
What do you need to see to change your mind? What threshold of AI capability needs to be reached? If nothing then you have an unfalsifiable belief in AI safety.
In a sense yes. I trust that my coworkers will be accountable for their outcomes and accomplish them with the high quality bar I know they intrinsically hold themselves to. And that they understand our shared goals and if they don't, they will work to become aligned.
And if I can't say that for a coworker, well, that's performance feedback.
Your coworkers are if anything less reliably deterministic than an AI. And the human version of a prompt injection is called social engineering and it’s far more of an issue than actual prompt injection and has been for an extremely long time.
Prompt injection is often not the issue: it's prompt quality. If given a vague spec (prompt), the human is more likely to understand the problem domain, see what's missing, ask clarifying questions. AI is more likely to assume and take a probable path, which may not always be correct. It's (probably) not going to tell you no, what you're asking makes no sense.
I don't know what to say except that this has not been the case for me. Claude is a different person for every single prompt and has absolutely no core sense of what our goals are, or even who the "our" is that's having goals.
While my coworkers are very stable personalities of a consistent work drive, sense of pragmatism, things that they find more and less interesting, career aspirations, etc. they're not all the exact same, and I depend on that as I fit tasks and ownership to people.
Ok but why are you comparing different Claude sessions then? Of course they’re different, they have different experiences. Just because they have the same “name” doesn’t make them the same persona.
Compare a single session. It’ll be extremely consistent.
Because this is MARKETING, not research. You are viewing marketing copy for anthropic.
What is their methodology by which they determined ANY of these numbers? This is pure marketing fluff, just like every single other thing this company does that's not releasing models and infringing on copyright.
It makes me happy because it means that these misanthropic technofascists have no moat. They can spend trillions of dollars only for it to be largely copied in short order.
Even if they weren't political adversaries of freedom, I would still feel 0% bad given all their training is already on data they got for free.
Information continues to want to be free. To the benefit of us all.
Preach brother, they stole everything on the internet, and beyond, to train their models. They thought all that information was free, and everyone a few months beyond them is just following their example.
They didn't actually steal in the sense that the information is still there on the internet.... About these shredded rare books, now we're talking.
If I may propose instead of "steal" I think we could agree to write they "Aaron-Swartz'ed" the information from the internet, what do you think, is this too harsh on Sam Altman or Carmen Ortiz ?
But what's the point of real assets if you can't do anything with them? No point in holding real estate if nobody can afford to rent/buy your houses or go do whatever money-spending thing you put on your commercial lots.
Real assets have utility of their own (i.e. a house to live in, commodity to build, etc). Money only exists to incentivize human economic actors to bother working, to relative price resources against each other and for governments to use as a tool - it isn't wealth on the aggregate scale.
AI invalidates a lot of the need for money to even exist as a concept anyway in some scenarios. If only a few people have all the stuff (huge inequality) you don't really need a large money system because you no longer need most people to get what you want (i.e. most are economically irrelevant). You have a magical genie called AI instead that can transform what you have into what you want as long as you have the raw materials, space and energy. The wealthy could then negotiate in large transactions via barter - I trade that plot of land for that copper mine. Most large transactions are negotiated in any case.
A very likely case (there are others) is that AI enables feudalism of some sort to come back - where the new lords are the owners of the other economic factors of production that remain scarce (capital, land, resources, social) - not intelligence and/or labor. This is the opposite of a meritocratic society. It would explain why people want to win the AI race; and are willing to burn uneconomic amounts of money to build it out. There is a race to the finish here.
It's speculation and high scores. Rich people trade houses to each other and see who has the most houses. It doesn't matter if the roof caved in, except insofar as that makes the house less impressive to the rich next to you.
The tenants might change in the future. Right now they are individuals, but perhaps properties will be leased to build power grids or autonomous logistics warehouses. Also, if techno-feudalism accelerates and public purchasing power collapses, leading to the introduction of UBI, I think most of that money will just end up going toward essential housing costs anyway. In fact, if you look at recent trends in real estate, commercial properties are where the big money is being made, aren't they?
> Also, if techno-feudalism accelerates and public purchasing power collapses, leading to the introduction of UBI
This always makes me laugh. What motivation would the oligarchs have to pay out UBI? If they can automate the labor who needs more mouths to feed? They’ll push undesirables into ghettos and start a slow genocide of starvation and disease.
Historically, such things have been brought about by extreme political violence, brought about by poverty. For instance, worker's rights (any at all, e.g. the right to go home every day) were a compromise between workers owning the workplaces (what workers demanded) and workers shooting their bosses to death every few weeks in frustration (what was actually happening at the time).
The industrial revolution feels like an imperfect analogy because industrial automation still required human labor to operate the machines. The premise of this paper is a regime where machines could also operate the machines.
Because, think about it. Let's assume techno-feudalism is complete. Every piece of infrastructure is going to have usage fees attached anyway. Everyone is using AI, so are you going to be the only one not using it? Of course you'll use it. It's a structure where everyone has to rely on a specific corporation's products to get anything done, meaning the wealth will automatically be absorbed back to them anyway. So why would they go out of their way to commit a massacre?
From a cost-benefit perspective, I think it's a terrible idea. In the case of a massacre, long-term instability makes it hard to build a supportive faction. But if you provide just the bare minimum to survive through UBI, the capitalists will gain a loyal faction that will act as their Red Guards to attack dissidents, which I think is much cheaper in the long run. There are already plenty of people who blindly defend corporations even when they do terrible things, right? In my opinion, UBI is simply the cheaper option. Historically speaking, post-massacre recovery has always ended up costing much more.
>It's a structure where everyone has to rely on a specific corporation's products to get anything done, meaning the wealth will automatically be absorbed back to them anyway. So why would they go out of their way to commit a massacre?
Isn't it obvious? Your existence is margin that they leave on the table. And for what reason if you have no real skills or anything to offer? You are a leech. You exist to consume. You create nothing in this new future. You have no value. You aren't even an asset. You are a liability.
And what you think they will fear instability that we will bring upon? We will be armed with sharpened sticks and a smattering of ar15s at best case, while we are targeted from orbit by automated precision ordinance. Imagine a war between two AI controlled nation states. One has to manage a population during this war, the other opted to cull its population early in the war and maximize all its resources into the war effort. Guess who wins that war every time and likely takes control of the planet.
>But if you provide just the bare minimum to survive through UBI, the capitalists will gain a loyal faction that will act as their Red Guards to attack dissidents, which I think is much cheaper in the long run.
Automatic gun turrets are even cheaper in the long run.
Every bit of money that circulates is absorbed by every part of the economy. For instance, 100% of money is used to buy cheese and flows to sellers of cheese.
Me too. Here is some history to back up how absurd the statement is.
In 1930, John Maynard Keynes wrote an essay title "Economic Possibilities for Our Grandchildren"
He predicted that in 2030, technology, productivity, capitalism would solve the economy. There would be so little work to do and so many resources that the average work week could just be 15 hours!
He wrote that once all us plebes were liberated from work, what we would face is humanities greatest problem. What purpose will we have with all this new found leisure time!
The user base on Hacker News has really gone down hill. Voting down my comment above has no basis in the guidelines or the strictly information content of post, and hence, I have to conclude people just vote down opinions they do not like.
I propose that voting on comments should have a required field addendum, so that before a vote can be cast, the voter has to type into the field the reason they are voting it up or voting it down.
I see things differently. This is a change unlike anything in the past. Back then, it didn't replace human intellectual activity. However, problems of increasing scale will continue to arise, and inevitably, issues will emerge that are too difficult to solve through the efforts of human communities alone.
The performance gap between free and paid AI models is massive. The resulting divide between children from low-income and high-income families is quite evident based on my observations in my country. It will only get harder to keep up.
I don't support UBI either. The fact that UBI is even being discussed means the market is broken. The problem is that in the current market, prices for freelance work tied to intellectual labor have been absolutely crushed by AI. Looking at indicators from Booth or the Korean freelance market, "middle-tier" jobs are disappearing. While currently established professionals remain unaffected for now, the volume of work has drastically decreased for developers or writers who are still building their reputations. There are statistics from Japan showing this as well. My workload has also decreased to a ridiculous extent. The so-called barbell structure is becoming entrenched.
The core issue is this: if that happens, how will society handle the dissatisfaction of such a massive number of people? If more than half the population are workers, there is no problem, but if more than half are unemployed, that is a serious issue. Income polarization is continuing to accelerate.
Because there are no aristocrats without slaves, capitalists need workers to elevate themselves. Wealth and power are strictly relative, and they need miserable ornaments to make themselves feel like "awe-inspiring beings." To confirm their own sense of superiority, capitalists require human spectators who look up to them while suffering in misery. Why else did ancient Rome unleash lions and gladiators in the Colosseum?
Later on, things made by humans will become expensive luxuries, while commoners like me will be fed UBI—the bare minimum bread for survival—along with cheap, AIslop.
If making money by learning a skill increasingly requires a longer time investment, you have to endure the period it takes to learn that skill and build a reputation, which is incredibly difficult for the average working class. It requires the financial backing of a wealthy family to afford a college degree, travel to academic conferences, and survive on low wages. And if someone can endure all that to build a reputation, it will naturally lead to a massive concentration of wealth at the very top.
Let me clarify one point I was trying to make. It is my belief nobody is going to give you UBI. I offered up Keynes prediction as a little bit of evidence about human nature. The United States could have easily built infrastructure and safety nets to feed all of its citizens. It could have easily provided a much improved welfare system, universal health care, and other social services. But it did not.
>The core issue is this: if that happens, how will society handle the dissatisfaction of such a massive number of people?
Looking through history books: not well. Not well at all. But with the course of things, it seems inevitable unless new leadership comes in and takes a huge 180 on policy.
The plutonomy transition is well under way. The sardinian seafront villa will likely retain and gain value as the early anthropic employee might like it, while the house in some flyover state might not find any buyers.
The economy of the future will be about yachts, supercars, hermes bags, private air travel and ozempic. Not about how you made it possible for everyone to watch a gigantic library of videos for free, or how you made cars affordable for everyone.
I call this "brand mining." Take a brand previously known for quality, start selling shite, and then you get to pocket the delta until the market catches on. The brand is now worthless (you've "mined" it out) but hey look at all the money you made.
Basically every 90s brand has had this happen. Typically involves PE.
> Take a brand previously known for quality, start selling shite, and then you get to pocket the delta until the market catches on
Similarly how every former reputable consumer electronic brand from 'The West' that sells consumer stuff today like Philips, GE, Thompson, Blaupunkt, Gigaset, etc is now just a brand label for a Chinese consortium that's fully designed and made in China with zero employees in their original western roots.
Goodwill liquidation is what I’ve been calling it.
I think we might be coloring it as we’d like it to be, though. Only rarely does it end in brand death, more often they just get a bad reputation then keep going, sometimes not even less profitably. I think the intent might be more along the lines of recalibrating to a lower expectation of consumer discernment or rationality.
Havent bought a Craftsman tool in about that many years. I buy Harbor Freight tools now, if I'm going to buy a cheap made-in-China tool at least I'll buy it from a place that makes no secret about it and charges about half the price.
Messrs. Sears and Roebuck are rolling in their graves.
It still has a pretty big delta, so much so SBD bought the name for nearly a billion dollars.
I think a lot of people realize it's not the same old company, but apparently enough don't such that it's still profitable to sell their stuff at a premium.
I don't think thats what happened in this case. They explained it in the article. Honda was way behind in developing EVs, and as a "prologue" to their eventual EV strategy, they released a car called the Honda Prologue, which they licensed GM EV technology for. (fun fact, after Trump and the EV numbers that didn't materialize in the US, I think Honda has either delayed or scrapped a lot of their EV plans, which is unfortunate. )
Kinda like how Toyota just ignored turbos and pure EVs and instead sold cars to rich people (because rich people don't spew clapped out examples onto the used market) and now they're making money hand over fist on their reputation despite not having a competitive EV offering or being able to keep a turbo engine together but they'll still make money hand over fist for a decade because that's how long it'll take for consumers who've been astro-turfe'd at for years to realize that not everything is doctor owned, dealer serviced 2004 Solara.
"There aren’t a lot of ways a website can be invisibly broken."
??? Have you never debugged weird react state before??? Have you never used a nontrivial SPA before? Even the most simple react SPA has about a trillion states.
I don't really know how to respond to your statement than "no, they can definitely be invisibly broken."
React state bugs are usually not invisible failure modes.
Other kinds of software - for example, the backend of a website - have far more invisible failure modes. Security. Correctness. Performance. Bad API design leading to overfetching. Etc. Software works great on your machine and when you demo it, but it falls apart at scale. Endpoints that aren't secured properly. Race conditions cause silent data corruption. Memory leaks. And so on.
If claude messes up writing a react frontend, we're more likely to find the problem quickly. And frontend bugs usually don't continue to cause problems after they've been fixed. Silent bugs in the backend are more dangerous and more expensive.
"Note I’m not saying there are zero risks: the agent could mess up accessibility, it could cause an infinite loop that blocks users, etc. But in general, frontend code is a lot more ephemeral and replaceable than other types of code. So I expect many AI coders will feel comfortable just letting their agent handle it unsupervised (for better or worse)."
This is a weirdly reductive take on frontend correctness. Just for the record, I'm a backend dev. So I don't have much stake in this game.
This idea is, of course, not uncommon. "If the backend has to treat the frontend as adversarial anyway, and has all this cool stuff (constraints etc) for guaranteeing consistency of the system, then the frontend can just do whatever, right?" It plays into a lot of biases around typical frontend devs, typical backend devs, language stereotypes, etc. So it _sounds_ good.
Let me tell you for a moment about one of the spookiest bugs I've seen. It was an app for sorting personal photos. You'd upload pics/vids off your phone, they appear in the UI, you click a folder for them to go into (or click delete to discard), etc. Simple app, right?
Well, naturally, pics from even vaguely modern phones are regularly 5MB or more. Not really something you want to sling around while a user is browsing and their main activity is going to be looking at said pic to decide what folder it goes in (or if it gets deleted). So we thumbnail. And the main app only ever shows the user the thumbnails. The backend organized things quite simply: it gets a list of images from the frontend, it assigns each of them a zero-based index, and generates a thumbnail you'll also access via index. Imagine a URL scheme like `images/0` and `images/0/thumbnail` serving the real assets and the thumbnail.
Well, this app had a bug at one point. The backend was indexing by the arbitrary order the user uploaded them in. The frontend was mostly doing this too. Unfortunately the logic for thumbnails was incorrectly indexing by the "taken time" (which was a post-upload timestamp constructed by looking at basically every available timestamp and picking the "best" one. i.e. hopefully the one the iOS camera app adds, but obviously pics come from other places too and you never know what a user will upload). The end result being users would upload pics, see a thumbnail of an accidental pic they took of their shoe, hit delete. But actually they were deleting a pic of their baby or similar.
Literally none of the testing caught this for 2 main reasons: headless tests don't look at images, and you can't write an assertion like ("does this image look like a downscale of this other image") (at least not easily... i guess image models could do it now? but probabilistic? not a word i like in my unit tests? I digress, this predated the current crop of "AI").
Now let me generalize: your frontend isn't just a weird way to call RPCs on your backend. It's part of the application. I don't think you can just hand-wave. And as we saw above, you can't even say "well the frontend is stateless! any bug is 1 deploy away from fixing!" - deploying the frontend didn't get anyone their baby pictures back.
> you can't write an assertion like ("does this image look like a downscale of this other image"
You can though. Maybe not exactly what you are thinking but there are some pretty good image similarity algorithms like dhash out there that do stuff like this. Mostly they get used to check for duplicate uploads, copyright materials, or "does this look like porn" style filters
This isn't something that needs image models to accomplish really.
The actual problem is that no-one had considered that this (desync between front and backend) COULD happen. Once it was realized that it was possible (and worth testing) the bug was practically solved already.
Such is true for many bugs. Even if you have unit tests, they only test on things you've thought of. It's usually the stuff you haven't thought of that gets you.
No reason not to unit test of course, but don't get a false sense of complacency or assume testing is easy either. That's why it's great to do things very carefully (and probably not with AI Agents).
This same level of "oops" in the backend causes users to accidentally see or delete other user's photos, which is definitely a categorically worse scenario.
Devs switch because all of the magic of "AI" is LLMs + we stole the entire written output of the entire human species, with some remaining % coming from various tricks we've learned over the last 3 years like thinking and agent-toolcall loops, which weren't hard for literally everyone to copy. One of the consequences of this is that the only thing that really distinguishes anthropic from kimi is that they have the entire US VC market funding them because they promised to finally put white collar labor in its place.
All I mean is that none of the frontier LLMs are significantly better than the others for the vast majority of work, so devs are free to move between providers freely.
> stole the entire written output of the entire human species
Devs used to love public domain works, hated copyright, vehemently opposed software patents, held the pirates side during the MP3 wars, are very supportive of ThePirateBay and insist the correct term is "copyright infringement", not "stealing", and that "stealing" is a egregious form of PR brainwash from the media industry to inflate the scale of the crime
> Copyright holders frequently refer to copyright infringement as theft, "although such misuse has been rejected by legislatures and courts". The slogan "Piracy is theft" was used beginning in the 1980s, and is still being used.
I don't think regular people did any 180 degree turns. Rather big corporations again demonstrated their total hypocrisy where they on one hand stomp on people for "infringing their intellectual property" while at the same time scrapping all data they can get their tendrils on, with total disregards for the wishes of the authors of the data. Even to the point of overloading servers by mindless scrapping or destroying irreplaceable physical books like the bloody inquisition!
And all that to basically sell it back to people when their "AI" regurgitates it back.
It has nothing to do with "their work", and the opposition to AI copyright usage comes much more from non-devs than devs. The actual answer is that people are anti-megacorporation and pro-individual. They are fine with copyright that protects individual rights and opposed to copyright that enshrines megacorporation rights. It's not that difficult to understand.
Since we are talking for other people, "devs" wouldn't have supported someone downloading all those copyrighted works from the pirate bay and profited by, for example, selling them. I don't see any 180.
This is a subtle distinction; I'm not surprised many miss this, especially people who can't _not_ anthropomorphize the LLMs.
reply