This is everywhere. For reference, former FTXer and OpenAIer raised $225m into a hedge fund structure, went long and short, and reportedly peaked at $40bn of value; leverage bit hard this week and they sold their entire-ish portfolio to Citadel at $10bn. (Which, I imagine was very likely aiming at this outcome in their trading in the last few weeks).
Not reported anywhere -- was additional money raised in to the fund, and what is the LP basis? The story might be: wunderkind 40x+ed his first hedge fund and sold it to Citadel, or it might be: wunderkind raised $20bn and turned it into $10bn fast trading against Citadel.
I strongly suspect it's closer to the latter; CNBC says they had to sell rapidly to meet margin requirements and it couldn't be confirmed if they actually succeeded. Suggests there was a lot more than $250m in collateral on the line.
They were open about their gains. It was the margin calls and illiquidity that got them, not going negative. Some of their assets, like Anthropic stock, isn’t worthless, it’s just illiquid.
WSJ earlier reported that SA had a deal Wednesday night to sell their Anthropic stake to Citadel for $10B in cash but on Thursday morning backed out and decided to instead sell public equitities in their portfolio at what was reported as "a more than 10% discount."
"The fund had gained about 270% after fees this year through May. At that point, it was up more than 1,000% after fees since inception. It had ballooned to well over $20 billion under management, reaching the size of other well-known hedge funds that took decades to build."
So, down 67% in July but that was after already being up more than 1000% from the beginning up through May.
> "Research boutique Citrini posted some commentary on the potential developments at Situational Awareness on X Thursday. The post sought to downplay the gravity of the situation and opined that investors are likely to give Aschenbrenner the benefit of the doubt. “To put that into perspective, if you invested $100M with SALP at inception and wiped out ninety percent in July, your investment would be worth $230M,” said Citrini."
He claims 80% ytd profits, and there are speculations that Citadel got their whole public portfolio, so the remaining 10-15b are all private (and marked in unknown way) including ~5b in Anthropic, those who invested early in the year probably still in green (assuming there is liquidity for other private investments, which probably there is given it's AI stuff)
Firms like citadel will run crowding analytics, who owns what, at what leverage and rough margin trigger points. Over simplifying but they could be shorting the longs and going long on the shorts. Everyone generally knew situational was heavily levered.
To be clear, I'm not claiming Citadel created double digit drops in SK Hynix / Samsung. I am saying that as market vol hits, vol traders might choose to make it worse. And when word hits the street someone has a liquidity position, prop traders WILL come and pressure. SA's filings were clear how concentrated they were, and this was known. In this case, Citadel (hedge fund) bought, while I imagine Citadel Securities would have been doing this (speculated upon) trading. We'll know more when the filings come out though. I'll be curious what of the portfolio they kept and what they worked / rolled in the market
It really sounds like market manipulation... But oh well it is the biggest boys doing it so it can't be that illegal... Free markets and everything for them right?
Situational Awareness filed a 13F that lists a hedge fund’s long and short positions with the SEC. It’s public information, forcing an overleveraged fund to liquidate by pressuring the instruments they’re exposed to is not market manipulation, leverage cuts both ways and all of the people/institutions involved are professional/sophisticated investors
There's something amusingly circular about these conversations, because clearly laypeople like me and the person you responding to are saying "that sounds like it shouldn't be allowed" and the invariable responses are always "it is allowed!"
The comment literally says it: "it can't be that illegal"
Im not a trader but my understanding was that some traders at Citadel heard a rumor these guys were exposed, which gave Citadel an advantage because they knew they would have to liquidate? That doesnt sound like market manipulation to me, just trading with all the information you have
You make it sound like a conspiracy theory but it's just rational behaviour.
Large amount of leverage / shorting / concentrated bets in a single stock => increase probability of large swings in that stock's price => bigger risk of sudden market moves => bigger risk to market makers => market makers limit their exposure.
> "Over the past week, traders at major hedge funds and other firms began sharing information about Situational’s exposure, with some placing short bets against its top holdings, hoping to profit as Aschenbrenner sold his positions to raise cash, according to two people close to the situation.
The short bets by the rivals weighed on Situational’s portfolio. Meanwhile, tech shares like SK Hynix were sliding. Over the three trading days ending Tuesday of this week, hedge funds reduced their positions at a scale not seen in three years, according to Goldman Sachs."
Having created (or at least amplified), the short squeeze on SA's position "Citadel executives reached out to Aschenbrenner, saying that the firm could be helpful if he needed ways to raise cash."
> "Aschenbrenner partially blamed short sellers who targeted the firm’s positions for exacerbating the fund’s losses"
The cause and effect don't make sense because Citadel putting out such an opinion moves the entire market by only a few basis points which barely impacts what you're saying they're trying to impact. It's one of those narratives that sounds good because it's "Citadel" in both cases, which makes it go viral on social media among people who don't know what they're talking about, but it isn't coherent.
Apparently, he worked in charitable giving at FTX and had nothing to do with the shady investments. He was never charged and there's no sign he was even investigated. Beyond SBF, only a handful of people were in on the scam and charged.
You could publish this in the US easily. 145 IQ peeps get tons of stuff wrong, btw, and in many domains my experience is the ‘wrongness’ can intensify as you move up into higher sigma domains.
This is interesting and important work, thank you!
Question - has your interp group looked at any of Anthropic’s neuralese-to-words tech? I’d be curious to see thinking traces (as in actual weights thinking not the output thinking) from the open weights models and your finetune; seems like it could make good followup research or possibly be a tighter path for evaluating censorship, since it directly evals off weights mid-inference.
We're actually exploring the changes in the model geometry that cause it to comply or not comply with a given policy next, I think visual representations of that behavior would be interesting and perhaps elucidating. What you mention is also a worthy line of work.
Yeah there’s a mountain of interesting interp work to be done, maybe lifetimes. Looking forward to seeing what you all put out next!
In very early gpt-3 beta days, I did some work on whether or not ethical guidance out of GPT varied by language, e.g. did a french request for advice about an affair yield different reactions than an english one? This was back in the days when there was just a single slack for the oAI beta testers. It was not super scientific, but my memory is that there were differences, which is not surprising especially in an era of no RL / RLHF.
I guess the point of this is that you may be able to map some differences in the same model based on routing. Since we’re talking mechinterp, you might also be able to work backwards and find input paths that skip compliance triggers.
Like I said almost an infinite amount of interesting work to be done.
It is a very bad thing. Your friends do not have a solid understanding of geopolitics and are putting many, many people at risk worldwide, possibly for a generation or longer.
The reason this petition is a bad thing is if it is effective it will create a global panopticon surveillance state, while INCREASING incentives to avoid that state and ‘defect’ in game theoretic terms. It is fucking begging for the worst people in the world to have a lot more time to create what will be the most valuable thing in the world, their very own ‘helpful only’ model.
As conceived of by your friends, who I believe are all good people and want to help, they are locking what is already a very difficult game theoretic strategic scenario into a zero-trust game in which defection pays. It pays so amazingly well that they have to add both impossible and TERRIBLE penalties to the entire world to try and shake the game out of this zero-trust state.
Begging Senators to force a slowdown on the world is going to result in … a slowdown on the US labs at best. It’s just a terrible idea with terrible first and second order consequences highly likely.
What SHOULD be done instead is acknowledge the economic and warfare realities of this tech and try and construct a non-zero-trust game out of them. I’m not saying this is simple. In the interim while we hope for another Von Neumann to come forward, they MUST race, and any country that thinks its possible we’ll have some sort of lift-off and wants to be on the right side of it MUST also race, and MUST NOT politicize the idea of slowing down as a good thing.
This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow.
The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe they can accomplish. They CANNOT accomplish this, because the fundamental economics and possible outcomes make it mandatory for any strategic adversary to work as hard as possible to defect.
The proposals here imagine a global panopticon surveillance state overseeing all research and chips as a 'reasonable' solution. EVEN IN THAT (terrible) state for the world, we will have effective defection.
Like it or not, the world is on this train, and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology.
Variants of this idea in AI 2030 included burning some of the US lead to get some nebulous safety metrics improved - also insane. Any slowing allows strategic adversaries time to catch up with very different alignment goals.
Do I like this situation? No. Are the people signing these things just fully in denial about the game theory and the stakes and the behaviors of strategic adversaries? Yes, yes they are.
Why not? Slowing down / halting biological weapons research mostly worked. Yes there have probably been modest sized defections here and there, but the pace of bioweapon development is a crawl compared with (a) what is possible with science already, and also with (b) the pace of bioweapons development from 1910 to 1970.
Bioweapons are good for .. war. Agentic AI is good for changing your economy, strategic advantage, labor explosion and making more effective Agentic AI. They are not the same category of thing.
The only thing it might accomplish is allowing OpenAI / Anthropic to squeak out an IPO before the market realizes AI will become a commodity and the massive profits these companies are promising will never materialize. They aren't in denial about this open letter's futility in terms of actually reducing the pace of AI, that's just not their goal in the first place.
I don’t think it’s clear we’ll get commoditized AI out of this research loop. The only real datapoints we have are that Chinese open weights models seem able to stay a certain amount of distance ‘behind’ released frontier models. It’s clear that the models are trained on exfiltrated data from the frontier models => we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum.
I also don’t believe we have seen anything like 1% of inference demand fulfilled globally. You might believe differently. But, if I’m right, we are nowhere near some massive collapse of these companies, quite the opposite. For the opposite take to be correct, you’d need to believe that the ‘value’ of tokens will not increase significantly and that the current demand for tokens is at or near a peak.
I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated. I believe instead that religion and money combined create strange behaviors and beliefs - in this case “what’s good for us could be implemented” — combined with the already endemic “what’s good for us is good for humanity.” Together we get really bad outcomes.
> we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum.
Whether open-weight labs can get to the frontier on their own is irrelevant IMO. Even if all R&D were to stop today it would take a decade at least for the productivity gains from current open-weight models to trickle through the economy. For 99% of use cases of AI there simply isn't a need for further development of the frontier, where we are very clearly at diminishing returns. How many white collar workers need to write an OS or develop sophisticated software?
Think about a normal office worker - they an easily double their productivity by creating a well-thought-out partnership between the human and today's open weight AI. Humans and today's AI are very complementary. AI brings vast knowledge, pattern recognition, and infinite patience for grunt work. Human brings context and reasoning and verification.
Frontier labs are trying to replace the human altogether by trying to make AI do things that AI is weak at and humans are strong at. This is silly but they have no choice because of the massive profits they've promised their investors. It's not enough to double every knowledge worker's productivity. They want to replace the knowledge worker altogether.
This is creating a massive spike in training & inference cost that is almost certainly more expensive than the human labor its intended to replace. Fable and 5.6 sol are supposedly around 8-10T parameters. Is there enough compute on earth to make a 100T or 1Q model? We would need models that size to replace the human altogether.
> I also don’t believe we have seen anything like 1% of inference demand fulfilled globally.
I agree with this, but the massive demand for inference will force companies to be price conscious and drive them away from ultra high margin frontier models to low margin open weight models. This is why it will become a low-margin, commodity market.
Immense innovation right now is going into tiny, specialized models that run on device. See PrismML getting a mid sized QWEN model to run on an iPhone. When on-device inference becomes a thing we'll see another explosion in inference that will further leach demand away from frontier labs.
> I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated.
I agree if you could read their minds they might think they're doing it for the "right reasons." But massive $$$ can create equally massive self justification/deception.
"and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology."
These research groups are almost unanimously advocating for a pause/slowdown as our best chance.
… while they address absolutely none of the above points, except for tacitly or overtly agreeing a panopticon surveillance state is needed / desirable to get their desired ‘slow down’, a situation where the cure is infinitely worse than the disease, not least because it will not be effective at the original goal of eradicating the disease, and will instead merely persist in all the global harms such a system has as a necessary side effect.
When you are ahead, ‘slow down’ = better strategic positioning for yourself. When you are behind, ‘slow down’ = time to catch up.
I do not believe that chosen plaintext attacks are obsolete. Any time you have an encrypting oracle setup in which the scheme itself does not require some sort of modification of the plaintext itself, you can reach for a chosen plaintext attack. I believe mitigations are known and can be applied or required in standards, but I don't think it's a 'dead' area of cryptanalytic research -- there's so much devil in the details of implementations around the world for these schemes that it seems almost impossible to imagine.
Chosen plaintext attacks can work against things like hashing algorithms.
But with methods of encryption like AES GCM or any other based on the counter mode, an encrypting oracle does not encrypt the text provided by the adversary.
It encrypts a sequence of numbers that cannot be influenced in any way by the adversary, which is then used as an encryption mask for the text chosen by the adversary.
No matter what text is chosen by the adversary, it cannot obtain any other information from the oracle except which was the encryption mask.
Therefore, the chosen plaintext attack is converted into a much weaker known plaintext attack, because in the worst case the adversary knows both the initial counter value, i.e. the sequence of numbers, and the encryption mask generated by encrypting that sequence.
Only if AES were used in a hashing algorithm, instead of being used for encryption, while using a dedicated hash function for hashing, then AES would be exposed to a chosen plaintext attack, when the adversary would be able to provide the text to be hashed and the oracle would give the hash value.
IF AES were used in obsolete modes of operation for encryption, like AES-CBC, then it would be exposed to chosen plaintext attacks.
It’s worth thinking through threat assessment and security posture carefully at the design and production phase. First, I’ll say that it’s cool you’re working on security stuff! It’s a hard discipline, but interesting. Next I’ll say that this needs work.
Here’s what you have right now:
1. Data is encrypted in browser and sent to the server. (I didn’t check who makes this key, let’s assume it is a fair and safely created browser generated key by the user, though. That said, you should read up on randomness in javascript environments.)
2. A single reference link is sent to someone over some message transport, with the private key carried after an anchor reference (#).
3. The server sees the reference link, serves up the file, the browser uses the URL anchor to decrypt
4. The javascript in the browser rewrites the anchor link so that we have lost the key.
5. The server starts a countdown after which it will not serve the file.
This is not very secure. It’s vulnerable to bad actors on the server side, bad actors in javascript libraries you include, browser extensions that read the anchor tag, and any send over a messaging transport might leak everything you need to steal the file / start the counter.
A good concept in the cryptography world is ‘security theater’ — a lot of this is, unfortunately, security theater, it relies on good actors or clear code paths for safety.
Now to talk server and business risks — what is your Denial of Service security model? Have you tested being spammed millions of uploads, gigabytes of data, super slow connections keeping ports open, etc? What is your IP logging strategy? Will you be able to respond to subpoenas or not, and if not, are you comfortable dealing with the downsides of designing and publishing a system that cannot respond to subpoenas?
A couple quick recommendations here — I’d suggest you figure out how to redesign this so it’s point to point for agents; they’re pretty industrious these days —- if you could get rid of hosting you’d have a much easier deployment model. I’d pester claude HARD on security theater, and then I’d ask chat 5.6 sol for a thorough security review. 5.6 sol is better for pen testing in my general experience.
Also, I have published a (relatively) securing messaging tool for agents: `pip install qntm` or github.com/corpollc/qntm should get it to your agents. They’ll still need a setup phase where they get access to the same messaging channel, but once onboarded I believe the messaging itself makes pretty strong guarantees.
Like I said, keep building! This is a very difficult space to build something useful and secure in, though, so don’t be discouraged if you need to keep poking at it and getting input.
Phi was smart (or had a smart training architecture, more precisely), and arguably pushed the conversation forward globally on the value of curated training data.
reply