It turns out multi step backpropaganda is better.
paper has a beautiful way of improving backpropagation. One iteration cleanly gets us backprop, multiple iterations get us a preconditioned update.
rohan anil @_arohan_
513240180K
rohan anil✓
@_arohan_
co-founder @coreauto - and continuing to aspire to understand deep learning.
2.4K Volgend46.3K Volgers11.6K Berichten
- Berichten @LinYorker0587180K
- I really love the full bandwidth transformer research - One of the beautiful works of art of this year! “For example, the Bytedance experiments suggest a variant of the Full Bandwidth Transformer might be comparable to a model of twice the size, implying significant savings in training and inference for the same capability. Do the math—how much is it worth to halve training and inference cost given that the industry has investments approaching a trillion dollars?”John Langford @JohnCLangfordThe End of AI Frontiers . There's a bit of a train wreck here that's painful to express. Nevertheless, it's a significant event for me and quite a few other folks involved so I wanted to memorialize a bit.13442.2K
John Langford@JohnCLangfordThe End of AI Frontiers . There's a bit of a train wreck here that's painful to express. Nevertheless, it's a significant event for me and quite a few other folks involved so I wanted to memorialize a bit.151621627.6K- rohan anilRT @BEBischof: A rat can run a restaurant, but can an agent run a bakery? Welcome to The Great Business Bake Off: our AI Agent Hackathon. On December 11, you and your agents get five hours to make the most profitable bakery in town. Spots are limited:0200159
- rohan anilRT @MTSlive: SITUATION DETECTED: OpenAI has launched GPT-6 Cyber, its most advanced cybersecurity model, through Project Daybreak.0100178
- rohan anilRT @MTSlive: What if training frontier AI no longer required thousands of GPUs in the same data center? Core Automation cofounder @_arohan_ explains how distributed "compute islands" could make that possible: "Either you have compute to train a frontier model or you don't. The top two models matter, and everything else has a very limited shelf life." "Accruing a lot of compute is not just money. It's co-location, it's data centers, it's power. But we were getting hundreds or thousands of GB300s and B300s at different locations." "If you have a training algorithm that can use them to get to frontier, you unlock compute wherever it is." "This is not new. People have been working on local SGD, federated learning. It's an emerging area of training on compute islands." "In our experiment we used two types of chips with completely different parallelism, slightly different numerics, completely different batch size, and we get similar or slightly better results on a very efficient frontier scaling law." @coreautorohan anil @_arohan_Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models. At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training. I hope you all enjoy this write up!🪐🛰️080213
- Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models. At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training. I hope you all enjoy this write up!🪐🛰️661331.6K180K
- I am ready to buy islands.MTS @MTSliveCore Automation's @_arohan_ explains why thousands of GPUs sit idle while frontier AI labs struggle to get enough compute: "The market doesn't look that efficient. It's much harder to get tens of thousands of Grace Blackwells than a few thousand. These chips are sitting idle till they get sold, because the people who demand a lot more are looking for co-location." "Do you want 10,000 Blackwells? Sure. Will you take 10,000 separated compute islands? Without this technology, I wouldn't. With this, yeah, for sure. Algorithmic pain we don't feel because we work on these algorithms day and night." "It could also look like multiple organizations coming together to build something bigger if they're aligned on the same utility function, because compute is spread out." @coreauto61618.3K
MTS@MTSliveCore Automation's @_arohan_ explains why thousands of GPUs sit idle while frontier AI labs struggle to get enough compute: "The market doesn't look that efficient. It's much harder to get tens of thousands of Grace Blackwells than a few thousand. These chips are sitting idle till they get sold, because the people who demand a lot more are looking for co-location." "Do you want 10,000 Blackwells? Sure. Will you take 10,000 separated compute islands? Without this technology, I wouldn't. With this, yeah, for sure. Algorithmic pain we don't feel because we work on these algorithms day and night." "It could also look like multiple organizations coming together to build something bigger if they're aligned on the same utility function, because compute is spread out." @coreauto154615.7K- Ok I went and talked about this post and also met these cool people at MTS office. Very intense situations being monitoredrohan anil @_arohan_I am joining MTS about the situation on training on compute islands due to the compute situation.20343.7K
- I am joining MTS about the situation on training on compute islands due to the compute situation.MTS @MTSliveMIDTERMS WRAPPED | FAL, BLOCK, CORE AUTO ON MTS12388.1K
- Berichten @_arohan_Coffee there is better than one at coreauto office so I need to fix this20101.3K
- Congrats @dvsaisurya on your successful defense! Its been a pleasure of my life to collaborate with you! @inderjit_ml @cho_jui_hsieh @Devvrit_Khatri
1541098.6K - This looks incredible on first look, due to the sheer scaleAxiom @axiommathaiMost think vibecoding at scale is slop. Here’s a counterexample. Today we release a 16M-line formalization of the entire Stacks Project, critical infra underlying modern math, with optimized code quality and its highly interconnected structure preserved. A mathematical city 🌆278210.4K
Most think vibecoding at scale is slop. Here’s a counterexample. Today we release a 16M-line formalization of the entire Stacks Project, critical infra underlying modern math, with optimized code quality and its highly interconnected structure preserved. A mathematical city 🌆8461.6K7.1K6.9M- You can train to frontier today by cobbling together <20 MW data centers. I think there is no moat for training, deep learning finds a way.Jessica Lessin @JessicalessinThe fact the market is up right now means investors don't think the data center bans are going to fundamentally delay the big hyperscalers on infrastructure. That's interesting.8514012.2K
The fact the market is up right now means investors don't think the data center bans are going to fundamentally delay the big hyperscalers on infrastructure. That's interesting.1937035.5K- Berichten @_arohan_I think only moat is space if such a thing is possible. I don’t know.0061.2K
- Datacenter voting results today makes me want me to post my video about really distributed training.
20533.1K - I legit read this as Ram from RamanayaSam Altman @samaPeople should always be in control of the future. OpenAI is against the notion that it would be acceptable for people to be relegated to a role of doing things that “AI systems think makes sense to reward in humans”. People should not cede control of the future to superintelligence, and that requirement should ultimately constrain AI progress.12324110.3K
People should always be in control of the future. OpenAI is against the notion that it would be acceptable for people to be relegated to a role of doing things that “AI systems think makes sense to reward in humans”. People should not cede control of the future to superintelligence, and that requirement should ultimately constrain AI progress.5.5K1K20.4K10.5M- Few easter eggs on this blog, - you can click on the cats - hover over the globe to reveal ocean life and tiles - also Elon’s tweet quoterohan anil @_arohan_Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models. At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training. I hope you all enjoy this write up!🪐🛰️20524.6K
- I made my first music video about distributed training. People want this to stay at coreauto, so I am giving you screen grabs.
rohan anil @_arohan_Tonight I am becoming a Claude Slopcore artist.120644.2K - Meer berichten tonen
