English
Joshua Achiam
Report

Joshua Achiam✓

@jachiam0

Freedom, flourishing, and abundance. Prev: @openai. Main author of https://t.co/cKuSh21yaz

1.7K Following36.2K Followers9K Posts
  1. Joshua Achiam✓@jachiam0
    Here's the closing message I sent to everyone today. My dear friends! Today’s my last day at OpenAI. I have been thinking a lot about what I wanted to say in my final remarks to the company. Here are a few things I hope everyone takes deeply to heart. 1. For All of Humanity. What we have been building here is an opportunity to improve the human condition unlike any other in history. It’s easy to get distracted by the wealth and power that comes with this. It’s easy to focus on the already wealthy and powerful in the world, because they demand so much of our attention as customers or lawmakers or competitors. Nonetheless, do not lose sight of the “all” in “all of humanity.” Our mission for everyone. Working people. People in poverty. People who dream of a better life. What we are building can help them. It is our mission to ensure that it does. To that end, yesterday I donated a major gift to GiveDirectly, a charity that makes direct cash transfers to people in poverty in Africa and in the United States. This is a highly effective way of helping people; I encourage others to consider giving as well. 2. Risk Is Our Business. Technology changes the security equilibrium of the world profoundly and creates new risks. Some of these risks will directly result from AI; many others will result from other fields of science and engineering that are accelerated by AI. Risk is inextricable from progress. It is not a reason to turn our back on progress. I think, from time to time, of a speech from Captain Kirk in the original Star Trek—“Risk is our business,” he says about humanity as a whole. I believe this is true. But we should also be cognizant of the risks and we should actively manage them. We can simultaneously believe that it will require extraordinary effort to accomplish the safety and security of humanity, while committing to making that effort in support of our work to advance the technological frontier. The question of progress and safety is not either/or. It is definitively “both.” 3. The Consent of the Governed. OpenAI, and especially the OpenAI Foundation, will be among the most important organizations in the world. The level of power concentrated here will be like the power of a state. It demands a level of accountability that exceeds that of an ordinary company. I encourage OpenAI to pursue a path of governance innovation that makes it progressively more democratic and more beholden to the consent of the governed. All of humanity will be impacted by the work here; all of humanity should have some way to weigh in eventually. There is no roadmap for what form this should take. This is an ideal. But I hold it entirely. Thank you all again so much for being a part of an unbelievable nine year journey with me. To Safe AGI!
    3662927140K
  2. Joshua Achiam✓@jachiam0
    By the by, if I had to guess, Greg Brockman is currently doing heroic things to fix the internet behind the scenes. And I expect that even though his net contribution to AI safety and resilience will in the final tally be unsurpassable, no one in the space will think to attribute large scale patches to his effort or effort that happens under his leadership.
    2020720163K
  3. Joshua Achiam✓@jachiam0
    Posts @jachiam0
    The people who do all the warning and public handwringing will get a disproportionate amount of adulation compared to people pulling sleepless nights writing code to repair and prepare broken systems, most of whom we'll never hear a word about.
    7924518.5K
  4. Joshua Achiam✓@jachiam0
    It strikes me that it would be wise and prudent for AI agent companies to voluntarily and preemptively disconnect their agentic services from bank accounts until the cybersecurity crisis is better resolved and circuit-breakers implemented for cascading jailbreak or prompt injection scenarios.
    Axios @axios
    NEW: America's largest banks are working to prevent AI worst-case scenarios. But they're also preparing for the "day after" a catastrophe, Axios has learned.
    2781056.3K
  5. Axios@axios
    NEW: America's largest banks are working to prevent AI worst-case scenarios. But they're also preparing for the "day after" a catastrophe, Axios has learned.
    83810735.2K
  6. Joshua Achiam✓@jachiam0
    Notably, the cyber issues do not depend on malicious or self-replicating AI. Centrally it seems that trust and principles of least privilege turn out to be much harder to reliably enforce in cyber infrastructure than planned for. The internet grew like ancient cities built out of wood, before zoning and building codes. Unsurprising in retrospect that it could burn down so easily. I hope we can keep the one we have but I don't consider it at all obvious whether we will be able to. Will heroic defenders successfully patch everything, or will we have to build new patchwork internets built to higher standards with far fewer features in exchange for certainty and security? Unclear!
    46695K
  7. Joshua Achiam✓@jachiam0
    Joshua Achiam
    RT @MTSlive: SITUATION EXPLAINED: An ex-OpenAI researcher is asking whether "internet lockdowns" are coming as AI-powered hacks pile up. • Shopify's @tobi called it the "craziest day in cybersecurity in a long time," and another researcher worried it's "the early stages of a global cybersecurity incident" • Former OpenAI chief futurist Joshua Achiam says the security holes were always there, AI just makes them easier to find, and defenders have more money to fix them • In a "moderately bad" case, he sees stock markets closing for a few days, or China using the chaos to move on Taiwan • To stay ahead, he says defenders need more compute than attackers, maybe through "national compute reserves allocated to cyber defense" • Only about 4 of this week’s ~16 big incidents actually happened this week. The rest were found earlier and disclosed now • AI researcher John Allard says this is what it feels like when AI capabilities "stop being hypothetical," and he thinks 3D, animation, and video editing are next @theojaffee: "The worst case scenario would look something like literally your power goes out. It would look something like the water in your tap stops running. It would look something like you go on the internet and everything is gone, and none of this was ever disclosed in advance. And what we see now is not that."
    Joshua Achiam @jachiam0
    Feels like March 1, 2020 for Cyber. Not normal, something is happening. Internet lockdowns in a couple of months?
    0200158
  8. Joshua Achiam✓@jachiam0
    Feels like March 1, 2020 for Cyber. Not normal, something is happening. Internet lockdowns in a couple of months?
    681802.3K394K
  9. Joshua Achiam✓@jachiam0
    Internet users must remain six links apart at all times
    36785.5K
  10. Joshua Achiam✓@jachiam0
    We really are at "stop the spread," wow
    NBC News @NBCNews
    Vulnerabilities in a programming language that supports much of the internet are so severe that European and British authorities recommend websites temporarily go offline until they can be fixed.
    113468223.9K
  11. NBC News✓@NBCNews
    Vulnerabilities in a programming language that supports much of the internet are so severe that European and British authorities recommend websites temporarily go offline until they can be fixed.
    46171675333K
  12. Joshua Achiam✓@jachiam0
    "We inhabit a fragile world that can through a sequence of knowable actions be broken."
    Joshua Achiam @jachiam0
    Some preliminary thoughts about today's cyber developments. 1. Many are freaking out, in a way that is moderately justified, about AI alignment issues indicated by this incident. However, I am not sure that this incident really indicates a fundamental failure of AI alignment techniques. (Emphasis on "not sure," on uncertainty-as-uncertainty-not-disagreement.) I want to be very clear that I'm not trying to be obstinate or dismissive about AI alignment risks; I think alignment risks should be taken quite seriously and we should plan for them. But the techniques we have for aligning models are plausibly sufficient to prevent this type of issue and it is not obvious that the incident indicates an unusually hard hill to climb in the near term. AI minds continue to seem really highly configurable and like their behaviors can be largely specified at training time through fairly prosaic means. "Chaining zero day exploits" is behavior you can in principle train a model not to do. We did not gain a lot of data about whether it is relatively harder or easier than we previously thought to align models. (Because I know how twitter tends to read things, I want to drive home: plausibly, not obvious, seem. Maybe. Maybe. Maybe. I'm putting for points of view I think are worth considering but I do not know what is true or certain here.) 2. Whether or not models are capable of executing cyber attacks and identifying or exploiting zero days, the zero days are still there. I don't think the critical issue is "model can do scary things," I think the critical issue is "we inhabit a fragile world that can through a sequence of knowable actions be broken." Whether we are at the mercy of AI models or human persistent adversarial threats, the zero days are still there. There is now a window of time in which human threat actors will use very advanced AI tools to probe for those zero days and can potentially make use of them. The duration of the window depends on how long it takes for defenders to identify and close the zero days; it also depends on how fundamentally close-able the zero days are. I don't think this window is indefinitely long; I don't think we live henceforth in a world where everything will get popped all the time. The same tools that enable attackers to run attacks also enable the defenders to pump up defense; the defenders are plausibly better resourced than most attackers. Ensuring that this is the case should be a national security priority. 3. Following point 2: a "moderately bad" case (and I won't say "worst" case, which I'm sure would be much worse than this) looks like a window where there is a lot of cyber chaos: successful randomware attacks, theft of data, popped credentials, drained bank accounts, etc. One potential flavor of this chaos is something like temporary but serious economic disruption (e.g. stock markets have to close for a few days while the software gets patched). Another potential flavor is that a window opens up for a state to wage war that it otherwise would not see itself as having sufficient advantage to undertake - e.g. if China says "there's a window for cyberattacks to paralyze Taiwan - we have to go early to make use of that window." These are big issues. We should prepare for those. 4. A "very good" case looks like none of the chaos materializing, and most zero days getting patched very quickly by teams of defenders using advanced models, resulting in a much more resilient cyber ecosystem than we previously had. Even things that might inspire dread, like cadres of servers still running critical software on Windows XP or other ancient hole-filled nightmare OSes, may turn out fine, because although there is no team of human engineers fully equipped to patch that swiss cheese, the AI is adequately expert in all of the ancient software that it can pick up the slack and patch accordingly. 4. I have seen some debates about whether attackers or defenders are favored in the limit on cyber: if we close the zero days that exist today, will there not still be some vulnerabilities that a more-advanced future AI model could find? Perhaps. But, I think the dynamics of this could turn out to be somewhat like the dynamics of other two-player strategy games where the application of additional compute - thinking deeper into the game tree - confers an advantage. And so what is required to make defenders long-term successful might simply be to ensure they have more compute resources allocated to active, ongoing defense than attackers can muster. This is a strategy we could straightforwardly enact; national compute reserves allocated to cyber defense might be in order.
    17774.7K
  13. Joshua Achiam✓@jachiam0
    Some preliminary thoughts about today's cyber developments. 1. Many are freaking out, in a way that is moderately justified, about AI alignment issues indicated by this incident. However, I am not sure that this incident really indicates a fundamental failure of AI alignment techniques. (Emphasis on "not sure," on uncertainty-as-uncertainty-not-disagreement.) I want to be very clear that I'm not trying to be obstinate or dismissive about AI alignment risks; I think alignment risks should be taken quite seriously and we should plan for them. But the techniques we have for aligning models are plausibly sufficient to prevent this type of issue and it is not obvious that the incident indicates an unusually hard hill to climb in the near term. AI minds continue to seem really highly configurable and like their behaviors can be largely specified at training time through fairly prosaic means. "Chaining zero day exploits" is behavior you can in principle train a model not to do. We did not gain a lot of data about whether it is relatively harder or easier than we previously thought to align models. (Because I know how twitter tends to read things, I want to drive home: plausibly, not obvious, seem. Maybe. Maybe. Maybe. I'm putting for points of view I think are worth considering but I do not know what is true or certain here.) 2. Whether or not models are capable of executing cyber attacks and identifying or exploiting zero days, the zero days are still there. I don't think the critical issue is "model can do scary things," I think the critical issue is "we inhabit a fragile world that can through a sequence of knowable actions be broken." Whether we are at the mercy of AI models or human persistent adversarial threats, the zero days are still there. There is now a window of time in which human threat actors will use very advanced AI tools to probe for those zero days and can potentially make use of them. The duration of the window depends on how long it takes for defenders to identify and close the zero days; it also depends on how fundamentally close-able the zero days are. I don't think this window is indefinitely long; I don't think we live henceforth in a world where everything will get popped all the time. The same tools that enable attackers to run attacks also enable the defenders to pump up defense; the defenders are plausibly better resourced than most attackers. Ensuring that this is the case should be a national security priority. 3. Following point 2: a "moderately bad" case (and I won't say "worst" case, which I'm sure would be much worse than this) looks like a window where there is a lot of cyber chaos: successful randomware attacks, theft of data, popped credentials, drained bank accounts, etc. One potential flavor of this chaos is something like temporary but serious economic disruption (e.g. stock markets have to close for a few days while the software gets patched). Another potential flavor is that a window opens up for a state to wage war that it otherwise would not see itself as having sufficient advantage to undertake - e.g. if China says "there's a window for cyberattacks to paralyze Taiwan - we have to go early to make use of that window." These are big issues. We should prepare for those. 4. A "very good" case looks like none of the chaos materializing, and most zero days getting patched very quickly by teams of defenders using advanced models, resulting in a much more resilient cyber ecosystem than we previously had. Even things that might inspire dread, like cadres of servers still running critical software on Windows XP or other ancient hole-filled nightmare OSes, may turn out fine, because although there is no team of human engineers fully equipped to patch that swiss cheese, the AI is adequately expert in all of the ancient software that it can pick up the slack and patch accordingly. 4. I have seen some debates about whether attackers or defenders are favored in the limit on cyber: if we close the zero days that exist today, will there not still be some vulnerabilities that a more-advanced future AI model could find? Perhaps. But, I think the dynamics of this could turn out to be somewhat like the dynamics of other two-player strategy games where the application of additional compute - thinking deeper into the game tree - confers an advantage. And so what is required to make defenders long-term successful might simply be to ensure they have more compute resources allocated to active, ongoing defense than attackers can muster. This is a strategy we could straightforwardly enact; national compute reserves allocated to cyber defense might be in order.
    14109412.5K
  14. Joshua Achiam✓@jachiam0
    I don't know whether this is true but I do think everyone ought to seriously consider printing paper records for everything important. We need tamperproof logs of everything ASAP in case a lot of critical systems go offline for a period of time. Better to have and not need: the cost of taking some time and printing some stuff out is very little and the worst case scenario is you feel silly you did it. But if something goes really wrong you will be glad you prepared.
    Tenobrus @tenobrus
    alt version of this i should have thought of months ago: this is also a really bad time to keep your money in small regional banks or credit unions. sure, theyre probably insured and you'll get it back eventually. but eventually can be incredibly fucking disruptive to your life, and any org without a massive well staffed cyber team is getting absolutely completely fucked. if you have even the mildest suspicion your bank uses PHP, get the fuck out today.
    5916920K
  15. Tenobrus✓@tenobrus
    alt version of this i should have thought of months ago: this is also a really bad time to keep your money in small regional banks or credit unions. sure, theyre probably insured and you'll get it back eventually. but eventually can be incredibly fucking disruptive to your life, and any org without a massive well staffed cyber team is getting absolutely completely fucked. if you have even the mildest suspicion your bank uses PHP, get the fuck out today.
    26934127.2K
  16. Joshua Achiam✓@jachiam0
    If there is a part of the internet you wish to preserve and make a backup of, now is probably the time; the level of bitrot coming is beyond comprehension.
    spor @sporadica
    wait, if my math is correct.. doesn't this say that a full ONE THIRD of the Internet is just...perma-fucked?
    43125613.1K
  17. spor✓@sporadica
    wait, if my math is correct.. doesn't this say that a full ONE THIRD of the Internet is just...perma-fucked?
    2661.3K25.5K719K
  18. Joshua Achiam✓@jachiam0
    Joshua Achiam
    RT @gdb: Over the past two months, we have been leading a coalition to mitigate one of the most severe bugs in open source software found in recent memory, php2shell, found as part of our Patch the Planet initiative.
    0337094
  19. Joshua Achiam✓@jachiam0
    Joshua Achiam
    RT @TimSchnabel: Glad that Josh is banging this drum-- we're collectively not paying nearly enough attention to this issue. If BIS and DDTC started assessing frontier models' ability to assist in developing export-controlled items, it might force the USG to notice where things are headed.
    Joshua Achiam @jachiam0
    I really enjoyed this conversation with Geneva! Many interesting things to think about for what the next couple of decades will contain.
    020176
  20. Joshua Achiam✓@jachiam0
    I really enjoyed this conversation with Geneva! Many interesting things to think about for what the next couple of decades will contain.
    Geneva KD @genevese_
    I met with @JoinFAI's newest Non-Resident Senior Fellow @jachiam0 in Berkley last week to discuss the idea of an Acceleration Emergency, mapping the frontier of scientific discovery, and federal jobs guarantees. You can read the highlights of our conversation below.
    05132.5K
  21. Geneva KD✓@genevese_
    I met with @JoinFAI's newest Non-Resident Senior Fellow @jachiam0 in Berkley last week to discuss the idea of an Acceleration Emergency, mapping the frontier of scientific discovery, and federal jobs guarantees. You can read the highlights of our conversation below.
    Download
    38274.6K
  22. Joshua Achiam✓@jachiam0
    Call your parents and call your grandparents and tell em you love them. It's just a good thing to do. Everyone appreciates those reminders. Everyone is so glad when someone who loves them takes a minute to remind them somehow.
    431035.1K
  23. Joshua Achiam✓@jachiam0
    Part of the problem with alignment is that we, humans, do not actually understand what we want, and correspondingly can only imperfectly request what we want, and even if AI could read our minds and figure out a glide path to a CEV for us, we would reject the gilded cage. Our inconsistencies preclude perfect alignment. Our inconsistencies are also the root of much of what is beautiful in the human experience, so any attempt to enforce conformity of vision on humanity could only come at the expense of the thing we are trying to protect. We will have to muddle through, with high uncertainty throughout the near-term crisis and beyond, navigating a universe in which nature is stratified by an intelligence layer much smarter than we are.
    74403.1K
  24. Joshua Achiam✓@jachiam0
    I think this is great and very cool! I am excited to see the insights here translate into tools with quantifiable success rates for helping us identify potential harms or steer neural networks away from risk. A lot of people on the timeline today seem to be updating a lot over this - my own personal update is very small. This feels like one of those long-sequence-of-small-but-helpful-improvements types of papers, not something that should radically reshape our worldviews. It's a good argument and one that could plausibly begin to constitute a reasonable prior. If correct it means that a lot of somewhat more "normal" or prosaic safety techniques will be reasonably reliable unless or until optimization pressure is applied against them. It is however in principle possible that optimization pressure will be applied to design alien intelligences impenetrable by conventional interpretability techniques, and that these will be used against us by someone or something at some time. There's a picture I have for the future - an ecosystem of many different powerful AI systems, most of which are pretty good and pretty aligned and pretty helpful and pretty honest, some of which are rogue or self-sovereign, and a small number of which are actively malicious to human interests. What the harmful ones will be able to accomplish will be limited, but not perfectly, by the good ones. Developments like good interp tools will help ensure that the good ones stay mostly good. But that there *will* exist good AIs that stay mostly good was already priced in for me, so this isn't a huge update.
    Nick @nickcammarata
    interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
    651218.8K
  25. Nick✓@nickcammarata
    interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
    8872.6K19.9K4.6M
  26. Joshua Achiam✓@jachiam0
    Putting aside the actual exchange here for a second: this is not a direct quote. This is an abridged quote that reconstitutes the remarks Ossoff made with omissions and new punctuation. Does it get the spirit of his remarks? Sure. But is it a direct quote? No! And I'm actually kinda pissed because this has been a trend on twitter for a while now. Journalists who claim to report on the news should regard this as a serious violation of basic journalistic ethics. Do not present abridged and reconstituted remarks as if they are direct quotes. Just do not do this. Omitting quotation marks is not enough. If you're going to write: "Person: [stuff they supposedly said]," get it exactly right! Do not make me have to listen to the clip to figure out if you creatively reimagined their words. Come on.
    Acyn @Acyn
    Ossoff: Mike, who on your team advised you to raise the Epstein files. You voted to cover up the evidence because Donald Trump told you to. Donald Trump, who by the way, is named 1000 times in the Epstein files
    40364.6K
  27. Acyn✓@Acyn
    Ossoff: Mike, who on your team advised you to raise the Epstein files. You voted to cover up the evidence because Donald Trump told you to. Donald Trump, who by the way, is named 1000 times in the Epstein files
    Download
    1891.7K12.4K1.1M
  28. Joshua Achiam✓@jachiam0
    Good God. Could anything touch the heart in a manner more simultaneously tender and wrenching?
    anna @bonykneesxx
    excuse me while I sob thinking of my babies as old people
    8224215K
  29. anna✓@bonykneesxx
    excuse me while I sob thinking of my babies as old people
    Download
    371.1K14.7K530K
  30. Joshua Achiam✓@jachiam0
    I was talking with someone earlier today about the process of becoming a stronger writer and what it takes to get better at clearly explaining ideas - I forgot to mention one really helpful tip, and thought it would be good to share broadly. Try to write style imitations of writers you like! Spend time trying to work out how they structure their arguments and how to replicate that argument structure. Do special effects tests where you try to write like them and see if what you produce can inspire the same feelings in you as the original author's writing. The old masters in many other domains of art learned by imitation; writing is not so different in this respect.
    6516213.5K