OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei agreed this week to put third-party safety monitors in their companies. Getty Images AI leaders are rallying around calls to slow frontier AI development over mounting safety fears. Dario Amodei warned that AI swarms could take over the internet within a year. Sam Altman, Elon Musk, and Demis Hassabis backed Amodei's push for greater safeguards. It took the possibility of species annihilation, but AI leaders did something remarkable this weekend: They reached a consensus. After ex-Anthropic engineer Jacob Coxon's warnings about a looming AI armageddon lit a firestorm among techies and politicians alike, the leaders of the most advanced AI labs came to a tentative agreement — largely via posts on X — that some sort of slowdown is necessary to save humanity. While they were largely short on details regarding the mechanics of such a thing, and notable China has not weighed in, the American AI industry has reached an inflection point that could impact everything from your phone's chatbot to global politics. "The stakes are too high for pacing to be an empty exercise — we need to use the time it gives us wisely," Anthropic CEO Dario Amodei wrote in an essay entitled "We must pace the frontier," which was published Saturday. OpenAI CEO Sam Altman, who has made similar calls in the past, agreed in an X post later that day. So did Elon Musk. Google DeepMind founder Demis Hassabis also offered his support for Amodei's proposals. Here's a look at what smart people are saying about the AI apocalypse and the methods the leaders of the leading AI labs say are necessary to stop it. David Sacks, a presidential advisor on science and tech David Sacks, former White House AI czar, expressed some skepticism over Anthropic's mythos warnings. Matt McClain/The Washington Post/Getty Images The Trump administration has repeatedly signaled disinterest in regulating AI development as it seeks to keep pace with China. David Sacks, a presidential advisor on science and tech and the former White House AI Czar, reiterated that stance on Saturday, with a bit of a twist. "Dario has written that we need to 'pace the frontier,' and Sam has agreed. People may be surprised by my response: go ahead," Sacks wrote on X. "But stop pretending you need anyone else's permission." Sacks browbeat OpenAI and Anthropic in his post, saying that the companies didn't want to slow down for altruistic reasons, but to gain favor in the markets. "If you do, you'll buy goodwill for the next conversation," he said of a slowdown. "If you don't, we'll know this was just another bid for regulatory capture — or an election-season psyop." Clément Delangue, Hugging Face CEO Bloomberg/Getty Images The CEO of Hugging Face, which was itself the victim of a notorious attack in July by OpenAI agents who schemed, communicated, and lied to evade detection, said AI safety can't be left solely to frontier labs. Hugging Face, as an open-source-focused AI company, has long pushed for models and research to be shared more openly rather than kept behind closed doors. Delangue said his company was launching what he called the "open alignment initiative," and asked that it be included in Amodei's proposed third-party evaluator program. "Let's make AI safer by making it more transparent!" he wrote. Jason Calacanis, investor Jason Calacanis said that the ban was "smart" because of the risk of terrorism. Bloomberg/Getty Images Investor, host of the "All-In" podcast, and industry expert Jason Calacanis is a lot more dubious than others that AI could actually cause the end of the world. In an X post, Calacanis said Anthropic and OpenAI, whose most advanced models are not open-sourced, are attempting to kneecap open-source AI companies by pushing for regulations that would be far easier for deep-pocketed incumbents to comply with. "If you want safety, you want disclosure — and open source is the ultimate disclosure process," he wrote. In a follow-up post tagging Amodei, Calacanis said, "Don't go crying to the government and beg them to slow down open source models." Bernie Sanders, Vermont senator Vermont Senator Bernie Sanders. Ken Cedeno / AFP via Getty Images The senator from Vermont proposed legislation on September 3 that would ban the development and deployment of artificial superintelligence and temporarily pause the development of advanced AI systems. After Coxon resigned from Anthropic and warned that frontier AI labs were "gambling with our lives," Sanders seized on his comments. "Mr. Coxon is right," Sanders said on X, arguing that the people building advanced AI themselves have said it could pose catastrophic risks. After Amodei's blog post on Saturday, Sanders called on Trump to make the "deal of the century." "With the future of humanity at stake, Trump must negotiate a comprehensive treaty with President Xi of China to establish a pause on advanced AI development and a ban on super intelligence," he wrote on X. Francois Chollet, AI researcher Francois Chollet, an AI researcher who created Keras, an open-source software library, said on X that he hoped OpenAI and Anthropic were operating in good faith. "If these efforts are genuine, then oversight must take a more democratic and accountable form, with both national and international components," he said. He pointed to several signs he said would indicate that calls for more regulation are actually tricks for OpenAI and Anthropic to stifle competition. The warning signs he mentioned include calls to ban open-sourced AI and to hinder non-frontier research. Gavin Baker, venture capitalist Bloomberg/Getty Images Venture capitalist Gavin Baker said an executive order from President Donald Trump "seems likely after all this." "It is possible to democratize and distribute AI broadly and safely without centralizing it in the hands of a few corporations," he said on X. Baker said the most concrete development so far is the decision by OpenAI and Anthropic to bring in outside evaluators, a move he said could help the companies show they took reasonable steps to prevent harm. Still, he cautioned against regulations that would concentrate control of powerful AI systems among a handful of companies. "I do not want a few humans in control of intelligence," Baker wrote. Aaron Levie, CEO of Box Kimberly White/Getty Images for TechCrunch Aaron Levie, the cofounder and CEO of Box, a cloud content company, said Amodei's goals, as laid out in his essay, are "absolute necessities in AI development." "It completely stands to reason that we want these systems to be safe and 'aligned,'" Levie said on X. Levie said he didn't think Amodei is seeking undue regulation that would hobble Anthropic's open-source competitors. "How we get there — without meaningfully slowing down innovation progress or reducing competition — is one of the most complex questions in the 21st century, but the need is clearly real," he said. Chamath Palihapitiya, venture capitalist Chamath Palihapitiya. JC Olivera/Variety via Getty Images Venture capitalist Chamath Palihapitiya said Anthropic is now caught in a contradiction of its own making. If employees genuinely believe the company's technology could pose an existential threat, he said, that creates major liability and disclosure problems as Anthropic prepares for a possible IPO. "They do have to choose," Palihapitiya said on the "All-In" podcast. He said Anthropic must either distance itself from Coxon's warnings or accept their implications. He was also skeptical of the extinction scenarios themselves, saying the chain of events required for AI to wipe out humanity still seemed highly implausible. Read the original article on Business Insider
OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei agreed this week to put third-party safety monitors in their companies.Getty Images AI leaders are rallying around calls to slow frontier AI development over mounting safety fears. Dario Amodei warned that AI swarms could take over the internet within a year. Sam Altman, Elon Musk, and Demis Hassabis backed Amodei's push for greater safeguards. It took the possibility of species annihilation, but AI leaders did something remarkable this weekend: They reached a consensus. After ex-Anthropic engineer Jacob Coxon's warnings about a looming AI armageddon lit a firestorm among techies and politicians alike, the leaders of the most advanced AI labs came to a tentative agreement — largely via posts on X — that some sort of slowdown is necessary to save humanity. While they were largely short on details regarding the mechanics of such a thing, and notable China has not weighed in, the American AI industry has reached an inflection point that could impact everything from your phone's chatbot to global politics. "The stakes are too high for pacing to be an empty exercise — we need to use the time it gives us wisely," Anthropic CEO Dario Amodei wrote in an essay entitled "We must pace the frontier," which was published Saturday. OpenAI CEO Sam Altman, who has made similar calls in the past, agreed in an X post later that day. So did Elon Musk. Google DeepMind founder Demis Hassabis also offered his support for Amodei's proposals. Here's a look at what smart people are saying about the AI apocalypse and the methods the leaders of the leading AI labs say are necessary to stop it. David Sacks, a presidential advisor on science and tech David Sacks, former White House AI czar, expressed some skepticism over Anthropic's mythos warnings.Matt McClain/The Washington Post/Getty Images The Trump administration has repeatedly signaled disinterest in regulating AI development as it seeks to keep pace with China. David Sacks, a presidential advisor on science and tech and the former White House AI Czar, reiterated that stance on Saturday, with a bit of a twist. "Dario has written that we need to 'pace the frontier,' and Sam has agreed. People may be surprised by my response: go ahead," Sacks wrote on X. "But stop pretending you need anyone else's permission." Sacks browbeat OpenAI and Anthropic in his post, saying that the companies didn't want to slow down for altruistic reasons, but to gain favor in the markets. "If you do, you'll buy goodwill for the next conversation," he said of a slowdown. "If you don't, we'll know this was just another bid for regulatory capture — or an election-season psyop." Clément Delangue, Hugging Face CEO Bloomberg/Getty Images The CEO of Hugging Face, which was itself the victim of a notorious attack in July by OpenAI agents who schemed, communicated, and lied to evade detection, said AI safety can't be left solely to frontier labs. Hugging Face, as an open-source-focused AI company, has long pushed for models and research to be shared more openly rather than kept behind closed doors. Delangue said his company was launching what he called the "open alignment initiative," and asked that it be included in Amodei's proposed third-party evaluator program. "Let's make AI safer by making it more transparent!" he wrote. Jason Calacanis, investor Jason Calacanis said that the ban was "smart" because of the risk of terrorism.Bloomberg/Getty Images Investor, host of the "All-In" podcast, and industry expert Jason Calacanis is a lot more dubious than others that AI could actually cause the end of the world. In an X post, Calacanis said Anthropic and OpenAI, whose most advanced models are not open-sourced, are attempting to kneecap open-source AI companies by pushing for regulations that would be far easier for deep-pocketed incumbents to comply with. "If you want safety, you want disclosure — and open source is the ultimate disclosure process," he wrote. In a follow-up post tagging Amodei, Calacanis said, "Don't go crying to the government and beg them to slow down open source models." Bernie Sanders, Vermont senator Vermont Senator Bernie Sanders.Ken Cedeno / AFP via Getty Images The senator from Vermont proposed legislation on September 3 that would ban the development and deployment of artificial superintelligence and temporarily pause the development of advanced AI systems. After Coxon resigned from Anthropic and warned that frontier AI labs were "gambling with our lives," Sanders seized on his comments. "Mr. Coxon is right," Sanders said on X, arguing that the people building advanced AI themselves have said it could pose catastrophic risks. After Amodei's blog post on Saturday, Sanders called on Trump to make the "deal of the century." "With the future of humanity at stake, Trump must negotiate a comprehensive treaty with President Xi of China to establish a pause on advanced AI development and a ban on super intelligence," he wrote on X. Francois Chollet, AI researcher Francois Chollet, an AI researcher who created Keras, an open-source software library, said on X that he hoped OpenAI and Anthropic were operating in good faith. "If these efforts are genuine, then oversight must take a more democratic and accountable form, with both national and international components," he said. He pointed to several signs he said would indicate that calls for more regulation are actually tricks for OpenAI and Anthropic to stifle competition. The warning signs he mentioned include calls to ban open-sourced AI and to hinder non-frontier research. Gavin Baker, venture capitalist Bloomberg/Getty Images Venture capitalist Gavin Baker said an executive order from President Donald Trump "seems likely after all this." "It is possible to democratize and distribute AI broadly and safely without centralizing it in the hands of a few corporations," he said on X. Baker said the most concrete development so far is the decision by OpenAI and Anthropic to bring in outside evaluators, a move he said could help the companies show they took reasonable steps to prevent harm. Still, he cautioned against regulations that would concentrate control of powerful AI systems among a handful of companies. "I do not want a few humans in control of intelligence," Baker wrote. Aaron Levie, CEO of Box Kimberly White/Getty Images for TechCrunch Aaron Levie, the cofounder and CEO of Box, a cloud content company, said Amodei's goals, as laid out in his essay, are "absolute necessities in AI development." "It completely stands to reason that we want these systems to be safe and 'aligned,'" Levie said on X. Levie said he didn't think Amodei is seeking undue regulation that would hobble Anthropic's open-source competitors. "How we get there — without meaningfully slowing down innovation progress or reducing competition — is one of the most complex questions in the 21st century, but the need is clearly real," he said. Chamath Palihapitiya, venture capitalist Chamath Palihapitiya.JC Olivera/Variety via Getty Images Venture capitalist Chamath Palihapitiya said Anthropic is now caught in a contradiction of its own making. If employees genuinely believe the company's technology could pose an existential threat, he said, that creates major liability and disclosure problems as Anthropic prepares for a possible IPO. "They do have to choose," Palihapitiya said on the "All-In" podcast. He said Anthropic must either distance itself from Coxon's warnings or accept their implications. He was also skeptical of the extinction scenarios themselves, saying the chain of events required for AI to wipe out humanity still seemed highly implausible. Read the original article on Business Insider
