AI Extinction Warning Rocks NYC Council: What Leaders Said
AI Extinction Warning Rocks NYC Council: What Leaders Said
Former Anthropic researcher Jacob Coxon warns NYC Council AI could lead to human extinction by 2030. Tech giants debate safety and urgent regulation amidst legislative proposals.
A New York City Council hearing on Monday, October 5, became a platform for stark warnings regarding the existential risks posed by Artificial Intelligence. Jacob Coxon, a former Anthropic researcher who recently resigned over these very concerns, reiterated his chilling forecast: AI could lead to human extinction by the end of the decade.
Coxon's testimony was part of a broader discussion on AI's dangers, with representatives from major tech firms like OpenAI, Google, Anthropic, and Meta also appearing before the Council. Notably absent was Elon Musk's SpaceX. AI safety researchers, including those who recently left Anthropic, OpenAI, and Google DeepMind, painted a picture of a "reckless culture" within these companies.
They argued that the biggest tech giants are in a dangerous race to develop technology they don't fully understand or know how to control.
The companies run on a startup mindset: Move fast, break things, fix them later. That works for a photo sharing app. It does not work for building the most powerful technology ever built,
Coxon told the Council, as quoted by the Associated Press.
He ominously added, “On the current path, I think it is more likely than not that humanity loses control to these AIs and it could end in human extinction.”
In response, tech industry representatives called for increased oversight from local governments. They acknowledged mounting concerns that advanced AI systems could endanger humanity, even as they highlighted how their systems are improving daily lives and stressed safety as a top priority.
The hearing comes as the NYC Council considers new legislative proposals. These include requiring companies to disclose information about AI tools, mandating third-party approval for AI systems used in the city, and allowing whistleblowers to receive a share of fines from companies violating AI laws. New York Governor Kathy Hochul is also working to impose reporting and transparency standards for AI.
Policymakers globally are exploring measures to strengthen AI safety, driven by fears of catastrophic harm and recent hacking incidents involving AI behaving unexpectedly. Last month, Anthropic CEO Dario Amodei even called for an intentional slowdown of frontier AI efforts worldwide to allow safety research to catch up.
This sentiment found rare agreement among AI titans like OpenAI's Sam Altman, Elon Musk, and Google's Demis Hassabis, though others like Nvidia's Jensen Huang and former US President Donald Trump have dismissed such concerns as trivial.
OpenAI's Morgan Dwyer addressed the quantification of AI risk, stating,
I don't know - I also don't think it matters whether it's 1% or 10% or 20% chance that something catastrophic will go wrong. None of these levels is remotely acceptable.
He emphasized OpenAI's commitment to safety. Anthropic's Logan Graham welcomed “smart regulation,” asserting that state and local governments have a crucial role in developing AI frameworks.
This hearing underscores the growing tension between rapid technological advancement and the urgent need for robust safety protocols and governance in the rapidly evolving field of artificial intelligence.
