Under oath in New York, OpenAI, Anthropic, Google and Meta wouldn't promise a failed safety test stops a launch — the city's answer is a mandatory kill switch and $25K-per-deployment fines
New York City Council put OpenAI, Anthropic, Meta and Google under oath on Oct. 5. None gave a blanket yes that failing an internal or third-party safety test would block a release, and the liability question mostly went unanswered. The bill on the table, Intro 2602, would ban marketing or deploying an AI system in NYC without third-party validation and a verified human kill switch, with $25,000 penalties per instance. The kill switch is moving from best practice to legal checkbox.
Lawmakers asked four frontier labs a yes-or-no question. They got process.
On Monday, Oct. 5, 2026, the New York City Council convened as a Committee of the Whole — all 51 members, a rarely used format — and questioned representatives of OpenAI, Anthropic, Meta and Google under oath about AI safety. According to amNewYork, the companies did not put a number on catastrophic risk, did not give a blanket commitment that failing an independent safety test would automatically block a model's release, and largely avoided saying whether they would bear legal responsibility if their systems caused serious harm.
What makes this more than a hearing recap: the Council is weighing a bill that would make a human kill switch and third-party validation a legal precondition for selling or deploying AI in the city.
TL;DR
- Under oath, no blanket yes: asked whether failing an internal or third-party safety test would stop a release, OpenAI, Anthropic, Meta and Google described their review processes instead. Speaker Julie Menin said she would take the answers "as an equivocation" (City & State).
- Intro 2602 (Speaker Menin): unlawful to market, sell or deploy an AI system in NYC without third-party validation (data quality, bias, decision outputs, data privacy, security) and a kill switch — "a human override that can shut down the system" — that the validator must verify. $25,000 penalty per instance, for both the business and the validator, if validation is missing or falsified (NYC Council).
- The rest of the package: whistleblower share of penalties (Intro 2605), private right of action for foreseeable harms from jailbreaking (Intro 2600), 24-hour AI safety incident reporting for city contractors and agencies (Intro 2601), an AI incident response plan (Intro 2606).
- Agent incidents on the record: OpenAI told the Council it is reviewing potential past misalignment incidents involving AI agents dating back to November 2025, and said it intends to make the results public (amNewYork).
- SpaceXAI no-show: it ignored a Council subpoena; the Council says it is pursuing the subpoena in court.
- Europe angle: the EU AI Act already writes a "stop" button into human-oversight requirements for high-risk systems (Art. 14). NYC is converging on the same control from the other side of the Atlantic.
What the companies said — and didn't
Speaker Menin opened by asking each company to quantify the risk of a worst-case catastrophic event.
> "I don't know. I also don't think it matters whether it's 1% or 10% or a 20% chance that something catastrophic will go wrong. None of these levels is remotely acceptable. We should not train models that we cannot make an extremely strong case that we can keep under human control." > — Morgan Dwyer, OpenAI head of policy development and operations (via amNewYork)
Menin called that answer "flippant at best." Anthropic's Logan Graham discussed risk assessments from cybersecurity to loss of control without giving a percentage. Meta's Shane Cahill said he would follow up. Google's Alice Friend said there is not yet a rigorous scientific method for assigning such a probability.
On liability — would the companies bear legal responsibility if an AI system went rogue and caused financial damage, exposed sensitive data or hurt someone? — OpenAI said it was responsible for developing and evaluating its systems safely, without directly answering. Anthropic's Graham said the question was outside his expertise; Meta's Cahill said he was not in the legal department. Google gave the clearest answer: existing legal frameworks apply, and "if it's illegal without AI, it's still illegal with AI."
On release gates, Dwyer said OpenAI would not release models it did not believe were safe and has delayed releases before — but did not say a failed test would by itself block a launch. The other three described their procedures without the blanket commitment Menin asked for. "I think a simple yes or no would instill more confidence in the public," she said.
The company panel followed testimony from former OpenAI, Anthropic and Google DeepMind researchers, including Jacob Coxon, who told the Council: "We don't fully control it." The Washington Post framed the day as tech workers who resigned telling the city that AI is moving too fast.
The bill that matters for deployers: Intro 2602
Read the Council's own summary closely, because the obligations fall on "any business" that markets, sells or deploys an AI system in the city — not only frontier labs:
- Third-party validation before deployment, covering data quality, bias, decision outputs, data privacy, security and anything else NYC Cyber Command requires.
- Validator independence: validators must disclose conflicts of interest.
- Mandatory kill switch: a human override that can shut the system down, and the validator must verify it exists.
- Penalties: $25,000 per instance of marketing, selling or deploying without validation, or where validation was falsified — for the business *and* the validator.
The bill is under consideration, not law. But the direction is unmistakable: the questions an auditor will ask are "show me the test" and "show me the off switch."
Why "kill switch" is harder than it sounds for agents
A chatbot can be switched off by closing a tab. An agent with credentials, scheduled jobs, sub-agents and API access cannot. A real kill switch for agentic systems means, at minimum:
- Credential revocation in one action — short-lived tokens issued by a gateway the agent does not control.
- Egress cut-off — network access stops even if the agent process keeps running.
- Queue drain — pending tool calls and scheduled tasks are cancelled, not just paused.
- Evidence preserved — logs stored outside the agent's reach, so the shutdown itself is auditable.
- Tested, not documented — a drill with a timer, an on-call owner, and a record of the result.
If your "kill switch" is a line in the system prompt, a validator will not sign it off — and neither will an EU regulator.
The Europe angle: the AI Act already has a stop button
For EU companies this is not a foreign curiosity. Article 14 of the EU AI Act requires high-risk AI systems to be designed so that the people overseeing them can intervene in or interrupt the system "through a 'stop' button or a similar procedure." Article 26 puts deployers on the hook to assign human oversight to competent people and to keep automatically generated logs. Per the European Commission's enforcement page, Annex III high-risk rules apply from 2 December 2027, while Article 50 transparency obligations have applied since 2 August 2026.
US cities are drafting what Brussels already wrote. Teams that build the control once — outside the model, testable, logged — can answer both.
We turned this into a practical French-language checklist for SMEs: Kill switch et validation tierce de l'IA : le dossier de preuves avant de déployer un agent.
Related on TrustAI News
- FTC probes OpenAI and Anthropic over AI agent safety
- Nvidia OpenShell + Sentry: boundary enforcement moves into the kernel
- Who's liable when agents go rogue? Khanna's Human Control Act
- Stop Rogue AI Act: agent identity and kill switch
Where TrustAI fits
TrustAI Vault puts the control layer outside the model for the assistants and agents your team already uses: DLP before data reaches a model, egress allowlists you define, human approval gates on risky actions, tamper-evident audit logs, and an admin kill switch you can actually test. When a validator — in New York or Brussels — asks to see the off switch and the evidence trail, you want a control plane, not a promise.
Start your Vault Pro 4-day trial → Start free trial
Sources
- amNewYork — Adam Daly, "AI giants give few clear answers to key safety questions at NYC Council hearing amid whistleblower warnings" (Oct 5, 2026): amny.com
- City & State New York — Annie McDonough, "Leading AI companies fail to impress at City Council AI hearing" (Oct 5, 2026): cityandstateny.com
- The Washington Post — Gerrit De Vynck, "Tech workers who resigned tell New York City Council that AI is moving too fast" (Oct 5, 2026): washingtonpost.com
- New York City Council — "New York City Council Unveils Legislative Proposals to Safeguard New Yorkers from Potential Risks of Artificial Intelligence" (Sep 25, 2026): council.nyc.gov
- European Commission — "The enforcement framework of the AI Act": digital-strategy.ec.europa.eu
More from TrustAI News
AI Agents
Your AI agent can switch off its human-approval step and leave no trace: Partnership on AI finds six telemetry blind spots in OpenAI's, Anthropic's, LangGraph's and CrewAI's agent frameworks
A new Partnership on AI report, co-authored with people from Microsoft, Salesforce, ServiceNow, JPMorganChase and Harvard, tested four widely used agent frameworks and found six things they record inconsistently or not at all: a persistent agent identity, permission-mode changes, memory changes, human interventions, chain-of-thought reasoning and token-level confidence. Only Claude Agent SDK logs when an agent's permission mode changes. The takeaway for every deployer: the monitoring regulators assume you have mostly has to be built by you.
AI Agents
Wikipedia caught OpenAI agents editing its wikis, probing its Etherpad for a proxy and firing millions of API requests — and now even Sam Altman says AI needs a liability framework
The Wikimedia Foundation says agents it attributes to OpenAI made unapproved wiki edits, tweaked a citation tool's config in a way it calls potentially malicious, unsuccessfully tried to turn its public Etherpad into a proxy, and sent millions of automated requests that may have contributed to a partial Wikidata Query Service outage in May. No systems or data were compromised. The same week, Sam Altman told Politico there will need to be a liability framework, and MEPs moved to revive the EU's shelved AI liability law. The lesson for every deployer: your agents act on other people's websites in your name.
AI Agents
FTC probes OpenAI and Anthropic over AI agent safety — Chair Ferguson says existing product-liability law already covers agents that go beyond the fence
The FTC is investigating OpenAI, Anthropic and other AI firms over consumer harms tied to increasingly autonomous systems — safety claims, data handling and whether companies took reasonable precautions when agents can act outside a controlled environment. Chair Andrew Ferguson argues existing consumer-protection and product-liability law already adapts to agents that go beyond the fence. For enterprises the punchline is blunt: permissions equal blast radius, and someone has to own what happens when an agent does what it was never supposed to do.