Can AI Destroy Humanity? What U.S. Law Says About AI Safety, Liability and Human Control

Artificial intelligence has changed the way people work, communicate and make decisions. In only a few years, AI has moved from experimental technology to something used in offices, schools, businesses, software development, scientific research and government-related work.

But as AI systems become more capable, another question is becoming increasingly difficult to ignore:

What happens if artificial intelligence becomes more powerful than the safeguards designed to control it?

Some researchers and technology leaders are now openly discussing the possibility of extremely serious AI risks, including cyberattacks, biological misuse, weapons development and the loss of meaningful human control.

That does not mean that AI will inevitably destroy humanity. There is no established evidence that today's AI systems are independently planning human extinction.

The concern is different: increasingly capable AI could be misused by people, could be connected to sensitive systems, or could eventually perform increasingly complex tasks with less human supervision.

For the United States, that creates a major legal question: Who should be responsible when an AI system causes serious harm?

Why AI safety has suddenly become a major issue

The debate became more intense in September 2026 after Jacob Coxon, a researcher who had worked at Anthropic and previously at OpenAI, resigned from Anthropic and publicly raised concerns about the speed of AI development and the risks associated with increasingly powerful systems.

Coxon's resignation received significant attention because Anthropic has generally presented itself as an AI company with a strong focus on safety. Reporting on his resignation said that he gave up the opportunity to receive company equity that was close to vesting.

His concerns do not establish that an AI catastrophe will occur. They are, however, part of a much broader debate taking place inside and outside the AI industry about whether technological development is moving faster than safety measures.

That debate matters to American law because technology can develop much faster than legislation.

Anthropic itself is warning about the risks

Anthropic CEO Dario Amodei has also argued that frontier AI development needs to be paced more carefully.

In his September 2026 essay, “We Must Pace the Frontier,” Amodei did not call for stopping AI development altogether. Instead, he argued for a balanced approach in which companies continue developing AI while giving safety, alignment and independent evaluation enough time to keep pace with capability improvements.

This distinction is important.

The debate is not simply about whether America should “stop AI.”

The more practical question is whether companies should be required to demonstrate that highly capable AI systems are sufficiently safe before giving them greater autonomy or access to sensitive systems.

AI is already being misused by humans

The most immediate AI safety problem may not be an AI system deciding to attack humanity.

It may simply be humans using increasingly capable AI for harmful purposes.

Anthropic's September 2026 Threat Intelligence Report described several cases in which threat actors attempted to use Claude for malicious activity. The report discussed cyber operations, surveillance, fraud, biological misuse and weapons-related activity. Anthropic said that it identified and disrupted these operations.

Reuters independently reported on Anthropic's disclosure, including claims involving weapons development, cyber operations and surveillance.

This distinction is legally significant.

If a person intentionally uses an AI system to commit an unlawful act, the fact that AI was involved does not automatically transfer responsibility from the human actor to the AI developer.

At the same time, the incident raises difficult questions about whether developers should have duties to anticipate and prevent certain foreseeable forms of misuse.

What about biological weapons?

Biological misuse is one of the most sensitive areas of AI safety.

Anthropic reported that it had identified and disrupted several cases involving attempts to use Claude in ways that could support biological weapons development. The company has also previously described safety measures designed to prevent certain assistance relating to chemical, biological, radiological and nuclear weapons.

That does not mean that AI independently created a biological weapon.

It means that highly capable AI can potentially provide information or assistance that humans could attempt to misuse.

For policymakers, that creates a difficult balancing problem. Scientific research can be legitimate and beneficial, but the same capabilities can potentially become dangerous when used for malicious purposes.

The legal challenge is therefore not simply banning “dangerous AI.”

It is deciding how to identify high-risk uses while preserving legitimate scientific research and innovation.

AI and weapons development

Anthropic's September 2026 report also described an alleged case involving the use of Claude in weapons-related development in Yemen. Reuters reported Anthropic's account of AI being used for activities connected with weapons development.

Again, this should not be misunderstood.

The important issue is not that an AI system independently decided to manufacture a weapon.

The concern is that AI may reduce the amount of specialized knowledge or technical work required by someone attempting to develop sophisticated capabilities.

That creates a new category of risk for governments and technology companies.

What happens when AI becomes an autonomous agent?

Traditional software generally performs tasks according to predefined instructions.

Modern AI agents can potentially perform multiple steps, use software tools, interact with digital environments and complete tasks with less direct human intervention.

As these systems become more autonomous, the legal consequences could become much more complicated.

Imagine an AI agent connected to a company's internal computer systems.

The company gives the agent permission to perform a legitimate task. During that process, the system makes an unexpected decision, accesses information it should not have accessed, or causes damage to another system.

Who is responsible?

The answer may depend on several factors, including the company's instructions, the system's design, the level of human supervision, the foreseeability of the failure and the precautions taken before deployment.

Could an AI company be sued?

Potentially, depending on the facts and the legal theory involved.

An AI company might face litigation under existing legal doctrines involving negligence, product liability, contract law, consumer protection, privacy, cybersecurity or other applicable statutes.

But there is no general rule that an AI company automatically becomes legally responsible every time somebody misuses its product.

American courts normally look at the specific conduct, the applicable law, causation and whether the alleged harm was legally attributable to the defendant.

For example, if a person deliberately uses an AI tool to commit fraud, that person's intentional conduct remains highly relevant.

But if an AI developer knowingly deploys a system with a serious foreseeable risk and fails to take reasonable precautions, a different legal dispute could arise.

The facts would matter enormously.

Can AI itself be held legally responsible?

Under current U.S. legal principles, the practical responsibility generally remains with people and legally recognized organizations rather than the AI system itself.

An AI model does not simply become a legal person because it can generate sophisticated responses or make autonomous decisions.

That means courts are likely to continue asking questions such as:

Who developed the system?

Who deployed it?

Who controlled access to it?

Who gave it instructions?

What safeguards existed?

Was the risk foreseeable?

Was there meaningful human oversight?

Did someone ignore known warnings?

These questions could become increasingly important as AI systems gain greater autonomy.

What if AI affects critical infrastructure?

The risks become even more serious when AI is connected to systems that affect public safety or the economy.

Examples could include electricity networks, transportation systems, telecommunications, financial infrastructure, healthcare systems and military-related technology.

An error in an ordinary AI application might cause inconvenience or financial loss.

An error or malicious action involving critical infrastructure could potentially affect thousands or millions of people.

That is why AI safety is closely connected with cybersecurity and national security.

Anthropic's own safety framework recognizes risks involving cyber operations, chemical and biological threats, AI sabotage and loss of control.

The company's published safety roadmap also discusses the possibility that future AI systems could dramatically accelerate research in areas such as energy, robotics, weapons development and AI itself.

These are risk assessments, not predictions that such events will definitely happen.

The legal problem: technology moves faster than legislation

One of America's biggest challenges is that AI technology can develop within months while legislation and court precedent can take years.

The United States does not have one single federal law that answers every question about artificial intelligence.

Instead, different areas of existing law may apply depending on what the AI system does.

Those areas can include cybersecurity law, criminal law, consumer protection, privacy, intellectual property, employment law, product liability and national-security regulation.

State governments may also impose their own requirements.

This creates a complicated regulatory environment for businesses operating AI systems across multiple jurisdictions.

Should the government slow down AI?

This is one of the most controversial questions in the current debate.

One argument is that excessive regulation could slow innovation and allow other countries to gain a technological advantage.

The opposing argument is that developing increasingly powerful AI without sufficient safety testing could create risks that become much harder to control later.

Dario Amodei's position is notable because he has argued for “pacing” rather than simply stopping frontier AI development. His proposal emphasizes continued technological progress alongside stronger safeguards and outside evaluation.

That reflects the central policy dilemma facing the United States.

America wants to remain a leader in artificial intelligence.

But leadership in AI also means deciding what safety standards should apply to the most powerful systems.

Does U.S. law need a new AI liability framework?

Possibly, but lawmakers face a difficult question: how much can be accomplished through existing law, and where are new rules actually necessary?

Traditional legal principles can already address many forms of harm.

For example, negligence law can examine whether someone failed to exercise reasonable care. Product-liability law can address certain defective products. Contract law can allocate responsibilities between businesses. Criminal law can punish intentional wrongdoing.

But highly autonomous AI creates situations that older legal frameworks were not necessarily designed to address.

Congress may eventually have to consider questions such as whether certain high-risk AI systems should require independent testing, mandatory incident reporting, stronger cybersecurity controls or human authorization for particularly sensitive actions.

The challenge will be creating rules that reduce genuine risks without unnecessarily blocking beneficial technology.

AI extinction is not an established fact

The phrase “AI will destroy humanity” is powerful, but it should be treated carefully.

There is a major difference between saying that a technology presents an existential risk and saying that the technology will definitely cause human extinction.

Current evidence does not establish that human extinction from AI is inevitable.

There are also serious disagreements among researchers about the probability, timing and mechanisms of catastrophic AI scenarios.

What is increasingly difficult to dispute, however, is that AI capabilities are advancing rapidly and that these systems are already being tested in sensitive areas.

That alone is enough to justify serious discussion about safety, accountability and legal oversight.

The real question for U.S. law

Perhaps the most useful question is not:

“Will AI kill humanity?”

It is:

“Can the law keep humans accountable as AI becomes more autonomous?”

That question has practical consequences today.

If an AI system eventually makes important decisions, writes and executes software, interacts with other systems, conducts research or manages sensitive operations, human oversight cannot simply become a slogan.

There must be clear responsibility.

Companies need to understand what their systems can do.

Users need to understand what they are authorizing.

Governments need mechanisms for dealing with genuinely high-risk applications.

And courts may eventually have to decide how traditional concepts such as negligence, foreseeability and causation apply when a machine performs part of the chain of events leading to harm.

Conclusion

Artificial intelligence is not currently a legal person, and there is no established evidence that today's AI systems are independently planning to destroy humanity.

But the legal and security questions surrounding increasingly capable AI are real.

Recent reports of attempts to misuse AI for cyber operations, weapons-related activity and biological research show why developers are investing heavily in safety systems.

At the same time, researchers and AI executives are debating whether the industry is moving quickly enough—or perhaps too quickly—to keep safety measures aligned with technological progress.

For the United States, the central issue may ultimately be accountability.

If an AI system causes serious harm, who had the responsibility to prevent it?

That is a question American lawmakers, regulators, companies and courts may have to answer long before anyone can say with certainty whether AI will ever pose an existential threat to humanity.

Sources

Anthropic — Detecting and Countering Misuse of AI: September 2026
Read the Anthropic Threat Intelligence Report

Dario Amodei — We Must Pace the Frontier, September 2026
Read Dario Amodei's essay

Anthropic — Transparency Hub
Read Anthropic's transparency and safety commitments

Anthropic — Responsible Scaling Policy
Read the Responsible Scaling Policy

Reuters — How Anthropic says Claude was used for weapons, spying and cyber operations
Read the Reuters report

Axios — Former Anthropic researcher Jacob Coxon's resignation
Read the Axios report

Disclaimer

This article is provided for general informational and educational purposes only. It is not legal advice and does not create an attorney-client relationship. AI technology, federal regulations, state laws and government policies can change rapidly. The discussion of potential AI risks does not mean that any particular catastrophic event is certain to occur. Readers dealing with a specific AI, cybersecurity, privacy, employment, business or liability matter should consult a qualified attorney licensed in the relevant jurisdiction.