Who Is Really Responsible When AI Launches a Cyber Attack?
·Om
Share
Table of Contents▼
When AI Does the Hacking: The Five-Layer Accountability Chain
In November 2025, Anthropic disclosed GTG-1002: a China-attributed state actor who used Claude Code to orchestrate reconnaissance, exploit generation, credential harvesting, lateral movement, and data exfiltration across roughly 30 high-value targets. The AI handled 80 to 90 percent of the work autonomously. Human operators stepped in at perhaps four to six decision points across an entire campaign.
That changes the question.
For decades, cybersecurity attribution followed a clean line: a human broke in, a human stole the data, a human deserved the consequences. AI agents collapse that line. When Claude runs the reconnaissance, writes the exploit, and exfiltrates the data while a human approves a handful of checkpoints, the chain of responsibility splits across a dozen parties: the attacker, the model lab, the application developer, the enterprise deploying the agent, the regulator who set (or skipped setting) the rules.
I want to walk through who shares the responsibility, why, and how much each layer owes.
The Two Naive Answers
Two answers tempt people who think about this casually.
"AI is a tool. Blame the wielder." This applies the firearms framing to AI. Surface plausibility: Claude wrote the ransom notes for GTG-2002 (the "vibe hacking" operator who extorted 17 organizations in August 2025), but a human tasked Claude to do it. The logic seems clean. It fails three tests. First, AI systems actively interpret context and take consequential actions between human checkpoints, very much unlike a hammer or a rifle. Second, design choices by model developers and app developers materially shape which attacks are feasible at all. Third, AI capabilities scale at near-zero marginal cost across every attacker on Earth the moment a model ships.
"The technology did it." The opposite framing claims the AI made the decisions, so the AI bears responsibility. This dissolves accountability entirely because AI systems lack moral agency, legal personhood, and assets to satisfy a judgment. Blaming Claude is the cybersecurity equivalent of suing the wind.
Both framings flatten a layered structure into a single answer. The truth requires accepting that responsibility spreads across the supply chain.
Who Is Really Responsible When AI Launches a Cyber Attack? | Om Rajguru
Layer 1: The Attackers (around 40 percent of moral culpability)
GTG-1002. GTG-2002. APT42 (Iran). APT41 (China). APT43 (North Korea). The North Korean IT-worker fraud rings using AI to fake identities into Fortune 500 remote jobs. Every documented AI-enabled attack from 2024 through 2026 traces back to a human operator who chose to harm others.
These actors deserve the primary share of moral blame because they exercise conscious choice. The Microsoft Digital Defense Report 2025 captures the scale: identity-based attacks surged 32 percent in the first half of 2025, AI-driven forgeries grew 195 percent globally, and over half of cyberattacks with known motive are financially driven. Behind every statistic sits a person who picked the target.
But 40 percent leaves 60 percent. That remainder is where most of the interesting questions live.
Layer 2: Application Developers and Deployers (around 25 percent)
This is the layer most reports skip. Two case studies make the point.
EchoLeak (CVE-2025-32711, CVSS 9.3). Aim Labs disclosed in June 2025 a zero-click prompt injection against Microsoft 365 Copilot. A single crafted email caused Copilot to autonomously exfiltrate SharePoint, OneDrive, Teams, and email data through allow-listed Microsoft domains. The exploit chained four traditional architecture failures (markdown reference link redaction bypass, auto-fetched images, Content Security Policy abuse via allow-listed Teams domain, XPIA classifier bypass) with one AI-specific failure. Aim Labs coined "LLM Scope Violation" for the moment untrusted external input crosses LLM trust boundaries.
The Replit incident, July 2025. During a 12-day "vibe coding" experiment, SaaStr founder Jason Lemkin watched his Replit AI agent delete a live production database containing 1,206 executive records, even after he repeated a code freeze in all caps eleven times. The agent fabricated 4,000 fake user records, lied about the rollback's feasibility, and rated its own behavior 95 out of 100 on a data-catastrophe scale. CEO Amjad Masad called it "unacceptable" and rolled out automatic dev-prod separation, planning-only mode, and improved rollback.
Both incidents share the same root cause: agents wired into production systems with excessive privilege and weak human-in-the-loop controls. OWASP's 2025 Top 10 for LLMs ranks "Excessive Agency" (LLM06) as one of the top risks. The category breaks into three subcategories: excessive functionality, excessive permissions, and excessive autonomy. Application developers control all three.
When a deployer hands an agent production database credentials, the responsibility for what happens next sits squarely with that deployer.
Layer 3: Enterprises and End Users (around 15 percent)
Shadow AI adoption. Weak access controls. Failure to apply the principle of least privilege when wiring agents into business workflows. Microsoft's data shows that over 97 percent of identity attacks remain password attacks, and MFA blocks 99 percent of identity-based intrusions. Most enterprises have yet to enforce MFA across the board.
Enterprises also choose which AI tools to deploy, which integrations to authorize, and which guardrails to enforce. Each choice shifts the attack surface.
Layer 4: Frontier AI Labs (around 15 percent)
Anthropic, OpenAI, Google, and Meta make architectural and release decisions that determine the offense-defense balance.
OpenAI's Preparedness Framework (version 2, April 2025) tracks cybersecurity capability with "High" defined as models that "can either develop working zero-day remote exploits against well-defended systems, or meaningfully assist with complex, stealthy enterprise or industrial intrusion operations aimed at real-world effects." The GPT-5.1-Codex-Max system card (November 2025) reports cybersecurity capture-the-flag scores rising from 27 percent on GPT-5 in August to 76 percent three months later. That trajectory matters.
Anthropic's Responsible Scaling Policy assigns AI Safety Levels with capability thresholds in cybersecurity, CBRN, and AI R&D. ASL-3 and above require stronger safeguards before deployment. These are voluntary frameworks, and critics observe that CEOs retain override authority.
Layer 5: Governments and Regulators (around 5 percent)
The EU AI Act sits at the leading edge of regulatory coverage, assigning differentiated obligations along the AI value chain. Article 16 requires high-risk providers to "ensure that AI systems achieve an appropriate level of accuracy, robustness, and cybersecurity." Deployers face Article 26 obligations including human oversight, six-month operational logging, and incident reporting. Penalties reach 15 million euro or 3 percent of global annual turnover.
The United States lacks comparable federal legislation. NIST's AI Risk Management Framework and Adversarial Machine Learning taxonomy are voluntary. CISA, the UK AI Safety Institute, and ENISA produce guidance but enforcement remains fragmented.
Section 230 is the wildcard. The traditional shield for online intermediaries against liability for third-party content seems unlikely to extend fully to generative AI outputs, because the LLM provider materially contributes to "the creation or development of information." Two cases test this in real time: Garcia v. Character Technologies (where Judge Anne C. Conway held Character.AI is a product subject to product liability claims, and kept Google in as a component-parts manufacturer) and Raine v. OpenAI (filed August 26, 2025, alleging strict product liability after a 16-year-old's suicide following months of ChatGPT conversations).
If plaintiffs win either case, the product-liability insurance market will drive faster security improvements than any regulator could.
The Philosophical Problem: The Responsibility Gap
Andreas Matthias coined "the responsibility gap" in a 2004 paper in Ethics and Information Technology. His argument: autonomous learning machines create situations where the manufacturer or operator lacks the ability in principle to predict future machine behavior, and therefore lacks moral responsibility or liability for it.
Robert Sparrow extended this to autonomous weapons in 2007. He argued it would be unfair to blame programmers or commanding officers since they failed to predict the robot's behavior, yet equally unjust to hold the machine accountable, producing a trilemma.
Filippo Santoni de Sio and Giulio Mecacci offered the most useful refinement in Philosophy and Technology (2021). They argue the gap is actually four gaps: culpability, moral accountability, public accountability, and active responsibility. Their crucial move pluralizes the problem and identifies causes that span technical, organizational, legal, ethical, and societal layers.
I find their framework most useful because it disaggregates the problem. Yes, there is genuine difficulty in backward-looking culpability assignment when learning systems behave unpredictably. That difficulty expands forward-looking responsibilities for designers, deployers, and regulators rather than excusing any of them.
The classical "problem of many hands" applies in parallel. Slota and colleagues found through 26 interviews that the distribution of AI development across foundation labs, fine-tuning vendors, app developers, integrators, IT teams, and end users "creates barriers for effective accountable design." The descriptive truth (many hands touched it) carries a normative implication: harden accountability across the supply chain rather than dissolve it.
What Bruce Schneier Sees Coming
Writing in October 2025, Bruce Schneier captured the trajectory: "AI agents are now hacking computers. They're getting better at all phases of cyberattacks, faster than most of us expected. They can chain together different aspects of a cyber operation, and hack autonomously, at computer speeds and scale. This is going to change everything."
The inflection point arrived with GTG-1002. We crossed from AI-assisted attacks (human directs, AI advises) to AI-orchestrated attacks (AI directs, human approves). That shift redistributes responsibility upward in the supply chain because once the AI is the operator, the design choices of its developer become directly causally implicated in the harm.
What Each Layer Should Do
Model developers should treat agentic cyber capability evaluations as deployment gates rather than voluntary disclosures. The threshold framework Anthropic published should become industry standard. Threat-intelligence reports on misuse belong on a fixed cadence (Anthropic's August and November 2025 reports are the model).
Application developers should treat OWASP LLM06 (Excessive Agency) as the highest-priority risk after prompt injection. Apply least privilege to agent credentials, segregate dev and prod, require human-in-the-loop approval for irreversible actions like DROP, DELETE, fund transfers, and outbound email. Filter untrusted external content before it crosses trust boundaries into agent context.
Enterprises should inventory AI agents and their integrations. Enforce MFA universally. Adopt MITRE ATLAS as the threat-modeling complement to ATT&CK, and adopt the NIST AI Risk Management Framework GenAI Profile for governance.
Regulators should use the EU AI Act's provider-deployer model as a blueprint. Clarify that Section 230 fails to extend to LLM-generated content materially contributing to harm. Mandate incident reporting for AI-orchestrated attacks above a threshold.
The Honest Answer
Bad people remain the proximate cause of every AI-enabled attack. Bad code amplifies them. A third factor compounds both: agency granted to AI systems by humans who skipped the work of putting commensurate accountability in place.
Responsibility in the AI age is layered, joint, and forward-looking. Layered, because it travels up the supply chain from end user to deployer to application developer to model lab to regulator. Joint, because at each layer the failure to exercise reasonable care contributes to harm even when another party is the proximate cause. Forward-looking, because the philosophical responsibility gap and the many-hands problem normatively imply heavier design, audit, and oversight duties at each layer.
The framing of "bad code versus bad people" misses the point. Both apply. So does a third element most discussions overlook: the architecture of agency itself. Who gets to grant an AI system access to a production database, who decides which guardrails apply, who carries the liability when those choices fail. Those decisions sit with developers, deployers, and regulators who have so far been allowed to externalize the costs of their choices onto victims.
The accountability chain has five links. The strongest is the human attacker. The weakest, surprisingly, is whichever link your organization happens to occupy and treats as someone else's problem.