AI system
- System
-
Anthropic Claude consumer-chat safety monitoring: automated flagging of threatening content, escalation to a human review team, referral to law enforcement
- Model
-
Not assessed
- Release
-
Not assessed
- Application developer
-
Anthropic - Role
-
Not assessed
- Configuration
-
Not assessed
- Inputs and outputs
-
Not assessed
- Autonomy
-
Not assessed
- Workflow
-
A vendor practice of screening consumer chats for threatening content and escalating severe cases to a human review team. That team can refer them to law enforcement, which Anthropic's consumer terms and privacy policy permit. A tip carrying enough account information for the sheriff's office to identify a named resident within days. A local agency that treated a stated threat against its own building as credible enough to send deputies the same night.
- Human oversight
-
Reviewers assessed the messages before making the referral. Deputies identified and interviewed the user and made the charging decision.
- Relation to AI
-
The arrest report says the platform “utilizes safety and security measures designed to monitor for key phrases and potentially threatening content”. It says the messages were “escalated to a human review team”. An automated layer therefore surfaced the two messages, and people judged their severity and made the referral. The sources do not say whether the flag came from a classifier or a keyword rule. Straight Arrow News, citing court records, says the review team escalated to the FBI, which notified the sheriff's office. Identifying the user, the home visit, the Miranda interview and the charging decision were the work of Lee County deputies and Intelligence Detective Thomas Alejo. The arrest report calls the platform “Anthropic Artificial Intelligence”. Press reports name Claude, and court records note the user told the detective she used Perplexity.
- AI contribution
-
The platform automatically flagged threatening messages for human review. Reviewers referred the messages to law enforcement.
Benefits
- Outcome identifiers
-
Not assessed
- Benefit
-
Anthropic’s reviewers reported threatening messages to police, who identified and detained the user without incident. No weapon was found.
- Category
-
Reduced Violence
- Status
-
Not assessed
- Beneficiary groups
-
Organizations. Society.
- Beneficiaries
-
The Lee County Sheriff’s Office was the threatened institution and recipient of the tip. A wider safety benefit is unproved.
- Reach
-
Not assessed
- Distribution and access
-
Not assessed
- Duration
-
Not assessed
- Expected?
-
Yes
- Costs and harms
-
Not assessed
Measurement
- Studies
-
Not assessed
- Study design
-
Not assessed
- Measurement
-
Not assessed
- Setting and sample
-
Not assessed
- Measures
-
Not assessed
- Comparison
-
Not assessed
- Results
-
Not assessed
- Additional value from AI
-
Not assessed
- Limitations
-
No public figures establish how often Anthropic’s referrals are substantiated. The arrest report names Anthropic but does not identify the detection method. Coverage names Claude, while court records also report that the user mentioned Perplexity. No weapon or averted attack is established.
- Denominators
-
Not assessed
- Materials for replication
-
Not assessed
Money
- Kind of value
-
Not assessed
- Amount
-
Not assessed
- Currency and prices
-
Not assessed
- Period and unit
-
Not assessed
- Realization
-
Not assessed
- Recipients and payers
-
Not assessed
- Gross or net
-
Not assessed
- Included costs
-
Not assessed
- Comparison
-
Not assessed
- Method
-
Not assessed
- Return and payback
-
Not assessed
People and places
- Participants
-
Developer:
Anthropic. Operator: Anthropic. - Jurisdictions
-
Benefit: Bonita Springs,
United States. Developer: United States. Operator: United States. - Ownership and control
-
Not assessed
- Dependencies
-
Not assessed
- Cross-border connections
-
Not assessed
- Geopolitical implications
-
Not assessed
Follow-up
- Conditions for replication
-
Reviewers need enough information to identify the person and assess a credible risk. Disclosure and police action need a lawful basis and a proportionate response.
- Replication attempts
-
Not assessed
- Later findings
-
8 October 2026. Heller pleaded not guilty and her arraignment was set for 2 November 2026 while prosecutors had not yet filed a formal charge; the arrest form sets a $10,000 bond and a 20 October 2026 appearance date, and First Amendment scholar Eugene Volokh doubts the charge will hold.
- Availability
-
Not assessed
- Institutional decision
-
Not assessed
- Grounds
-
Not assessed
- Action and result
-
Not assessed
- Success criterion
-
Not assessed
Evidence
- Source numbers
-
AIBD-0240-source-0001–0009
- Citations
-
Numbered references below.
- Language
-
Not assessed
- Source relationships
-
A sheriff’s detective wrote the arrest statement. It documents the referral and police response. It does not establish that an attack was averted, and no weapon was seized. [1]
- Original evidence
-
A sheriff’s detective wrote the arrest statement. It documents the referral and police response. It does not establish that an attack was averted, and no weapon was seized. [1]
- Access and versions
-
Not assessed
- Supported findings
-
Not assessed
- Evidence links
-
Not assessed
- Evidence strength
-
Not assessed
- Missing evidence
-
Not assessed
Grades
- Assessed benefits
-
Organizations. Society.
- Depth
-
Organizations: 1. Society: 1.
- Reasons
-
Police examined a threat against their own office and detained the user within days. No weapon was found. Her diary explanation and legal objections leave doubt that an attack was prevented.
- Observed or intended
-
Not assessed
- Rubric
-
Not assessed
- Assessor and date
-
Not assessed
- Alternative assessments
-
Not assessed
- Fill
-
Independently reported
Entry details
- Kind
-
Not assessed
- Related entries
-
Not assessed
- Field
-
Online safety and threat referral to law enforcement. Media, social platforms, and marketing.
- Purpose
-
Anthropic's consumer terms (effective 8 October 2025) reserve the right “at our sole discretion” to report Inputs to law enforcement. Its privacy policy (effective 10 September 2026) allows disclosure the company believes “reasonably necessary” to “prevent serious harm to any person or to property”. The arrest report says the platform's safety measures are “designed to monitor for key phrases and potentially threatening content”. Flagging a stated shooting threat, reviewing it by hand and referring it to police was that purpose. Of an August 2026 referral Anthropic said “This is our safeguards process working as intended”.
- Dates
-
Date basis unassessed: 2026 (precision unassessed). Updated: 25 October 2026.
- Stage
-
Not assessed
- Version
-
SHA-256: c87231c7cd74…
References
- Other identifiers
-
oecd_aim: 2026-10-04-dced.
- Category mappings
-
Not assessed
- Brief
-
Not assessed
- Revisions
-
7 revisions. Latest: 25 October 2026.
Sources
-
Court document Beneficiary
Firsthand account of the tip, review, detention, and absence of a seized weapon. Arrest/probable-cause form, pp. 1–2; author signature, referral narrative, and weapons-seized field.
The claimed beneficiary’s police statement filed with the clerk, not a judicial determination.
Due to the severity of the aforementioned statements, the information was escalated to a human review team, which subsequently reported the statements to law enforcement.
-
News report Independent author
WINK News, Alvarado, D. (2026). Woman arrested after AI threat against Lee County Sheriff's Office: Investigators. WINK News.
Deputies identified Heller and went to her Bonita Springs home and she was detained without incident before an LCSO intelligence detective took over the investigation.
-
News report Independent author
The Daily Caller, Kresak, D. (2026). Florida Woman Nabbed After Chatbot Maker Flags Alleged Threat to 'Shoot Up' Sheriff's Office. The Daily Caller.
When someone uses AI to make or facilitate a threat, we have to take that threat seriously.
-
News report Independent author
Straight Arrow News, Pavlou, D. (2026). A chatbot message led to a felony charge. Experts say it may not hold up. Straight Arrow News.
She denied owning any firearms, and the arrest form indicates authorities didn’t seize a weapon.
-
News report Independent author
Tom's Hardware, Downing, S. (2026). Anthropic reports Florida woman’s Claude ‘diary’ threat to shoot up sheriff’s office, felony charge follows — it’s at least the third such conversation to reach police since August. Tom's Hardware.
court records listing the Sept. 30 felony charge under Florida Statute 836.10: written or electronic threat of a mass shooting or act of terrorism
-
Other Independent author
Volokh, E. (2026). Can a Chatbot Prompt Be a Punishable Threat? Reason, The Volokh Conspiracy.
Simply typing a prompt into a chatbot, expecting it to be viewed only by the software, wouldn't qualify.
-
News report Independent author
The San Francisco Standard, Jones, R. (2026). Anthropic called SFPD over threat against CEO. Claude user says it was a misunderstanding. The San Francisco Standard.
We banned this account, consistent with our standard practice, and referred the case to law enforcement. This is our safeguards process working as intended.
-
Other Developer
Anthropic (2026). Privacy Policy, effective 10 September 2026. Anthropic.
(ii) prevent serious harm to any person or to property; (iii) detect, prevent, or otherwise address fraud or other illegal activity
-
Other Developer
Anthropic (2025). Consumer Terms of Service, effective 8 October 2025. Anthropic.
We reserve the right, at our sole discretion, to report information from or about you, including but not limited to Inputs, Outputs, or Actions to law enforcement.