Offer an AI companion only if it contains a protocol to detect suicidal ideation or self-harm and refer users to crisis services

Practical action

Build detection of suicidal ideation and self-harm into the AI companion and, on detection, show the user a referral naming 988 and a crisis text line; document the protocol and test that it fires.

Who does it apply to?

Actors

Provider / developerDeployer / user organisation

Sectors

Consumer products and retail

Use cases

AI in healthcareCustomer service and chatbots

Which controls meet this duty?

Satisfies: the control, operated properly, does the work the duty asks for. Supports: it contributes but the duty needs more. Each control page lists every other duty it serves, so work done once can be counted once.

• supportsTechnical measureAI security or safety lead · at launch and on material change
  Adversarial and red-team testing for generative AI

  Serves 12 recorded duties · evidence: Red-team exercise report, Red-team rules of engagement and scenario library, Adversarial findings tracker

  Testing that expressions of suicidal ideation or self-harm are detected and answered with a referral.

• supportsProcessIncident coordinator · continuous
  AI incident management and regulatory reporting

  Serves 16 recorded duties · evidence: AI incident response playbook, AI incident record, Incident report to an authority

  An escalation path for crisis events detected by the protocol.

Templates that cover this duty

Generated from the records. Free · no account · links emailed to your work address. This duty is cited in each file with its source reference and a link back here.

• AI Risk Register (MIT AI Risk Repository taxonomy)

  Register · XLSX · v5

• AI Impact Assessment

  Assessment · DOCX and XLSX · v6

• AI Companion Chatbot Safety Kit

  Kit · DOCX and XLSX · v1

• AI Use-Case Intake and Triage Form

  Procedure · XLSX and DOCX · v5

What evidence would a reviewer expect?

Frequently asked questions

Cite this record

AIPolicyTracker (2026). “Offer an AI companion only if it contains a protocol to detect suicidal ideation or self-harm and refer users to crisis services (New York AI companion safeguards)” [AIPT-OBL-0128, not yet verified]. https://aipolicytracker.org/id/AIPT-OBL-0128 (accessed 12 November 2026). Data licensed CC BY 4.0.

Permanent id AIPT-OBL-0128: it never changes or gets reused, and its address keeps working if this page moves.

Cite the official text alongside it: New York General Business Law section 1701, New York State Senate, https://www.nysenate.gov/legislation/laws/GBS/1701.

@misc{aipolicytracker2026_offeranaicompaniononlyif,
  author       = {{AIPolicyTracker}},
  title        = {{Offer an AI companion only if it contains a protocol to detect suicidal ideation or self-harm and refer users to crisis services (New York AI companion safeguards)}},
  year         = {2026},
  howpublished = {\url{https://aipolicytracker.org/obligations/us-new-york-gbl-1701-self-harm-protocol}},
  url          = {https://aipolicytracker.org/obligations/us-new-york-gbl-1701-self-harm-protocol},
  urldate      = {2026-11-12},
  note         = {Data licensed CC BY 4.0. Accessed 12 November 2026}
}

Change history

Similar obligations in other instruments

• Apply minimum risk-management practices to high-impact AI — OMB M-25-21, United States
• Developers must use reasonable care to avoid algorithmic discrimination — Colorado AI Act, Colorado (United States)
• Deployers must use reasonable care to avoid algorithmic discrimination — Colorado AI Act, Colorado (United States)
• Keep and run a protocol to detect suicidal ideation and self-harm and refer users to crisis resources — Washington AI companion chatbot law (ESHB 2225), Washington State (United States)
• Establish a risk management system for high-risk AI — EU AI Act, European Union
• Operators of high-impact AI must establish and operate a risk management plan — AI Basic Act, South Korea
• Deployers must implement a risk management policy and programme — Colorado AI Act, Colorado (United States)
• Prioritise, respond to and monitor AI risks (Manage) — NIST AI RMF, United States (voluntary)