Events where AI systems cause or contribute to unexpected harm, operate outside their intended parameters, or fail in ways that impact users, organisations, or society. Tracking and learning from incidents is crucial for improving AI safety and reliability.

AI incidents range from minor performance issues to significant harms affecting individuals or systems. They provide valuable learning opportunities for improving AI design, testing, and governance. Establishing systematic approaches to incident detection, classification, response, and prevention helps organisations build more reliable AI systems over time. Incident management frameworks typically include monitoring systems, response protocols, root cause analysis processes, and feedback mechanisms to prevent similar incidents.

‍

A customer service chatbot generating offensive responses to certain user questions, prompting customer complaints, negative media coverage, and an immediate system review to identify and address the underlying causes.