Dutch Probation Service Suspends Faulty AI Recidivism Algorithm After Discriminatory and Inaccurate Risk Assessments

The algorithm is an AI system used to predict recidivism risk, which directly impacts decisions affecting individuals' rights and freedoms. The presence of serious errors implies that the AI system's use has led or could lead to violations of human rights or legal obligations. The recommendation to stop or adjust the system indicates recognized harm or risk of harm. Therefore, this qualifies as an AI Incident due to the direct or indirect harm caused by the AI system's malfunction or misuse in a sensitive context.

The article explicitly discusses AI algorithms (risk assessment models) used in criminal justice decisions that have produced erroneous outputs affecting real individuals' liberty and societal safety. The harms include potential wrongful release or harsher sentencing, which are direct harms to persons and violations of rights. The AI systems' malfunction (errors, outdated data, discriminatory features) is central to the incident. The event meets the criteria for an AI Incident because the AI system's use has directly led to harm and rights violations, not merely a potential risk or complementary information.

The event involves an AI system (OxRec) used in the criminal justice system to predict recidivism risk. The algorithm's development and use led to direct harm by misclassifying individuals' risk levels, which influenced judicial decisions and could result in unjust sentencing or supervision conditions. The use of discriminatory variables without proper safeguards constitutes a violation of rights. Therefore, this qualifies as an AI Incident because the AI system's malfunction and misuse have directly caused harm to individuals and potentially breached legal and human rights obligations.

OxRec is an AI system used in criminal justice to predict recidivism risk. The inspection found that it underestimates risk for vulnerable groups, which can lead to discriminatory treatment and violations of fundamental rights. The system's outputs influence decisions by probation officers and judges, meaning the AI's use has directly or indirectly caused harm. The pause in usage and ongoing improvement efforts confirm recognition of harm. Hence, this event meets the criteria for an AI Incident due to realized harm linked to the AI system's use and its impact on human rights and justice outcomes.

The AI system (OxRec) is explicitly mentioned and is used in judicial decision-making processes. Its flawed outputs have directly influenced probation advice to judges, which can be reasonably inferred to have led to harm in terms of misinformed sentencing and potential discrimination, constituting violations of rights. The event involves the use and malfunction of the AI system leading to realized harm, thus qualifying as an AI Incident rather than a hazard or complementary information. The pause and improvement efforts are responses to the incident but do not change the classification.

The event involves an AI system (OxRec) used in criminal justice to predict recidivism risk. The system's inaccurate predictions and potential discriminatory bias constitute harm to individuals' rights and fair treatment, fulfilling the criteria for an AI Incident. The harm is realized as the system has been in use since 2018 and has never accurately predicted recidivism risk, directly impacting decisions in the justice process. Therefore, this is classified as an AI Incident due to violations of rights and harm to individuals caused by the AI system's use.

The OxRec algorithm is an AI system used to predict recidivism risk, influencing judicial decisions. The Inspectorate's findings that it underestimates risk and may discriminate imply that the AI system's outputs have caused or could cause harm to individuals' rights and fair treatment, fulfilling the criteria for an AI Incident. The pausing of its use is a response to these harms. Hence, this is not merely a hazard or complementary information but an incident where harm has occurred or is occurring due to the AI system's use.

The OxRec algorithm is an AI system used in judicial decision-making processes. Its malfunction—providing systematically inaccurate risk assessments and potentially discriminatory advice—has directly led to harm by undermining fair treatment and possibly influencing judicial outcomes negatively. The Inspectorate's findings confirm that the AI system's use has caused realized harm, including risks of discrimination and poor risk prediction, which affect individuals' rights and the justice process. Hence, this event meets the criteria for an AI Incident due to the AI system's malfunction and its direct link to harm.

The OxRec algorithm is an AI system used for risk assessment in the criminal justice process. The Inspectorate's criticism highlights that the system often underestimates recidivism risk and may discriminate, which can lead to violations of fundamental rights and unfair judicial decisions. Since the system's use has already led to recognized concerns about harm (discrimination and inaccurate risk assessment affecting legal outcomes), this constitutes an AI Incident. The pausing of the system is a response to these realized harms or risks that have materialized in practice, not merely a potential future harm or a complementary update.

The event involves an AI system explicitly described as an algorithmic risk assessment tool used in judicial decisions affecting suspects and offenders. The system's malfunction and misuse have directly caused harm by producing inaccurate risk scores, leading to potentially unjust detention and discriminatory treatment, which are violations of fundamental rights. The involvement of the AI system in causing these harms meets the criteria for an AI Incident. The article also mentions legal and privacy violations linked to the AI system's use, reinforcing the classification as an incident rather than a hazard or complementary information.

The event involves an AI system (OXREC) used for risk prediction in criminal justice, which has malfunctioned or been misused, leading to incorrect risk assessments that can harm individuals (e.g., underestimating recidivism risk) and society (public safety risks). This fits the definition of an AI Incident because the AI system's use has directly or indirectly led to harm through flawed outputs and systemic issues such as discrimination and privacy violations. The regulatory response and suspension of the system further confirm the seriousness of the incident.

The OXREC algorithm is an AI system used for predicting recidivism risk. The article details how errors in the algorithm's formulas and data have caused incorrect risk assessments, which can lead to harm to individuals and society by misguiding judicial and probation decisions. The use of discriminatory variables also constitutes a violation of rights. These factors fulfill the criteria for an AI Incident, as the AI system's malfunction and use have directly led to realized harms and legal violations. The probation service's response to suspend the algorithm is a mitigation measure but does not change the classification of the event as an AI Incident.

The article explicitly mentions the use of an algorithm (OXREC) to assess recidivism risk, which is an AI system as it infers from input data to generate predictions influencing decisions. The flawed and discriminatory nature of this algorithm led to incorrect risk assessments that influenced legal advice and decisions, constituting a violation of rights and discriminatory harm. The harm is realized and systemic, affecting thousands of cases annually. Hence, this is an AI Incident as the AI system's use directly led to harm in terms of discrimination and legal rights violations.

OxRec is an AI system used to predict recidivism risk, influencing judicial decisions. The investigation revealed serious configuration errors and omissions in the model, leading to incorrect risk assessments in a significant portion of cases. This misclassification can cause harm to individuals by affecting legal decisions, thus constituting harm to rights and potentially discriminatory practices. The direct link between the AI system's malfunction and realized harm to individuals' rights and fair treatment justifies classification as an AI Incident rather than a hazard or complementary information.

The event involves an AI system used in the criminal justice system whose malfunction and poor design have directly led to harm, including risks of discrimination and potentially unfair judicial decisions. This fits the definition of an AI Incident because the AI system's use has directly or indirectly caused violations of rights and harm to individuals. The harm is realized, not just potential, as the flawed algorithm influenced judicial decisions. Therefore, this is classified as an AI Incident.