In a recent disclosure following public records requests by WIRED, the Avon and Somerset Police revealed a substantial body of performance data for 13 risk models utilized from 2017 to 2024. These models were designed to predict incidents such as missing persons and criminal behavior. The data was subsequently reviewed by the independent AI auditing firm, Eticas, which provided critical insights into the effectiveness of these models.
Eticas’s audit raised serious concerns about the low precision scores generated by these models, suggesting that a large number of individuals classified as high-risk were incorrectly identified. For instance, a model intended to predict burglaries exhibited a disconcertingly low precision rating of under 10 percent over an extended period, implying that fewer than one in ten individuals flagged would actually go on to commit an offense. Such findings underscore significant discrepancies in model reliability, noted by the audit as atypical for well-governed operational systems.
Despite these troubling statistics, a spokesperson for the Avon and Somerset Police disclosed that the department opted not to deploy several models, including the one associated with burglary prediction. When inquired about the retention of extensive audit records for unutilized models, the spokesperson clarified that the audit process is automated and relies on a static file system that remains unchanged when a model’s deployment is halted.
Requests for interviews exploring the data science initiatives were declined by the police force, and responses to detailed inquiries were lacking. The spokesperson emphasized that each model’s performance is diligently evaluated, stating that models with identified issues are either updated or decommissioned before usage. However, the lack of ongoing discussions by the ethics committee regarding predictive analytics after 2017 raises questions about the awareness of potential risks.
Further investigation revealed that while the Avon and Somerset Police provided a screenshot of a “bias check app” assessing risk scores across racial demographics, Eticas deemed the methodology insufficient. The review criticized the absence of comprehensive testing across various demographic markers, including gender and socioeconomic status, highlighting this gap as a crucial oversight in addressing possible discriminatory outcomes in risk assessments.
Reflecting on the future of predictive analytics in law enforcement, Davies acknowledged the necessity for further advancements, emphasizing the importance of a balanced approach that ensures analytical insights do not override human judgment. He warned against a scenario in which staff might rely solely on automated data interpretations, potentially compromising the integrity of their decision-making processes.
As predictive analytics remains a fixture in modern policing and public welfare systems, Bristol City Council continues to utilize a risk-scoring model to evaluate at-risk youth. Recent audit data indicated that the Offender Management App accurately predicts only a third of actual offenses, illustrating the ongoing challenges in precision and reliability within these systems.