Many AI governance tools include faulty AI fixes, report finds

The World Privacy Forum’s report on AI governance tools reveals substantial issues with the current state of these tools. A review of 18 AI governance tools used by governments and multilateral organizations found that over a third of them include “faulty fixes,” which can undermine the trustworthiness of AI systems. The report points out that there is a lack of established requirements for quality assurance or assessment for these tools, leading to potential misuse and unintended consequences. The tools, developed or disseminated by companies like Microsoft, IBM, and Google, also lack proper documentation and context, which can result in varied, inappropriate applications.

The report defines AI governance tools as tools and techniques meant to evaluate and measure AI systems for their inclusiveness, fairness, explainability, privacy, safety, and other trustworthiness issues. It suggests that while regulators and the public might be reassured by the use of these tools, they can create a false sense of confidence and undermine the promise of AI systems. However, the report also highlights the opportunity to improve this whole ecosystem of AI governance in the wake of recent developments such as the EU AI Act and President Biden’s AI Executive Order.

The founders of the World Privacy Forum express hope for improvements to AI governance tools in 2024, with leading organizations like the OECD and the National Institute of Standards and Technology showing willingness to work towards this goal.