Study watch: Risky Artificial Intelligence

A review of 'Risky Artificial Intelligence: The Role of Incidents in the Path to AI Regulation' by Giampiero Lupo of the Institute of Applied Sciences and Intelligent Systems, National Research Council of Italy.

Champ Panupong Techawongthawon, Visualising AI by DeepMind

Earlier this week, Giampiero Lupo of the Institute of Applied Sciences and Intelligent Systems, National Research Council of Italy wrote in the Law, Technology and Humans journal on the role 'incidents' in shaping AI regulation. Lupo draws comparisons between AI and other new, complex technologies––such as the advent of nuclear power and the introduction of the automobile––to connect the cataloguing and analysis of AI incidents (defined as a situation in which AI systems caused, or nearly caused, real-world harm) with an ability to effectively regulate artificial intelligence.

The paper suggests that the spread of advanced technology tends to result in a divide between those excited by its novelty and those wary of its risks, while the coexistence of new and old technologies typically introduces novel challenges that require regulation in order to safely integrate them into society. It reminds us that, like other advanced technologies, concerns about safety, novel risks, and sudden changes disrupt the systems, structures, and arrangements that people are accustomed to and comfortable with.

On the one hand, norms can control technological change, restricting its scope and impact on the existing order; while on other, it is the sense of displacement that new technology brings that prompts a rush to regulate to restore or protect normalcy. This dynamic is particularly important "because AI refers to autonomous entities by definition and involves technologies that can act as intelligent agents that receive perceptions from the external environment and perform actions autonomously." As the paper later acknowledges, however, 'AI' is challenging to define: the technology exists as a constellation of data, people, power, and technological practice. AI systems might be agentic or tool-based, use-case specific or generalist, follow hard-coded rules or 'learn' from examples. Many of today's systems are not capable of 'autonomous' action, even fewer when using a restrictive definition. (Autonomy should not, however, be confused with capability or the potential to cause harm).

According to the paper, the regulation of emerging technologies (including AI) can be connected with two drivers: uncertainty about the potential impact of a technology and its risks, and information gathering in the aftermath of undesirable or unintentional events whose happening resulting in harm, injury, or damage. The latter, which is the primary focus of the author, are referred to as ‘incidents’. Incidents shape the regulation of emerging technologies because as technology becomes more sophisticated, it becomes harder to identify safety issues and their impact on individuals, society, and the environment in advance of their use.

Historical parallels: from automotive to AI

The theoretical basis through which AI incidents are connected with regulation is the notion that crises provide insight into the operation and effects of new technology. Drawing on Yale sociologist Charles Perrow's Normal Accident Theory, which suggests that situations exist wherein the complexity of systems unavoidably leads to accidents, the paper proposes that the sophistication of AI makes incidents both inevitable and unpredictable. In this framing, it is only after a crisis that safety issues can be identified, the technology's impact fully understood, and its inner workings revealed. Describing such moments as 'crises' is perhaps particularly apt when using the clinical definition: 'the turning point of a disease when an important change takes place, indicating either recovery or death.' Perrow's idea, which was originally formulated in the wake of the Three Mile Island disaster, is applied by the author to the automobile industry in the United States.

High risks and numerous casualties also had an impact on regulatory regimes that tried to improve the safety of the use of automotive technology. For instance, the 50,000 lives lost per year in 1966 in the United States contributed to the change of paradigm from ‘auto-safety’ to ‘crashworthiness’. The auto-safety paradigm was based on the assumption that as soon as nobody hits each other, no one will get hurt; therefore, this approach focuses on the ‘three Es’: (1) engineering roads to limit the possibility of collisions and equipping vehicles with reliable brakes and steering, (2) educating drivers and pedestrians to avoid collisions and (3) drafting and enforcing rules of the road to discipline drivers’ behaviour. In contrast, the ‘crashworthiness’ paradigm diffused since the late 1960s considered that a number of incidents on the road are unavoidable; therefore, car manufacturers had to design and implement technologies like seat belts and airbags that limit the impact of incidents on the human body.

By this account, the significance of the move away from 'auto-safety' and towards 'crashworthiness' represented a shift of responsibility for the consequences of incidents away from drivers and towards the developers of automobiles. Drawing a parallel with contemporary AI systems, the paper suggests that both are "characterised by a complex interaction between technology and human agents." As a result, determining where responsibility lies when failure occurs can prove challenging. For the automobile industry, the question is whether the driver, the car’s manufacturer (or possibly, a third party or the quality of transport infrastructure) is to blame. For AI, the question is more complex given the scale of the 'value chain' underpinning modern day systems: the developers building the technology, those deploying models for enterprise and consumer applications, and individual users all have the capability to cause harm. For both the AI and automotive industry, how best to determine liability amongst different parties in the aftermath of an incident can be a challenging question to answer.

AI incidents and AI regulation

The bulk of the paper explores the link between AI incidents and regulation by examining how incident analysis can inform the regulatory agenda. To do so, the paper describes two phenomena as reactions to the uncertainties and risks associated with the rapid diffusion of AI. First, there are the adoption of so-called 'soft laws' as frameworks, guidelines, and ethical principles associated with the development and deployment of AI. Second, the national and supranational legislation whose goal is to design and implement regulatory frameworks that seek to regulate the use of AI. The research draws on AI incident databases to understand their potential for law-making and compares legislation (focusing on the European Union's AI Act) with AI incident data. The databases used by the paper include: Where in the World is AI?, an interactive database managed by the Responsible AI Institute; The AI Incident Database developed by the Responsible AI Collaborative; and the AI, Algorithmic, and Automation Incidents and Controversies Repository, database containing incidents and controversies involving AI grouped by labels including type of harm, location, and system provider.

The first area of study is the Where in the World is AI? map, which reports incidents and contains general information about how AI is used in the world. The resource classified 323 out of 430 (75%) news articles written about AI as ‘harmful’, while only 22% were deemed to contain positive views about the technology. The paper suggests that "the media’s approach may contribute to the diffusion of a diffident attitude towards the use of AI that may also influence the evaluation of experts or policymakers involved in law-making." With respect to 'soft laws' such as ethical guidelines, the analysis found that 37% of 108 documents investigated included sentences indicating potential positive outcomes caused by AI. The study also found that the Where in the World is AI? map also advocated for a strategy of best practice analysis, which is present in the European Union's AI Act. We should note, however, that reporting itself is not an incident. While the tone and tenor of reporting shapes the way in which information is consumed, how best to delineate between the impact on the policy environment of reporting on incidents and the incidents themselves remains an open question.

Next, the paper draws on the AIAAIC repository to consider the different sectors most commonly affected by AI incidents. The analysis suggests that, aside from the 'technology' sector responsible for building and deploying AI, the second sector most commonly affected by AI incidents is 'government' (21.54%). While the paper notes that the high number of AI incidents associated with government usage is not reflected in soft laws (1.85% of the documents address AI applied in public administration), the AI Act categorises several types of AI applied in governmental services as 'high-risk' applications. While the study acknowledges that "it is not possible in this paper to assess how much the empirical reality of AI incidents in the government sector has influenced the EC strategy" it also proposes that it is "plausible that highly publicised events may have somehow affected the inclusion of some systems in the high-risk category."

The AIAAIC data show that a large proportion of reported incidents take place in the 'police' sector (39.8%), while the 'justice' sector was responsible for 5.6% of incidents. The paper notes that––despite a relative lack of incidents associated with the second category––the AI Act regulates different types of AI systems applied in the judiciary by classifying them as high risk. In doing so, the study suggests that this shows "an evident concern, partially corroborated by the empirical reality of incidents, towards such systems." Given the small number of incidents related to the use of AI in the legal system, however, it is difficult to determine the extent to which the decision to classify such systems as high risk is corroborated by the evidence presented by incident databases, partially or otherwise. Similarly, despite the high number of incidents regarding illegitimate surveillance (129 of the 871 incidents) within the AIAAIC database, the study found that ethical documents consistently overlooked the issue, with 13% of the documents investigated focused on regulating AI-based surveillance systems.

Conclusion: The case for incident analysis in AI policy

The paper makes a strong argument for viewing incidents as a significant drivers of the regulatory environment. The case for viewing AI incidents––in addition to uncertainty about the potential impact of the technology and its risks––as relevant factors for decoding the genesis of regulatory proposals is a compelling one. Perrow's Normal Accident Theory and the introduction of historical case studies focused on nuclear energy and the automotive industry provide a powerful lens through which to view the relationship between AI incidents and regulation. And while there is no 'smoking gun' whose presence neatly confirms nor repudiates the role of incidents in shaping policy, connecting AI incident data to specific governance proposals is a welcome attempt at analysing the process of regulatory development.

Read the full paper by Giampiero Lupo, Institute of Applied Sciences and Intelligent Systems, National Research Council of Italy here.