SHERPA’s technical notes on the EU AI Act
Understanding how artificial intelligence and big data will impact ethical and human rights issues is the main aim of EU’s Horizon 2020 SHERPA project. Outcomes of the project include recommendations for regulatory options in this space. On April 21 2021, the European Commission published its first draft of the Artificial Intelligence Act. This document proposes EU-wide legal and regulatory frameworks for uses and practices of artificial intelligence, and thus is of interest to the SHERPA project. The AI Act itself is still a work in progress and is being shaped by guidance from a diverse set of experts. The SHERPA project has already commented on the initial draft of this document [1] with respect to how well aligned the draft’s goals are with SHERPA’s high-level recommendations. This blog post reviews the AI Act draft from a technical perspective, with a focus on implementation details, interesting research topics, and cyber security.
Security and robustness of AI systems is an extremely important topic, and the first draft of the AI Act contains many recommendations and regulations that cover these issues. Applications that utilize artificial intelligence need to be trustworthy, and machine learning models can only be considered trustworthy if they are properly secured. SHERPA’s deliverables have included several reports covering the topic of security for machine learning. In deliverable 1.3, led by F-Secure, SHERPA presented a full review of the security issues, dangers, and implications of smart information systems [2, 3]. More recently, as part of SHERPA’s deliverable 3.5, F-Secure documented their research on a number of attacks against machine learning systems [4, 5] and studied mechanisms to defend against such attacks [6, 7]. Additionally, SHERPA’s research on the effects of manipulation on social network recommendation algorithms [8] was published by F-Secure.
The public are likely already familiar with the concept of adversarial attacks against AI systems from examples such as manipulated road signs that fool self-driving cars, and eyeglasses or makeup designed to trick facial recognition systems. Although these attacks are largely academic in nature, they represent the tip of the iceberg in terms of real-world attacks that can or could be mounted against machine-learning algorithms. Security for machine learning is already a topic of active research in both industry and academia, and we imagine it will gain a lot more momentum when the AI Act regulations come into effect.
Adversarial traffic signs. Source: https://medium.com/self-driving-cars/adversarial-traffic-signs-fd16b7171906
The AI Act defines a set of high-risk AI applications that will be subject to additional compliance requirements. Enforcement of these measures may include penalties for non-adherence. High-risk AI applications are defined in a short annex document that is subject to change on a regular basis. While we note that some high-risk applications defined in the annex are clear and well thought out, others may raise questions. One such example is 4a – “AI systems intended to be used for recruitment or selection of natural persons, notably for advertising vacancies, screening or filtering applications, evaluating candidates in the course of interviews or tests”. While it is clear that an AI tasked to decide whether an individual should be hired for a job or not is high-risk, it is not clear to us why an AI designed to decide where, how, and when to post online job adverts would be. We suggest that the set of high-risk applications be carefully considered and convincing explanations be provided for the inclusion of any specific application in the set.
The AI Act goes on to define a series of data quality requirements that will be especially enforced for applications defined in the high-risk category. One proposed requirement states that “all training, validation, and testing datasets be complete, error-free, and representative”. This requirement, understood literally, is unrealistic. Our stance is that data quality requirements must be achievable and clearly defined, especially for higher-risk use cases where severe consequences are attached to non-compliance. Indeed, properly defining data quality requirements is an interesting and very relevant research topic, given that some data sets in common use today (such as ImageNet [9], used to train image recognition models, and the ConLL2013 corpus [10], used to train natural language processing models) were recently found to contain multiple labelling errors. Data quality is also a very valid concern for recommendation systems that are often trained against real-time interactions created by the customers of a service. Recommendation systems can become skewed by popularity bias [11], and can be manipulated by the actions of fake accounts.
It is welcoming to note that the AI Act encourages transparency of AI systems through interpretability. Interpretability of an AI system is especially important in high-risk scenarios. For example, if an AI-based system makes a hiring decision, it should be possible to understand what factors went into that decision to prevent the possibility of bias or discrimination. When considering interpretability, defining the correct level of detail is also important and non-trivial. We all use computers in our everyday work, but not many of us understand how they work down to the most minute detail. In the same way, we should not expect the users of AI-based systems to understand them to the level of detail that an AI researcher would. Sufficient interpretability should allow a user to reasonably predict what should and should not happen given a set of inputs, but defining it unambiguously requires careful considerations.
The AI Act document includes a requirement to monitor a system’s performance on an ongoing basis that includes the need for logging and audits. Capturing and storing detailed logs of every input, decision, and output may, in some instances, require a large amount of storage space and possibly processing power. Such requirements may be financially prohibitive, especially for smaller companies, and thus these regulations may hinder EU business and startup competitiveness. We recommend that logging and audit requirements be accompanied by automated analysis methods able to detect problems in scalable and affordable ways (and thus reduce the need for storing a great deal of data). However, it is not clear whether the EU can or should mandate automated analysis for high-risk AI applications. While such a mandate may be beneficial, solutions in this space do not really exist currently, and the problem is perhaps too complex to be solved by catch-all mechanisms.
Finally, we approve of the AI Act’s initiative to designate a central EU body (Artificial Intelligence Board) whose task is to focus on making recommendations and issuing guidance to the EU and member states. We recommend that the board consist of a diverse enough staff that includes people with expertise in technical, regulatory, and ethical areas.
We are extremely happy with the fact that the EC is concerned enough with the possible downsides of AI that they are taking the situation seriously enough to propose regulations, especially with regards to the security and robustness of AI systems. The current AI Act draft proposals are thorough and well thought out. The addition of an annex document allows certain definitions (such as how AI is defined and what applications are high risk) to be updated as technological innovation progresses, without the need to update the main document. We are excited to see future drafts of the act and look forward to providing feedback and guidance where needed.
References:
[2] https://blog.f-secure.com/security-issues-dangers-and-implications-of-smart-information-systems/
[3] https://figshare.dmu.ac.uk/articles/online_resource/D1_3_Cyberthreats_and_countermeasures/7951292
[4] https://blog.f-secure.com/how-to-attack-distributed-machine-learning-via-online-training/
[6] https://blog.f-secure.com/poisoning-attacks-in-a-distributed-learning-environment/
[8] https://github.com/r0zetta/collaborative_filtering/
[9] https://arxiv.org/pdf/2103.14749.pdf
