Understanding Ethics and Human Rights in Smart Information Systems
How to Cite
Macnish, K., Ryan, M., & Stahl, B. (2018). Understanding Ethics and Human Rights in Smart Information Systems. ORBIT Journal, 2(2). https://doi.org/10.29297/orbit.v2i1.102
Introduction: Ethics and Human Rights Concerns of Smart Information Systems
In 2016 the Executive Office of the President of the United States published two reports covering possible consequences of the large-scale introduction of artificial intelligence (AI) into the US society and economy (Executive Office of the President, 2016a, 2016b). Similar reports were published around the world, for example by the UK House of Commons (House of Commons Science and Technology Committee, 2016), followed by the House of Lords (HoL, 2018), European Parliament (European Parliament, 2017) and many other legislative and governmental bodies. The reason why artificial intelligence attracts such a large amount of policy attention is that there are expectations that the integration of AI into economic and social processes is likely to have disruptive consequences and that policymakers will be called upon to react appropriately to the changes caused by AI. This is the currently accepted view and can be seen as political orthodoxy. AI is expected to change whole industries and bring huge economic benefits. At the same time, there are worries that AI will have significant ethical implications, may cause mass unemployment, discriminate against minority, disempower people and skew political processes.
While this description can be seen as the conventional wisdom of the day, it is typically not clear what it is based on and whether there is empirical evidence to support the benefits or the costs of AI. The power of the narrative is undeniable, but it is often difficult to assess where it originates and at which stage of this development we can reasonably assume to be. This lack of empirical is problematic, in particular for policymakers who are called upon to regulate the new technology but typically without a strong empirical basis on which policies can be based. While this state of affairs may be impossible to overcome completely, partly due to the complexity and mercurial nature of AI and partly because of the fundamental uncertainty of the future which is being legislated, it is nevertheless important to improve the evidence base of policy in this area. This editorial paper therefore sets out to provide the basis for the collection of such empirical evidence. It provides the shared conceptual and methodological basis of the 10 case studies that are published alongside this paper and should be read in conjunction with these individual case studies.
This editorial provides an example of how the much-applied case study approach can be made useful to gain a broader overview of topic as broad as that of ethical and human rights implications of SIS. By providing a detailed account of the development and implementation of the case study approach, the paper provides an example for similar wide-ranging case study work.
Following this introduction, the paper briefly defines the concept of SIS and justifies the choice of this term. It then explains the choice of the case study approach. The majority of the paper then describes the approach taken to identify suitable case study topics and the development of the methodology as summarised in the case study protocol. The paper ends with a discussion of lessons learned and an outlook of next steps.
Smart Information Systems
In this paper we use the term smart information systems or SIS to refer to a set of technologies and techniques that in current discourse are often referred to as artificial intelligence, machine learning, big data analysis and others. We do this to ensure that it is clear that we are not claiming to refer to a particular branch of these technologies but are more interested in their combination with other technologies and in particular in their social implications. The term AI is currently used widely and most of the policy-oriented publications that were cited earlier use some definition, such as
“Technologies with the ability to perform tasks that would otherwise require human intelligence, such as visual perception, speech recognition, and language translation”
There is, however, general agreement that there is no general agreement on an exact scientific definition of AI. As a recent study of AI on a global level by Elsevier (2018) has shown that the term AI covers a large number of existing fields of research but that in various domains (science, media, policy, education) there is little to no overlap in the use of the term. A typical strategy that policy-oriented reports use to define AI more narrowly is to highlight similar and related terms and current trends. It is probably uncontroversial to say that the current use of AI refers to a ‘narrow’ view of AI, which means that it aims at specific applications without suggesting that AI has general reasoning capabilities similar to humans (also known as ‘general’ AI). This narrow AI typically makes use of machine learning, more specifically based on neural networks and often deep neural networks. This type of technology can be characterised as a type of statistical analysis of data that is very good at pattern recognition in large data sets. It has come to prominence in recent years, even though the fundamentals of the technology and the algorithms have been around for decades, because the performance of affordable computers has increased dramatically at the same time, as the amounts of data collected for scientific, economic or administrative purposes has exploded. This led to a wide availability of large amounts of data (or ‘big data’) that was capable of being analysed through machine learning, which has led to impressive results.
We have decided to use the term SIS, following Stahl and Wright (2018), to indicate that we are not interested in a particular technology or data type. Instead, we are interested in the ethical, social and human rights aspects of these technologies. We therefore chose the term ‘information systems’, as this denotes a field that focuses on the use of information technology in organisations (typically, but not exclusively businesses) and aims to bring together technical and social expertise to understand the role of technology but also to provide lessons for effective practice (Avgerou and McGrath, 2007). We are not interested in any type of information system, but specifically those systems that relate to AI, which is why we chose the term ‘smart information systems’. We hope that this choice narrows the field of investigation sufficiently to allow us to cover it sufficiently, while at the same time being open enough to cover phenomena of interest.
The subject area of SIS is clearly broad. Our interest is in providing empirical evidence that can inform policy development. The next question we considered is therefore which methodological approach would allow for an adequate coverage of SIS.
The Multiple Case Study Approach to SIS
The motivation behind the research leading to this paper was to better understand the ethical and human rights aspects of SIS. Research in this area has grown exponentially in recent times, leading not only to the policy discourse referred to earlier, but also to a number of monographs (Mayer-Schonberger and Cukier, 2013; O’Neil, 2016), academic publications (Floridi et al., 2018; Mittelstadt and Floridi, 2016; Winfield and Jirotka, 2018) and contributions from companies and other organisations. Our starting point was the observation that, despite the high level of activity in this field, there is very little systematic empirical investigation of the nature and consequences of these technologies in their social environment. Our research question is therefore: how can ethical and human rights aspects of SIS be investigated?
There are of course numerous ways of doing such research that could include large-scale surveys, the analysis of existing data sets, including national statistics, literature analysis and many more. For our purposes it is important to collect data that not only provides an insight into the phenomenon in question but also that the outcomes can be communicated to an interested audience, including to policymakers. In addition, the research area is such that there is a large amount of literature on what might constitute ethical or human rights issues of SIS, but the phenomenon of interest for our investigation is how these are instantiated in practice. Part of the research is therefore to remain vigilant for gaps in the literature and use the research to produce additional insights which are not widely discussed yet.
For these reasons we decided to use a case study approach. Case studies are a well-established methodology in social and organisational studies (Eisenhardt, 1989; Yin, 2003a, 2003b). in our study the purpose of the cases was not so much the testing of hypotheses but the exploration of new phenomena with a view to contributing to policy development. We therefore decided to pursue case studies employing an interpretive paradigm (Klein and Myers, 1999; Myers and Walsham, 1998; Walsham, 1995). The clear downside of this approach is that cases only produce punctual insights, which was counter to our aim of contributing to a better understanding across the broad range of SIS. We therefore decided to undertake a set of 10 parallel case studies that would allow us to shed light on a range of technologies as well as ethical issues and human rights. The following sections explain in more detail how we developed this idea and turned it into a practical method.
Background: Developing Criteria
The study was undertaken in the context of an EU-funded project (SHERPA, www.project-sherpa.eu) which determined some of the boundary conditions of the research. It meant that the case studies had to be conducted by a diverse and distributed team of partners with different backgrounds and areas of expertise and that they had to be done in a rather tight time frame of eight months.
The process of case study development began in March 2018. Having successfully won the SHERPA contract, attention turned to understanding what would be involved in the case study process. It was decided that a number of criteria needed to be established early on to ensure the successful development and production of ten high-quality case studies. The work package leader (the University of Twente (UT)) created a process to distinguish the different types of case studies that should be carried out. This included a broad literature study of the main ethical issues in Big Data and artificial intelligence (AI), to determine possible concerns to be uncovered and addressed in the case studies, and the creation of an example case study for consortium partners to follow.
Several brainstorming sessions occurred at UT between March and April 2018 regarding the means of distinguishing the different case studies to effectively cover a broad reach of areas and ethical issues surrounding the use of SIS in practice (see Fig 1). One approach considered was to distinguish case studies by means of technical classification, and particularly by reference architecture. Hence case studies could have been distinguished along the lines of relational versus non-relational, batch processing versus real-time processing, or data type, to list a few examples. A 2nd means of distinguishing case studies considered was by ‘user type’. Hence case studies could have been distinguished by whether the user was an individual, a business, an individual within a business, or a process application. The 3rd means of distinguishing the case studies was by application domain. This focused on the field in which the SIS would be applied. Hence distinguishing features might include whether the application was a social network, a business system, or a part of the Internet-of-Things (IoT).
Fig. 1. Image of Brainstorming at University of Twente
Following further discussion, it was agreed that at the very least, case studies should include employees, consumers, citizens, and governments. The final set of case studies was then determined to encompass both targets and application domains, at least one case per type of data user, and a distribution of cases over the different technical categories. However, it was acknowledged that this would lead to an ideal situation, and one which may need to be tempered with the availability of willing case study participants. It was decided to establish a list of different social domains using and integrating SIS, with the attempt of covering a wide number of these areas within the case studies. The final agreed list of application domains can be seen in Table 1.
Having agreed the ideal list of application domains, attention turned to gaining a broad understanding of ethical issues currently faced across these domains. A literature review was carried out which approached both concerns with Big Data and AI in general and at the application level. It was sent to partners to understand the many ethical issues surrounding SIS. Finally, a review of the technologies employed in Big Data and AI was also carried out, so that partners were provided with technical knowledge and understanding of SIS before conducting domain specific research for their case studies and also preparation for their interviews with experts in those areas.
Alongside the literature review, UT carried out a review of the applications of SIS, along with a review of potential consortium partners who may be suitable for working on certain social domains from their expertise, research areas, and interests (prior to the May 1st launch date). Furthermore, UT also conducted a broad domain analysis of SIS applications. Through a combination of desktop research and personal contact lists, UT identified a number of organisations from the 16 social domains that use and integrate SIS in Europe. This list was distributed to SHERPA consortium partners as a suggestion of organisations to use for their case studies. Finally, an example case study was developed on Google DeepMind and the Royal Free Trust and was shared in May 2018.
|Banking and securities|
|Retail and wholesale trade|
|Energy and utilities|
|Manufacturing and natural resources|
|Communications, media and entertainment|
|Employee monitoring and administration|
|Law enforcement and justice|
|Defence and national security|
Table 1: Social Domains Identified
Preparation of Case Study Research
The next phase of the case study research started in May 2018 when partners involved in the cases met and discussed what the case studies should look like. Partners needed to understand the potential domain interests across the consortium. It was also discussed if partners had contacts in those domains or could think of organisations that would be suitable case study participants.
At the launch event the example case study was presented, alongside the application domains. There was considerable discussion as to which domains partners should focus on. The launch presented an opportunity to connect with different partners and understand respective interests. One of the objectives at the launch event was to correlate partner interests and connections with the application domain criteria which had been developed in advance.
Once the application domains had been agreed, a further decision was taken about whether certain application domains should be used as case studies (see Table 2). It is important to point out that in parallel to the case study research the consortium also developed a set of scenarios. These are not subject to this paper and will be described elsewhere. They are important to reference, however, as they form part of the attempt to gain a general overview of the field. This means that some of the topics or technologies of interest that were not covered by the case studies were allocated to the scenarios (see Table 3). The main difference between case studies and scenarios is that case studies refer to current technologies that are being used, whereas scenarios are likely future technologies that are not yet used or at least not on a large scale. There was a discussion of which domains would fit better as a case study or scenario. For example, defence and national security was determined to be better suited as a scenario. By contrast, cyber security, in which one of the consortium partners was a dedicated expert, was deemed to be more suitable as a case study. The initial agreement also covered the amount of time that could be spent on each case study and the process of collaboration which included a weekly teleconference.
Some of the case studies try to provide a broad analysis of a particular domain, most of the case studies delved into a specific area or topic within their domain of choice in order to provide a more detailed and interesting case study. For example, some case studies focused on one organisation to provide a broad overview of some SIS issues in that domain, such as: BASF for agriculture (CS03) and Liander for energy (CS07). Other case studies focused on a particular category within that domain to provide a more nuanced examination of SIS in that area, such as: a municipality for the government case study (CS02); smart cities as an aim within the Sustainable Development Goals (CS04); a large scientific research project (CS05); health insurance (CS06); and a supply-chain management risk prevention company (CS10).
Some of the case studies focused on specific applications, technologies, or departments within their domain of research: Internet-of-Things for employee monitoring (CS01); customer relations management in retail (CS09); and the use of SIS in cybersecurity department of a telecommunications provider (CS08). Overall, the case studies cover a wide range of areas, domains, specialisations, technologies, and approaches in the use and implementation of SIS, but were still able to provide cohesion and scientific rigour by following the case study protocol we completed before the case studies commenced.
|No.||Case Study Social Domain|
|CS01||Employee monitoring and administration|
|CS07||Energy and utilities|
|CS08||Communications, media and entertainment|
|CS09||Retail and wholesale trade|
|CS10||Manufacturing and natural resources|
Table 2: Case Study Division
|No.||Scenario Social Domain|
|SC02||Defence & National Security|
|SC05||Law Enforcement and Justice|
Table 3: Scenario Division
Developing Case Study Protocol
Having agreed on the topics of the cases to be developed and the attribution of work, the next step was to agree on the detailed methodology and steps to be taken. Following common academic nomenclature, we called this the ‘case study protocol’ (Yin, 2003a). The case study protocol was created to guarantee consistency of approach and timeliness of the case studies.
A compromise needed to be agreed between an ideal number of interviewees, the ability to find willing participants, and the time constraints of the work package. In the end, a minimum number of one interviewee was agreed upon – if that individual was knowledgeable on both the technical and ethical use of SIS – with a recognition that two or more interviewees would be better. The case study protocol was officially agreed on 16th August 2018, outlining a specific breakdown of tasks and deadlines to ensure successful completion of 10 case studies by January 31st (see Table 4). Following the finalisation of the protocol, a promising case study was identified to be used as a pilot (agriculture case study). This pilot served as the template to ensure consistency of style and format, as well as common approaches to content. In addition, case study authors were required to upload interview transcripts to NVivo for analysis.
A final consideration was the need for ethics approval by research ethics committees at participating universities. The project lead (UT) submitted the case study protocol to the University’s research ethics committee in July 2018, and received a favourable response within 2 weeks. This allowed for the case studies to be carried out by UT with full ethical approval, as well as case studies by non-university partners. The other universities needed to gain ethics approval from their own research ethics committees. Given that research ethics approval is required prior to interviews taking place, it was important that the research ethics process was completed by the end of July 2018. A significant aspect of gaining ethics approval was the development of a consent form and information sheet as part of the case study protocol, which could be used by partners prior to conducting interviews (see Appendix B.1 and B.2).
|27/06/2018||Identify partner’s Principal Investigator/POC and Co-Investigator/Deputy POC for case study|
|20/07/2018||Confirm agreement with organization(s) to participate in case studies in writing|
|03/08/2018||Carry out preliminary background research on organisation and its use of SIS|
|07/09/2018||Obtain written informed consent from interviewees|
|12/10/2018||Conduct preliminary analysis (see guide for final report, below)|
|09/11/2018||Send agreed draft report to UT|
|30/11/2018||Discuss draft report with UT|
|21/12/2018||Write up final report using template and send to UT and the QA officer|
|14/01/2019||Discuss final revisions with QA officer|
|31/01/2019||UT to undertake cross-case analysis and compile final report|
Table 4: Research Plan
Having agreed on the principles of collaboration, we now describe execution of the case study research. The full case study protocol that guided this work is available in Appendix A.
Conducting Case Studies
Conducting the case studies involved carrying out background research, both generic regarding the application domain, identifying interviewees, and research on the organisation(s) that they belong to, and conducting the interviews. These stages lasted from June 2018 (the beginning of general background research) until October 2018. The project lead rigorously kept track of partners’ progress, through an Excel Matrix, throughout this process to ensure deadlines were met (see Fig 2).
Fig 2: Case Study Progress
General Background Research
For each case study, existing as it did in its own application domain, general background research needed to be carried out. This came in addition to the more general research on ethical issues in Big Data carried out prior to the start of the project. For each case study, the author investigated articles on ethics and legal issues for the particular application domain across a range of media. These included academic journals, trade journals, and Internet searches. In the process of the background research, both primary and secondary interview candidates were identified and contacted. Once initial contact had been established, permission for an interview was sought, alongside sending the potential interviewee the case study protocol, an information sheet, and a consent form (see Appendix B.1 and B.2).
Specific Background Research
Once interested persons had been identified and confirmed to be interviewed, more targeted background research could be conducted. This research focused on the particular company or organisation to which the interviewee belonged. The specific background research enabled the case study author to gain a fuller understanding of the issues faced by the organisation which was to form a part of the case study and to be informed of likely issues during the interview. It also meant that the interviewer could incorporate questions retrieved from their background research into the interview, rather than solely following the research questions from the case study protocol.
Where possible, interviews were conducted at the place of work of the interviewee, so that they felt comfortable and were not inconvenienced by travel. Provision had been made for interviews to be held over Skype or by phone, in instances where an interviewee was too far away, or they opted to do it remotely. The interviewer(s) provided the information sheet and consent form to the participant days or weeks before the interview, and also prior to the commencement of the interview itself. These were explained to the interviewee before they signed the consent form.
One issue which was not discussed in advance, and from which the case studies would have benefited, was an interview methodology. Several of the case study authors had little or no experience in conducting social science interviews. As a result, some case studies were based on an interview following the precise questions found in the case study protocol, while other case studies took a more open interview style, covering many topics but in a less dogmatic way than indicated in the protocol. Furthermore, some interviews were carried out on a strictly one-on-one basis, while others were conducted in a workshop format involving one or two interviewers and up to three interviewees (for example, the telecommunications case study).
Over time, a high degree of flexibility was also required: Organisation who had nominally agreed to be interviewed pulled out (social media case study); some application domains were very difficult to find willing interviewees (energy case study); while some domains even proved to be impossible to find participants (banking & finance). These challenges were countered by changing organisations to those who were willing to be interviewed or changing the domain area entirely.
Following each interview, the case study author scanned and uploaded a copy of the signed consent form to a central secure location on our website. UT ensured that all documents were uploaded by the partners and that it was clear what the interviewees had consented to in their agreement. Notes from each interview were consolidated, and a thank you email sent to each interviewee. They were also informed that they would receive a first draft of the report once it was complete.
Reporting on Case Studies
The reporting process on the case studies began in October 2018. This consisted of having each interview transcribed and analysed in NVivo. Following this, a draft of the full case study was composed and sent to the interviewee for confirmation. The draft was then sent to WP 1.1 leader (UT) for peer-review. The review document was then returned to the initial author, who had until the end of December to return a final draft. This draft was then submitted for the QA process in January 2019, which resulted in further edits being required from each case study author by mid-January. Finally, the final version was sent to the deliverable coordinator in the last week of January. This enabled the writing of the final report by the deadline of 31 January 2019 (see Fig 3).
While each partner was responsible for the transcription of their interview(s), a transcription service was identified which could transcribe the interviews quickly and at a very competitive price. This company was recommended by several of the partners within the consortium and had a strong track-record in the industry. The structure of these transcriptions also allowed partners to analyse their data in a more consistent way in NVivo.
Data Analysis using NVivo
It was decided to follow normal qualitative data analysis procedures (Charmaz, 2006; Miles and Huberman, 1994). The starting point were the ethical and human rights issues identified in the literature review. In order to remain open to new issues, all data concerning a case study would be analysed using these nodes, but allowing case study authors to add and amend these. In order to ensure that this would be done consistently, the consortium decided to make use of a well-known qualitative data analysis tool, NVivo server 10. However, not all case study authors were familiar with interpretive and qualitative data analysis or the analysis tool. Therefore, a workshop was arranged for the 2nd week of October to introduce case study authors to NVivo and provide a focused period of time for the transcribed interviews to be analysed. The workshop environment meant that the methodology for using NVivo (such as whether to encode concepts or precise phraseology) could be discussed and agreed as a group. Prior to the workshop, comprehensive notes on the use of NVivo were compiled and sent to case study authors.
The NVivo workshop proved to be a strong success, with most of the nodes for the case study analysis being created over the two-day period. There were some technical issues with the system which were ultimately overcome.
Writing, Reviewing and Editing Drafts
Following the NVivo workshop, case study authors had 3 weeks to compile the first draft of their case study. This essentially involved pulling together background research and the interview to identify what the main ethical issues facing that particular organisation had been, and whether the interview had raised ethical issues which had not been identified in the literature.
Most case study authors were able to submit a first draft of their report by the first week of November. At this stage, each case study was returned to the interviewee for approval. This was to ensure that the interviewee agreed on the content of the interview as presented. However, most interviewees were extremely busy and did not respond to the case study author. Given that the case studies had already received consent in advance of the interview, it was decided that interviewee confirmation of the case study was good to have but not essential, as it could have led to indefinite delays on the finalisation of each case study. However, it was also agreed to anonymise the interviewee unless there was a specific written consent by the interviewee to include their names in the final document.
Following its submission to the project leader, each case study was subject to peer review by two academics at the University. The peer review process looked at a number of issues, including quality of content, quality of English, and consistency of style across all of the case studies. When the case study had been reviewed it was returned to the author for appropriate edits. On occasion, it proved more time efficient for the peer reviewers to make stylistic changes themselves and for the author to confirm using track changes. The revised version was returned by mid-December.
The QA process took place in January and performed the function of a third peer review and highlighted further issues which had not been recognised previously. Once each case study had been reviewed by the QA process it was returned to its originating author for final edits before being sent on to the leader of the case study task and the project coordinator. The task leader then combined all of the case studies into a single deliverable for submission to the European Commission by 31st January 2019, and the project coordinator conducted final edits before submitting them to the project workbook and the ORBIT Journal.
One question uncovered by the QA process concerned the intended audience of the case study which had ramifications for their formatting. On the one hand, the intended audience was the European Commission, and through them practitioners who would be looking at the case studies as a means of understanding ethical issues within their own particular application domain. On the other hand, the case studies had been carried out in a rigorous manner such that they were suitable for publication in a social science journal. This meant that for each case study there were 2 intended audiences: academic and non-academic. It was decided that the case studies should be accessible for both academic and non-academic audiences. In order to entice non-academic readers to engage with the cases and to render them more accessible, the authors were encouraged to use visual elements and limit the length of the text. As a result, the cases published in this special issue of the ORBIT journal are more visually diverse than is normally the case for academic journals, including the ORBIT journal itself.
This paper describes the development of 10 case studies of ethical and human rights issues in SIS, thus making an important contribution to the larger discussion of AI and big data. These SIS case studies bring the first large-scale empirically-grounded analysis of what is happening across a number of sectors in Europe, such as government, health care, cybersecurity, telecommunications and insurance. A number of key challenges were faced in the design and development of the 10 case, but along with this quite a few lessons have been learnt. This concluding section will look first at the challenges faced and then lessons learnt, which should be taken as recommendations for future case study management and development.
There were a number of challenges faced throughout the project. Contributing to many of these was the lack of experience of many of the case study authors. Of the 10 case studies, 6 were carried out by authors with little or no social science experience/background. This meant that these authors had to be trained in data collection (interviews) as well as data analysis principles and techniques.
A second challenge was the lack of time available to develop the case studies. It was agreed at an early stage that the case studies would be stronger with more interviewees. However, the realities of the time constraints provided with the work package meant that it was difficult to do this.
The third challenge was the time assigned to partners for writing the case studies. For some partners, this proved to be too demanding, and there was some shuffling of PMs between partners to focus efforts on a handful of.
A fourth challenge was that some partners dropped out of the process owing to red tape issues within their organisations. In yet other cases, no partners were found within the application domain at all. This was particularly true of the banking and securities sector, in which a case study had been envisioned to focus on the use of SIS in trading of securities.
Technical problems with the NVivo software formed a fifth challenge. As servers on which NVivo was hosted were migrated, crucial information was lost, leading to work having to be duplicated. Most of the information was regained from backups, but this delayed the work.
The sixth challenge was the focus on intended audience. The case studies had been conducted with a primarily academic audience in mind, with a view to the publication of the case studies in a special issue of an academic journal. However, it was decided to make the case studies attractive and approachable for non-academics, and so included elements that an academic paper would not necessarily concern itself with, such as pictures, charts, and tables.
|Experience||Lack of social science experience among some of the case study authors|
|Time||Lack of time for more interviews within organisations and additional organisations|
|Planning||Insufficient time allocated to some case study authors|
|Participation||Difficulty identifying, confirming, and liaising with case study participants|
|Technical||NVivo software crashing, incompatibility with Apple computers|
|Audience||Contradictory intended audiences|
Table 9: Challenges encountered during the case study research
The initial step of drafting the case study protocol, while not ideal, went a long way to shaping expectations and providing consistency for the case studies. This consistency was aided further by the identification of a pilot case study at a very early stage in the process. Although this meant that the author of the pilot case study had to work harder than others in terms of keeping ahead of the pack, it did mean that consistency could be achieved more effectively.
The weekly case study Skype calls were also crucial to the successful completion of the case studies to a high standard within the time allotted. These are both provided a sense of team coherence and avoided case study authors feeling neglected or ignored. They also introduced an element of accountability on a regular basis which meant that few case studies fell behind, because each author knew that they had to give an account about their progress on a weekly basis.
The data analysis workshop was another notable success within the project. In conjunction with the weekly Skype calls, this led to the case study authors meeting in person for the first time as a group. It allowed for collective discussion on methodology and approach, and ensured that the majority of the case studies were all at a suitable point of progress by mid-October 2018.
Finally, the project leader was supportive throughout the process of the development, writing, and reviewing of the case studies. He attended a significant number of the weekly Skype calls, making himself available for clarification purposes. He provided encouragement and support, rather than attempting to be disciplinarian. He was understanding towards the partners with no social science experience and was very accommodating to reviewing the case studies. This was very much appreciated by the case study authors and provided the effective guidance required for high-quality case study reports.
This account of the case study research has formed the basis of all case studies that are published in the same special section of the ORBIT journal. One central reason for providing this account is that it removes the onus from the individual case studies to report on methodology, leaving them free to focus on the substantive findings. The individual 10 case studies that were originally included in the project deliverable are published as separate papers, following a final round of review.
The special section stops here and provides the full cases. The obvious next step that will be reported elsewhere is the analysis across the 10 cases. This multiple case study exercise was conducted to collect insights across the broad range of SIS and a cross-case analysis is called for, to extract what those broader insights are. This work is under way and will be made available in an appropriate venue in due course.
This SHERPA project has received funding from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement No. 786641. The authors acknowledge the contribution of the consortium to the development and design of the case study approach.
Avgerou, C., McGrath, K., 2007. Power, Rationality, and the Art of Living Through Socio-Technical Change. MIS Quarterly 31, 295–315.
Charmaz, K.C., 2006. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Sage Publications Ltd.
Eisenhardt, K.M., 1989. Building Theories from Case Study Research. The Academy of Management Review 14, 532–550. https://doi.org/10.2307/258557
Elsevier, 2018. ArtificiaI Intelligence: How knowledge is created, transferred, and used – Trends in China, Europe, and the United States. Elsevier, Amsterdam.
European Parliament, 2017. Civil Law Rules on Robotics – European Parliament resolution of 16 February 2017 with recommendations to the Commission on Civil Law Rules on Robotics (2015/2103(INL)).
Executive Office of the President, 2016a. Artificial Intelligence, Automation, and the Economy. Executive Office of the President National Science and Technology Council Committee on Technology.
Executive Office of the President, 2016b. Preparing for the Future of Artificial Intelligence. Executive Office of the President National Science and Technology Council Committee on Technology.
Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., Luetge, C., Madelin, R., Pagallo, U., Rossi, F., Schafer, B., Valcke, P., Vayena, E., 2018. An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations. Minds and Machines.
HoL, H. of L., 2018. AI in the UK: ready, willing and able?, Report of Session 2017–19. Select Committee on Artificial Intelligence, London.
House of Commons Science and Technology Committee, 2016. Robotics and artificial intelligence.
Klein, H.K., Myers, M.D., 1999. A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly 23, 67–93.
Mayer-Schonberger, V., Cukier, K., 2013. Big Data: A Revolution That Will Transform How We Live, Work and Think. John Murray, London.
Miles, M.B., Huberman, A.M., 1994. Qualitative data analysis: an expanded sourcebook. SAGE.
Mittelstadt, B., Floridi, L., 2016. The Ethics of Big Data: Current and Foreseeable Issues in Biomedical Contexts. Science and Engineering Ethics 22, 303–341. https://doi.org/DOI 10.1007/s11948-015-9652-2
Myers, M.D., Walsham, G., 1998. Exemplifying interpretive research in information systems: an overview. Journal of Information Technology 13, 233–234.
O’Neil, C., 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Penguin UK.
Stahl, B.C., Wright, D., 2018. Ethics and Privacy in AI and Big Data: Implementing Responsible Research and Innovation. IEEE Security Privacy 16, 26–33. https://doi.org/10.1109/MSP.2018.2701164
Walsham, G., 1995. Interpretive case studies in IS research: nature and method. European Journal of Information Systems 4, 74–81. https://doi.org/doi:10.1057/ejis.1995.9
Yin, R.K., 2003a. Case Study Research: Design and Methods, Third Edition. ed. Sage Publications, Inc.
Yin, R.K., 2003b. Applications of Case Study Research, Second Edition. ed. Sage Publications, Inc, Thousand Oaks.
Appendix A: Case Study Protocol
We used exploratory case studies to uncover ethical issues which arise through the use of SIS across a number of different types of organisations. From the individual case studies, we will take a cross-case analysis to build a matrix of ethical issues experienced by different organisations. The primary research question was: How do organisations perceive ethical concerns related to SIS and in what ways do the organisations deal with them?
We believed that there would be discrepancies between how companies deal with the ethical considerations related to the SIS they employ. While some companies would search for the ethical considerations and deal with them well, some would search for them and deal with them poorly, others would search for them and not find any (even though they are present), while a fourth group would not search for them at all. We wanted to gain evidence from of each of these cases to address the empirical plausibility of the research question.
Researchers need to decide on the research approach that most accurately addresses their research questions. Epistemology is a viewpoint about the nature of enquiry, the kind of knowledge discovered or produced, and the kind of strategies that are consistent with this (Becker and Niehaves, 2007). Our research sought to understand the ethical and human rights issues arising from SIS from the perspective of those developing it. Our case studies adopted an interpretivist approach, which allowed for both rigour and flexibility (Walsham 1995). It enabled researchers to empathise with research participants and seek to elicit insights, rather than an objective truth. This allowed us to understand ‘reality’ as the blending of the various (and sometimes conflicting) perspectives which coexist in social contexts, the common threads that connect the different perspectives and the value systems that give rise to the seeming contradictions and disagreements around the topics discussed. Whether one sees this reality as static (social constructivism) or dynamic (social constructionism) was also a point of consideration, as they both belong in the same “family” approach where methodological ﬂexibility is as important a value as rigour.
With interpretivist approaches, it is often difficult for novice researchers to maintain consistency and coherence, as is for untrained evaluators to assess its rigour. Explicating the methods, protocols and procedures to be followed, as well as the reaching a common understanding of the semantics that underlie the research is of paramount importance. Hence, the following section explains in detail the methodology employed for the collection and analysis of the research material and their synthesis with the view to theorise around the ethical and human rights issues arising from SIS.
The methodology that was used to answer the research question would be through qualitative case studies (Yin, 2015, 2003). As such the case study is defined by the interest in individual cases, which are the object of the research (Stake, 2005). The cases that were used in the SHERPA project were contemporary and complex functioning units, investigated in their natural context. The essence of using a case study methodology is that it allows for triangulation in coming up with a matrix of ethical issues experienced by different cases. Also, the qualitative case study methodology was ideal because of the descriptive and exploratory nature of the research which is dominated by ‘how’ and ‘what’ questions in gaining insights on the ethical issues that result from SIS and how they are dealt within respective cases.
We looked for diverse examples of SIS use which will highlight both common issues and anomalous issues across the cases. Each case study is exploratory, aiming to uncover the insights of organisational representatives on the matter and possibly the position of their organisation as a result. The case studies adopted an interpretivist approach, which allows for rigour in the methodology, but also for the required flexibility (Walsham 1995). The unit of analysis for each case study was the use of SIS in the organisations (cases) investigated, be these corporate, governmental, NGO, project or other. We examined the cases from multiple lenses and levels of analysis (Rowley, 2002) in order to provide a holistic understanding across organisational levels, disciplines and technology domains.
In order to explore how different cases or organisations perceive ethical concerns related to SIS and in what ways they deal with them, the following types of data sources were used:
- Documents (e.g. organisation website and publications, company records, project documents and memoranda, illustrative materials (newsletters, publications that form part of the history), archival records) (see Table 5)
- Interviews (at least 2, open-ended, semi-structured, questions below – face-to-face is best, otherwise telephone/Skype) to be transcribed (see Table 6)
There was a minimum of one interview per case study from different professional disciplines (i.e. a business representative and an IT representative). It was important to address the interview questions to those most able to answer them (e.g. a Chief Technical Officer – CTO). However, in most cases, it was also valuable to speak to a Policy Advisor or others at the organisation who can address the impact of the technology on society. In some cases, the CTO was in a position to address all of the questions sufficiently (e.g. the government case study). Beyond the interview(s), supporting/supplementary information was required to develop a contextual understanding of the case study and lend credence to (or criticise) the responses of the interviewee.
|1||In which sector is the organisation located (e.g. industry, government, NGO, etc.)?|
|2||What is the name of the organisation?|
|3||What is the geographic scope of the organisation?|
|4||What is the name of the interviewee?|
|5||What is the interviewee’s role within the organisation?|
Table 5: Desk Research Questions
|1||What involvement has the interviewee had with SIS within the organisation?|
|2||What type of SIS is the organisation using? (e.g. IBM Watson, Google Deepmind)|
|3||What is the field of application of the SIS (e.g. administration, healthcare, retail)|
|4||Does the SIS work as intended or are there problems with its operation?|
|5||What are the innovative elements introduced by the SIS (e.g. what has the technology enabled within the organisation?)|
|6||What is the level of maturity of the SIS? (i.e. has the technology been used for long at the organisation? Is it a recent development or an established approach?)|
|7||How does the SIS interact with other technologies within the organisation?|
|8||What are the parameters/inputs used to inform the SIS? (e.g. which sorts of data are input, how is the data understood within the algorithm?)|
|9||What are the principles informing the algorithm used in the SIS (e.g. does the algorithm assume that people walk in similar ways, does it assume that loitering involves not moving outside a particular radius in a particular time frame?)
|10||Are there policies in place governing the use of the SIS?|
|11||How transparent is the technology to administrators within the organisation, to users within the organisation?|
|12||Who are the stakeholders in the organisation?|
|13||What has been the impact of the SIS on stakeholders?|
|14||How transparent is the technology to people outside the organisation?|
|15||Are those stakeholders engaged with the SIS? (e.g. are those affected aware of the SIS, do they have any say in its operation?)
If so, what is the nature of this engagement? (focus groups, feedback, etc.)
|16||In what way are stakeholders impacted by the SIS? (e.g. what is the societal impact: are there issues of inequality, fairness, safety, filter bubbles, etc.?)|
|17||What are the costs of using the SIS to stakeholders? (e.g. potential loss of privacy, loss of potential to sell information, potential loss of reputation)|
|18||What is the expected longevity of this impact? (e.g. is this expected to be temporary or long-term?)|
|19||What has been the impact of the SIS on stakeholders?|
|20||Are those stakeholders engaged with the SIS? (e.g. are those affected aware of the SIS, do they have any say in its operation?)|
|21||If so, what is the nature of this engagement? (focus groups, feedback, etc.)|
|22||In what way are stakeholders impacted by the SIS? (e.g. what is the societal impact: are there issues of inequality, fairness, safety, filter bubbles, etc.?)|
|23||What are the costs of using the SIS to stakeholders? (e.g. potential loss of privacy, loss of potential to sell information, potential loss of reputation)|
|24||What is the expected longevity of this impact? (e.g. is this expected to be temporary or long-term?)|
Table 6: Interview Research Questions
Recording and management of data
The interviews were recorded using a suitable tool that allowed access in a broadly accessible format (e.g. mp3). All interviews were transcribed by the researcher or by a reputable third-party company (UKTranscription). The individual undertaking the interview had to confirm and validate the transcription. Once the interview data has been analysed, the researcher deleted the audio recordings. All data including the interview transcripts were stored and logged in secure group repository (WordPress and NVivo). The audio files were deleted, once the analysis of the interview was completed.
The collected data was analysed using a thematic analysis technique. In using a thematic analysis, we were able to highlight, expose, explore, and record patterns within the collected data. The themes are patterns across data sets that are important to describe several ethical issues which arise through the use of SIS across a number of different types of organisations. In addition, NVivo was used as a tool for the data analysis. UT will also engage in a cross-case analysis of the ten different case studies, which will draw together ethical issues arising in the ten case studies. This will then form the backbone for the ethical analysis in WP1.4.
Guide for Final Report
Each case study culminated in a final report consisting of an analysis and identification of core ethical issues that arise in the case study. The issues included likely and unlikely ethical concerns (see Table 7). For consistent reporting across the 10 case studies, a template for the final report was provided (see Appendix C).
|Ethical Issue||Question Example||√|
|Privacy||Does the use of the technology raise concerns that people’s privacy might be at risk or endangered?|
|Personal Data||Does the technology or its use presume a particular group or person “own” the data? If so, who?|
|Security||Does the technology use personally-identifying data? If so, is this data stored and treated securely?|
|Inclusion of stakeholders||Are people affected by the technology involved in any way with its use or implementation? Do they have an opportunity to have a say in how the technology impacts them?|
|Consent of stakeholders||Have people affected by the technology been given an opportunity to consent to that technology existing or having the impact that it does on their lives?|
|Loss of employment||Does the use of the technology put people’s jobs at risk, either directly or indirectly?|
|Autonomy/agency||Does the use of the technology impact in any way on people’s freedom to choose how to live their lives?|
|Discrimination||Can/does the technology or its use lead to discriminating behaviour in any way? Does the technology draw on data sets that are representative of those stakeholders affected by the technology?|
|Potential for military/criminal/
|Could the technology be used for military, criminal or other ends which were not envisaged or intended by its developers?|
|Trust||Does the technology impact people’s trust in organisations, other people, or the technology itself?|
|Power asymmetries||Can or does the technology exacerbate existing power asymmetries by, for instance, giving a large amount of power to those already holding power over other people?|
|Inequality||Can or does the technology reduce inequalities in society or exacerbate them?|
|Fairness||Is the technology fair in the way in which it treats those affected by it? Are there unfair practices which arise in relation to the technology?|
|Justice||Does the technology or its use raise a feeling of injustice on the part of one or more groups affected?|
|Freedom||Does the technology or its use raise questions regarding freedom of speech, censorship, or freedom of assembly?|
|Sustainability||Is the technology or its use sustainable, or does it draw on limited natural resources in some way?|
|Environmental impact||Does the technology have any impact on the environment, and if so what?|
Table 7: Checklist of Ethical Issues
Outline of narrative
The final report took a narrative structure. In constructing the final report, partners addressed the hypothesis to avoid drowning in the data (there were many aspects arising in the course of the research which were not strictly pertinent to the ethical issues in the case study). The analysis rested on all of the relevant evidence, rather than “cherry picking” certain aspects. For instance, it may be possible to see a privacy violation occurring, but the organisation in question may have also recognised this and taken steps to resolve it. In such cases, both the potential violation and the mitigation were noted. This was not intended to be a comprehensive report on everything uncovered in the course of the research, but a report on the ethical issues. Ultimately, “the goal of the report is to describe the study in such a comprehensive manner as to enable the reader to feel as if they had been an active participant in the research and can determine whether or not the study findings could be applied to their own situation. It is important that the researcher describes the context within which the phenomenon is occurring as well as the phenomenon itself.” (Baxter & Jack 2008, p555).
We identified a number of potential issues that were to be addressed during the interviews, qualitative analysis and writing up of the reports (see Table 8).
|Issue||Explanation of Issue|
|Unbounded||We are interested in ethical issues arising from the use of SIS in organisations. Hence, we want information relating to the use of SIS and ethical issues that the organisation has recognised and either dealt with or chosen to ignore|
|Incomparable||It is extremely important that the case studies be comparable. We are therefore adopting a positivist framework and using this case study protocol to inform the design of every case study to ensure comparability. This will be supplemented by bi-weekly meetings to discuss progress and share experiences. “The rigour off case studies should therefore be judged by the same criteria [as any other empirical, scientific method] of internal validity, external validity, construct validity, and reliability” (Yin 1992 p124)|
|Contact does not have access to relevant material||Raw data is highly valuable in these case studies to allow for developed analysis. However, it is accepted that this may not be available (or shareable with us). In those cases, focus will be placed on issues faced and dealt with (or not as applicable) by the organisation.|
|Understanding issues||All case studies will be preceded by the principal investigator conducting background reading of Yin (1992), Mittelstadt et al (2014) and Macnish (2010) to gain an understanding of the purpose of the case study (Yin 1992), the range of issues currently identified in use of algorithms (Mittelstadt et al 2014) and an example of how analysis of an algorithm’s parameters can lead to ethical concerns (Macnish 2010).|
Table 8: Potential Issues
Becker, J., Niehaves, B., 2007. Epistemological perspectives on IS research: a framework for analysing and systematizing epistemological assumptions. Inf. Syst. J. 17, 197–214.
Rowley, J., 2002. Using case studies in research. Manag. Res. News 25, 16–27.
Stake, R.E., 2005. Qualitative Case Studies, in: The Sage Handbook of Qualitative Research. SAGE Publications Ltd, Thousand Oaks, CA, pp. 443–466.
Walsham, G. (1995) ‘Interpretive Case-Studies in IS Research – Nature and Method’, European Journal of Information Systems, 4(2), 74-81.
Yin, R.K., 2015. Qualitative research from start to finish. Guildford publications.
Yin, R.K., 2003. Case Study Research Design and Methods, 3rd ed. SAGE Publications, UK.
Appendix B: Ethics Approval
B.1 Consent Form
|I have read the information presented in the information letter about the case study|
|I have had the opportunity to ask any questions related to this study, and received satisfactory answers to my questions, and any additional details I wanted.|
|I am also aware that excerpts from the interview may be included in publications to come from this research. Quotations will be kept anonymous unless I give specific permission to the contrary (below).|
|I give permission for my name to be associated with excerpts from the interview which may be included in publications to come from this research.|
|I give permission for my organisation to be identified in any final publications produced by SHERPA.|
|I give permission for the interview to be recorded using audio recording equipment. (if necessary).|
|I understand that relevant sections of the data collected during the study may be looked at by individuals from or a project partner from SHERPA. I give permission for these individuals to have access to my responses.|
|I understand that the audio recording may be given to a transcription service company to transcribe. I give permission for these organisations to have access to my audio files for transcription purposes|
With full knowledge of all foregoing, I agree to participate in this study.
I agree to being contacted again by the researchers if my responses give rise to interesting findings or cross references.
□ No □ Yes
If yes, my preferred method of being contacted is:
□ Telephone: ……………………………………………………..
□ Email: ………………………………………………………….
□ Other: …………………………………………………………..
|Participant Name||Consent taken by|
B.2 Consent Form Information Sheet
SHERPA – Shaping the ethical dimensions of smart information systems (SIS) – a European perspective
Task 1.1 – Case Studies
Please take some time to read this information and ask questions if anything is unclear.
Contact details can be found at the end of this document.
What is the purpose of this study?
This study aims to develop a case study for the SHERPA project regarding ethical issues arising in the use of artificial intelligence and big data (Smart Information Systems – SIS). This information will be used to develop an analysis of these ethical issues and, thereafter, a workbook on the responsible development of SIS. The information gained from these interviews will identify potential gaps in understanding and reinforce comprehension of well-known issues.
Who is organising this research?
The research for this study is being undertaken by
A Research Ethics Committee has reviewed and approved this research.
Why have I been chosen?
The project aims to develop 10 case studies from a number of different contexts. These will include business, government, insurance, agriculture and banking. The aim is to develop a broad understanding of ethical issues spanning these contexts, recognizing similarities and differences. You have been chosen as a member of one of these contexts with a good understanding of the use of SIS within your own particular context.
Do I have to take part?
Participation in this study is voluntary and you may ask any questions before agreeing to participate. If you agree to participate, you will be asked to sign a consent form. However, at any time, you are free to withdraw from the study and if you choose to withdraw, we will not ask you to give any reasons.
What will happen to me if I take part?
If you agree to take part in this study we will interview you in person, or by phone/Skype, regarding your experience with SIS.
We may ask you to participate in a follow-up interview, though participation in this is optional.
What are the possible benefits of participating?
The study aims to develop an understanding of ethical issues, the purpose of which is to inform the SHERPA project proposals for the responsible development of SIS. In addition to helping the SHERPA project, advanced ethical analysis will be carried out on the case study to which you contribute, which may raise issues that your organisation would like to know about and take steps to remedy.
What are the possible risks of taking part?
There are no risks in taking part in this study. At any time during the interview you can choose to withdraw. You may also choose to withdraw your data from being used in the project at any time until 1 January 2019.
How will my interview be used?
The case studies will combine quantitative and qualitative elements and will be designed and analysed by SHERPA project partners who have experience in ethical analysis. The recording of the interview may be transcribed by parties outside of the consortium. If this happens, the transcription company will delete the recording and transcription after the transcription is approved. On the consent form we will ask you to confirm that you are happy for the SHERPA consortium to use and quote from your interview. Any such use will be anonymous unless you indicate otherwise on the consent form. Information which will identify your organisation will also be kept out of publications unless otherwise indicated on the consent form.
What will happen to the results of the project?
All the information that we collect about you during the course of the research will be kept strictly confidential. You will not be identified in any reports or publications and your name and other personal information will be anonymised unless you indicate otherwise on the consent form.
What happens to the interviews collected during the study?
The interviews will be transcribed by the interviewers or a designated, approved third-party agency. If we use a third-party transcription service, we will ensure that there is a signed data processing agreement in place. The audio files will be deleted, once the analysis of the interview is complete.
What happens at the end of the project?
You may request a summary of the research findings by contacting Kevin Macnish, University of Twente (firstname.lastname@example.org).
What about use of the data in future research?
If you agree to participate in this project, the research may be used by other researchers and regulatory authorities for future research.
Who is funding the research?
This research is funded by the European Commission] under grant no. 786641.
What should I do if I have any concerns or complaints?
If you have any concerns about the project, please speak to the researcher, who should acknowledge your concerns within ten (10) working days and give you an indication of how your concern will be addressed. If you remain unhappy or wish to make a formal complaint, please contact Kevin Macnish, email@example.com.
Fair Processing Statement
This information which you supply and that which may be collected a part of the project will be entered into a filing system or database and will only be accessed by the researcher and supervisor involved in the project. The information will be retained by the researcher’s institution and will only be used for the purpose of research, statistical and audit and possibly commercial purposes. By supplying this information you are consenting to us storing your information for the purposes above. The information will be processed by use in accordance with the provisions of the Data Protection Act 1998. No identifiable data will be published.
Appendix C Template for Final Report
Length: 7500 – 9000 words
- Introduction – 300 – 500 words
- The topic and describe the aim of the report
- Any case-specific research question(s),
- Methods (to the extent different from the general approach advocated for the case studies) and
- Outline of the remainder of the report.
- Background review: ethical issues in the application domain you are discussing (e.g. SIS in health care) – 1500 – 2500 words
- Description of the domain and its use of SIS (i.e. how SIS is currently being used in healthcare)
- Survey of ethics literature of SIS in the domain you will discuss, as well as own identification of ethical issues [Do this is a systematic way that gives an overview of ethical issues that have been identified and discusses each]
- Introduction to the case study and descriptive analysis – 2000 – 2500 words
(Empirical research, interviews, literature study presented in journalistic style)
Description of the setting:
- description of the organisation(s) and individual(s) interviewed – interview questions 1-6,
- of the SIS systems(s) being used by the organization(s) – interview question 7,
- the aims of the organization(s) in using these systems – interview question 8,
- does the system work as intended or are there issues with its operation – interview question 9,
- identify some general (non-ethical) impacts that the system has (had) on the organization – interview questions 10-12.
- the way in which the system(s) work – interview questions 13-14,
- the policies governing the system(s) use – interview questions 15-16,
- who are stakeholders and what is the impact on them – interview questions 17-24
- Ethical issues – 2000 – 2500 words
- Identify benefits and harms of the system from the point of view of stakeholders.
- Identify ethical issues with the system(s),
- That are the result of the design and normal operation of the system, and
- The particular uses and responses to it or its impact by stakeholders. (These can be ethical issues recognized by stakeholders or not recognized.)
- Discuss the extent to which these are recognized by stakeholders and any remedial measures that have been taken already (laws, policies, initiatives, redesigns, etc.).
- Draw upon the information gathered during the interviews; literature study; and research about the case
- Conclusion 500 – 1000 words
Summary of findings.
- What have we learned from the case study?
- What are possible ways in which the ethical issues can be mitigated?
- What further investigations should be done?