AI and why we should all be worried - AI's dirty secrets

I gave a new talk on AI for the Swindon Science Cafe on 3rd June. Here below are the slides. This talk is an updated version of a short talk I gave in June 2019.

Slide 1

Hi, my name is Alan Winfield. Thank you Rod and Claire for inviting me to speak this evening.

Slide 2

So, what does a robot and AI ethicist do? Well, the short answer is worry.

I worry about the ethical and societal impact of AI on individuals, society and the environment. Here are some keywords on ethics, reflecting that we must work toward AI that respects Human Rights, diversity and dignity, is unbiased and sustainable, transparent, accountable and socially responsible. I also work in Standards with both the British Standards Institute and the International IEEE Standards Association.

Slide 3

Before getting into the ethics of AI I need to give you a quick tutorial on machine learning.

The most powerful and exciting AI today is based on Artificial Neural Networks (ANNs). Here is a simplified diagram of a Deep Learning network for recognizing images. Each small circle is a *very* simplified mathematical model of biological neurons, and the outputs of each layer of artificial neurons feed the inputs of the next layer. In order to be able to recognise images the network must first be trained with images that are already labelled - in this case my dear late dog Lola.

But in order to reliably recognise Lola the network needs to be trained not with one picture of Lola but many. This set of images is called the training data set and without a good data set the network will not work at all or will be biased. (In reality there will need to be not 4 but hundreds of images of Lola).

But even a simple ANN can get things wrong. A famous example was an ANN like this trained on pictures of wolves. After training they input of a bear, but the network identified it as a wolf. Why? Because all of the wolf pictures had snowy backgrounds, so the network learned to recognize snow, not a wolf. The bear also had a snowy background.

This is one example of what we now call a ‘hallucination’ in big AIs.

Slide 4

We’ve been worrying about the existential threat of AI for a long time: here is a piece I wrote for the Guardian in 2014, when ‘the singularity was the main thing we worried about.

The singularity is the idea that as soon as AI is smarter than humans then the AIs will rapidly improve themselves, with unforeseeable consequences for human civilization.

But the singularity is a thing for the techno-utopians: wealthy middle-aged men who regard the singularity as their best chance of immortality. They are Singularitarians, some of whom appear prepared to go to extremes to stay alive for long enough to benefit from a benevolent super-AI - a manmade god that grants transcendence.

And it's a Thing for the doomsayers, the techno-dystopians. Apocalypsarians who are equally convinced that a superintelligent AI will have no interest in curing cancer or old age, or ending poverty, but will instead - malevolently or maybe just accidentally - bring about the end of human civilisation as we know it. History and Hollywood are on their side. From the Golem to Frankenstein's monster, Skynet and the Matrix, we are fascinated by the old story: man plays god and then things go horribly wrong.

Slide 5

Today we have influential scientists who worry about Artificial General Intelligence (AGI). Notable among these is physicist and cosmologist Max Tegmark.

At the AI safety summit in February 2025 Tegmark argued that we need a middle pathway between No AI and Uncontrollable AGI, which he calls guaranteed safe tool AI. He suggested a policy solution in which the US and China both launch national safety standards preventing their own AI companies from building AGI. Leading to what Tegmark rather optimistically calls ‘an age of unprecendented global prosperity powered by safe tool AI.

Is there an existential threat from AI itself? No. I fear human stupidity much more than artificial intelligence.

So should we be worried? Yes. But the things I worry about are rather more down to earth.

In the rest of this talk I will consider the energy costs of AI, then the human costs.

Slide 6

In 2016 Go champion Lee Sedol was famously defeated by DeepMind's AlphaGo. It was a remarkable achievement for AI. But consider the energy cost. In a single two-hour match Sedol burned around 170 kcals: roughly the amount of energy you would get from an egg sandwich. Or about 1 Watt – the power of an LED night light.

In the same two hours the AlphaGo machine reportedly consumed about 50,000 Watts. The same as a 50 kW generator for industrial lighting. And that's not taking account of the energy used to train AlphaGo.

Slide 7

A paper published in 2019 paper revealed, for the first time, estimates of the carbon cost of training large AI models for natural language processing such as machine translation. The carbon cost of simple models is quite modest, but with tuning and experimentation the carbon cost leaps to 7 times the carbon footprint of an average human in one year (or 2 times if you're an American).

And the energy cost of optimizing the biggest model is a staggering 5 times the carbon cost of a car over its whole lifetime, including manufacturing it in the first place. The dollar cost of that amount of energy is estimated at between one and 3 million US$. (Something that only companies with very deep pockets can afford.)

These energy costs seem completely at odds with the urgent need to meet sustainable development goals. At the very least AI companies need to be honest about the huge energy costs of machine learning.

Slide 8

At the same Paris meeting earlier this year AI ethicist Kate Crawford drew attention to both energy and water costs of AI. Crawford predicts that the energy cost of training generative AIs will soon overtake the total energy consumption of industrialized nations such as Japan.

She also drew attention to the colossal amounts of clean water that AI server farms need to keep them cool. Water that is already a scarce resource.

One study estimated that ChatGPT-3 requires 700,000 litres of clean water for training, And that each user interaction costs around half a litre of water.

Source: https://interestingengineering.com/innovation/training-chatgpt-consumes-water

Slide 9

The very same Kate Crawford, together with a colleague, produced this extraordinary map of the entire process behind the Amazon Echo.

The remarkable Anatomy of an AI System, shows The Amazon Echo as an anatomical map of human labour, data and planetary resources. 

The map is far too detailed to see on this slide but let me just zoom in on the top of this very large iceberg where we find the amazon echo and its human user, shown here in a yellow dashed box.

Slide 10

At the very top of this pyramid of materials and energy (on the left) and waste (on the right) is you – the user of the Amazon Echo – and your unpaid human labour providing habits and preferences that will be used as training data. 

I strongly recommend you check this out. It is truly eye opening.

Slide 11

Now I want to turn to the human cost of AI.

It is often said that one of the biggest fears around AI is the loss of jobs. In fact the opposite is happening. Many new jobs are being created, but the tragedy is that they are not great jobs, to say the least. Let me introduce you to 3 of these new kinds of jobs.

Conversational AI or chat bots also need human help. Amazon for instance employs thousands of both full-time employees and contract workers to listen to and annotate speech. The tagged speech is then fed back to Alexa to improve its comprehension. In 2019 the Guardian reported that Google employs around 100,000 temps, vendors and contractors: literally an army of linguists to create the handcrafted data sets required for Google translate to learn dozens of languages. Not surprisingly there is a considerable disparity between the wages and working conditions of these workers and Google's full-time employees.

AI tagging jobs are dull, repetitive and in the case of the linguists highly skilled.

Slide 12

Consider AI tagging of images. This is the manual labelling of objects in images to, for instance, generate training data sets for driverless car AIs. Better (and safer) AI needs huge training data sets and a whole new outsourced industry has sprung up all over the world to meet this need. Here is an AI tagging factory in China.

Slide 13

But by far the worst kind of new white-collar job in the AI industry is content moderation.
 

These tens of thousands of people, employed by third-party contractors, are required to watch and vet offensive content: hate speech, violent pornography, cruelty and sometimes murder of both animals and humans for Facebook, YouTube and other media platforms. These jobs are not just dull and repetitive they are positively dangerous. Harrowing reports tell of PTSD-like trauma symptoms, panic attacks and burn out after one year, alongside micromanagement, poor working conditions and ineffective counselling. And very poor pay - typically $28,800 a year. Compare this with average annual salaries at Facebook of $250,000+.

Slide 14

The extent to which AI has a human supply chain was a big revelation, and I am an AI insider! The genius designers of this amazing tech rely on both huge amounts of energy and a hidden army of what Mary Gray and Siddhartha Suri call Ghost Workers.

I would ask you to consider the question: how can we, as ethical consumers, justify continuing to make use of unsustainable and unethical AI technologies?

Slide 15  

AI ethics are important because AIs are already causing harm. Actually a *very* wide range of harms.

Fortunately, there is an excellent crowdsourced database which collects reports of accidents and near misses involving robots (including autonomous vehicles) and AIs.

The AI incidents database covers both accidents and near misses. I strongly recommend you check it out.

It is important to note that because the AI incidents database is crowdsourced from accidents and near misses that made it into the press and media, what we see is almost certainly only the tip of the iceberg of the harms being done by AI.

The database contains some truly shocking cases. One is a 14-year-old boy, who died by suicide after reportedly becoming dependent on Character.ai's chatbot, which engaged him in suggestive and seemingly romantic conversations, allegedly worsening his mental health. Source: Can A.I. Be Blamed for a Teen’s Suicide? New York Times, Oct 2024.

The database also highlights the criminal use of AI. Examples include criminals phoning parents claiming they have kidnapped a child and demanding a ransom, with deepfake audio of the teenage girl audible in the background. There are many examples of sextortion, using deepfake AI generated video of famous individuals engaged in sex acts.

The database really highlights the sad truth that AI is a gift to criminals.

Slide 16  

Another more recent database tracks the misuse of AI by lawyers when preparing court cases. The database only shows those instances where the judge (or another officer in the court) spotted the hallucinated decisions, citations or quotations. The cases were thrown out, and in some cases the lawyers found to be using Ai were fined or reported to their bar associations.

While this is not criminal misuse of AI, it does demonstrate a lack of understanding of AI, or at best, naivety. Perhaps the real culprits are hard pressed paralegals. This database underlines the need for professionals to be property trained in AI and its weaknesses.

Since I grabbed this screen shot the number if cases reported has grown.

Slide 17

Lawyer Graziano Mioli elegantly argues that we have a categorical imperative to be imperative when interacting with AIs. Noting that this is not an invitation to be rude.

For Graziano's slides see https://www.youtube.com/watch?v=tjBnGN4u1GA&ab_channel=GrazianoMioli  

I can see why a majority of people interact kindly with AIs. I think it reflects well on those who do say ‘please’, for the Kantian reason that we should not get into the habit of acting unkindly.

See https://www.theaihunter.com/news/ai-etiquette-why-some-people-say-please-to-chatbots/

Slide 18

Having mostly elaborated on the dangers of AI, I want to finish on a positive.

We already enjoy the benefit of useful and reliable AI technology, like smart maps and machine translation. DeepMind's diagnostic AI can detect over 50 eye diseases from retinal scans as accurately as a doctor, and DeepScribe provides automated note taking during a consultation, linking with patient electronic health records

Thank you!

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

How to make an ethical robot, and why we probably shouldn't

Slide 2

Robot ethics and machine ethics are two sides of the same coin.  Robot ethics are ethics for humans. Machine ethics are ethics for robots.

Slide 3

But robot ethics and machine ethics are at very different levels of urgency and maturity. Robot Ethics is a much more pressing concern, given the rapid pace of developing applications as diverse as driverless cars, assisted living robots and smart robot toys. Also robot ethics has a large and active community which is already making progress toward standards and policy. 

In contrast machine ethics remains the subject of basic research by a very small community of scholars. There are in fact no real-world ethical robots at the time of writing and it seems unlikely that there will be for some years. 

Slide 4

Wendell Wallach and Colin Allen, in their wonderful 2009 book posed the open question: “Do we have a moral imperative to try and build ethical robots?” and suggest that the answer is (a qualified) yes.

Slide 5

James Moor’s important and influential paper 2006 set out a set of four categories of moral agency, from none to full.

 

Examples of ethical impact agents are kitchen knives and hammers. Both can be evaluated for ethical use (i.e. surgery) and unethical use (i.e. stabbing someone).

 

An example implicit ethical agent is the kind of blunt plastic knife that comes with airline food.

 

An explicit ethical agent can reason about ethics. Very few explicit agents have been demonstrated, not least because they are very hard to build.

 

The only full ethical agents we know of are adult humans of sound mind.

Slide 6

Allen, Smit and Wallach defined 3 approaches to explicit ethical machines in their 2005 paper. A training approach, which they call top down; a constraint (rules) based approach which they call bottom up and a hybrid approach that combines the two.

 

The work I will describe in this talk is all bottom-up. I know of only one instance of a top-down approach. The wonderful work of Susan and Michael Anderson: see their paper shown here.

Slide 7

Is it possible to build a moral machine: a robot capable of choosing or moderating its actions on the basis of ethical rules? Until 2014 I thought the idea impossible. But I changed my mind. In fact, also developed and experimentally tested an ethical robot. What brought about this U-turn?

Slide 8

First was thinking about very simple ethical behaviours. Imagine you see someone not looking where they’re going - about to walk into a hole in the pavement. Most liklely you will intervene. But why? It’s not just because you’re a good person – you also have the cognitive machinery to predict the consequences of someone’s actions.

Slide 9

Now imagine it’s not you, but a robot with four possible next actions. From the robot's perspective, it has two safe options: standstill (A), or turn to its left (B). But if the robot can model the consequences of both its own actions and the human's - another possibility opens up: the robot could choose to collide with the human to prevent him from falling into the hole (action D).

 
Slide 10

Let’s write this down as a logical rule. Remarkably the rule appears to match Asimov’s first law of robotics: A robot may not injure a human being or, through inaction, allow a human being to come to harm. The through inaction clause is important because it allow the robot to be morally proactive.

Slide 11

So emerged the idea is that we might be able to build a robot with Asimovian ethics. We need to equip the robot with the ability to predict the consequences of both its own, and other(s) actions, plus the hard-wired ethical rule in the previous slide.

 

Slide 12

 

Then came the realisation that the technology we need not only exists but is mature and commonplace in robotics research – it is the robot simulator. Robot simulators provide developers with a virtual environment for prototyping robot code before then running that code on the real robot. 

 

Slide 13

 

But a robot simulator is not enough on its own. It also needs to be running inside the ethical robot. Thus, we set about designing a simulation-based internal model, which we call a consequence engine, shown here. On the right had the vertical green line describes the Sense Plan Act control system of most robots.

 

The consequence engine runs in parallel. The internal simulator has the three components shown here: a world model (with physics), a robot model (of itself and others), and an exact copy of the robot’s real controller.

 

For the current disposition of the robot – and others – the CE loops through all next possible actions, in order to estimate what might happen for each action. Then all of those predictions are evaluated, and the safety or ethics logic modifies the real robot’s action selection. Our robots are typically able to loop through 30 next possible actions every half a second.

 

Slide 14

 

The action evaluator codes the estimated outcome for each robot’s action (and the proxy human robot), on a scale of 0 to 10. Where 0 is completely safe and 10 is very dangerous. The value 4 codes for a collision; in reality simple obstacle avoidance, so no collision at all.

 

This simple table shows this mechanism assuming just 4 next possible actions. These numerical values allow the ethical robot to choose ‘ahead right’ as the least unsafe outcome for the proxy human. The lowest combined outcome values.

 

Slide 15

We built an ethical robot based on these ideas. We don’t have a real hole in the ground – just a danger zone, and we use robots as proxy humans. We ran two sets of experiments first with e-puck then with NAO robots. Let me show you these results – testing a simple Asimovian robot.

 

Slide 16

  

This short movie clip shows the robots of trial 2*. The ethical A-robot starts at the lower middle and the proxy-human H-robot starts from the left. The first run is in real time, then successive runs are speeded up.

Notice especially the moment when the A-robot ‘notices’ the H-robot is heading for danger and diverts from its path to intercept it. This is when Asimov’s 1st law is triggered.

We see the A-robot successfully prevents the H-robot from falling into the hole in every run.

*Trial 1 is simply the ethical robot avoiding the hole. 

 

Slide 17

After running trial 2 with the e-puck robots we decided to test our Asimovian robot with an ethical dilemma by introducing a second proxy-human H2 – also heading toward danger. As far as we know this is the world’s first experimental demonstration of an ethical robot facing a balanced dilemma.

Trial 3 is very interesting because on many runs the A-robot is seen to ‘dither’. We see this on the first run when the A-robot could have easily reached H2 to intercept it, but failed to do so, resulting in both H and H2 falling into the hole.

Because the consequence engine is running continuously, the A-robot can change its decision every half a second. This explains the dithering we observe here.

Paris conference on Safe and Ethical AI

Earlier this month I was privileged to part of the inaugural conference of the International Association for Safe and Ethical AI. The two day conference was held in Paris, on February 6^th and 7^th, and hosted by the OECD.  The timing and location of the conference was arranged to directly precede the governmental AI summit on February 10^th and 11^th. I was one of around 650 invited representatives from academia, civil society, industry, media, and government. It was a remarkable meeting, with terrific keynote talks, including three from Nobel prize winners, Geoffrey Hinton, Maria Ressa and Joseph Stiglitz.  

As someone who has been worrying about robot and AI ethics for longer than most who attended I was *very* pleased that there was a strong consensus around the need for regulation, supported by standards, alongside urgent concerns over the huge energy and water costs of AI that are completely at odds with sustainable development goals.

The conference concluded by publishing a Call to Action for lawmakers, academics, and the public ahead of the AI Summit, with ten critical action items. Overall, the action items are very good. I’m especially pleased to see ‘mandatory reporting of incidents’ in action 5. This is something I lobbied for. My one disappointment however is that the call for action statement has no explicit mention of the need to mitigate the energy costs of AI.

Here below are a few photos from the conference.

A slide from Joseph Stiglitz’ wonderful keynote: AI and Economic Risk: Assessment and Mitigation.

 

A slide from Kate Crawford’s excellent keynote: Hyperscaled: The Global Challenge of Sustainability in AI.

  

Me with Oxford colleague Pericle Salvini presenting our RoboTIPS work on accident investigation.

New paper: A Simulated real-world upper-body Exoskeleton Accident and Investigation

Back in February I posted a very brief account of our third RoboTIPS simulated accident and investigation, centred on an upper-body exoskeletion in an industrial setting. Since then we've published a paper with a full account. My colleague Pericle Salvini presented the paper at the 9th International Conference on Robot Ethics and Standards (ICRES 2024), last week.

Here is the paper abstract:

This paper describes the enactment of a simulated (mock) accident involving an upper-body exoskeleton and its investigation. The accident scenario is enacted by role-playing volunteers, one of whom is wearing the exoskeleton. Following the mock accident, investigators – also volunteers – interview both the subject of the accident and relevant witnesses. The investigators then consider the witness testimony alongside robot data logged by the ethical black box, in order to address the three key questions: what happened?, why did it happen?, and how can we make changes to prevent the accident happening again? This simulated accident scenario is one of a series we have run as part of the RoboTIPS project, with the overall aim of developing and testing both processes and technologies to support social robot accident investigation.

 The paper sets out, for the first time, the experimental method we have developed:

1. The accident scenario is enacted by human volunteers, role playing the subject of the accident, together with both direct  and indirect witnesses. The subject is the person to whom the accident happens. Direct witnesses are those who either witness or discover the accident, and indirect witnesses are those who might be supervisors or managers of the subject and/or the facility, or representatives of the robot's manufacturer. 
2. Prior to the enactment the project team brief the volunteers. Each briefing is specific to the role and, with the exception of the subject, volunteers are briefed only on their role, and not the whole scenario. This is so that they witness the accident (or it's aftermath) for the first time during the enactment. Only the subject is fully briefed on the scenario, including the safety aspects explained below, so that they are confident that they will not come to harm or be fearful during the enactment.
3. The enactment is stage managed by project team members. Although the simulation resembles a piece of theatre, volunteers are not asked to learn any lines. Apart from any specific action essential to the scenario (which will be prompted by the stage manager) the volunteers are invited to ad lib in a way that is appropriate to the roles they are playing. Volunteers are asked to wait in a side room until they are called a few moments before they are needed.
4. Safety of the volunteers, and especially the subject, is of paramount importance. Thus, if the scenario simulates physical harm to the subject, then – when the accident happens – the enactment is briefly suspended by the stage manager and the subject is helped into the position they might be expected to be in, following the accident. The project team conduct a safety risk assessment and if necessary modify the scenario and/or its stage management to mitigate any risks and the simulation is only undertaken after university research ethics approval.
5. The accident investigators are also volunteers and, ideally, the lead accident investigator has expertise and/or experience in accident investigation. Robotics expertise is not essential, as the aims and process of investigation are common to all accident or incident (near miss) investigations. The accident investigators are not briefed on the scenario, only the type of robot involved. Necessarily the accident investigators are not present during the enactment of the simulated accident. To reduce the time burden on all volunteers we stage the accident and its investigation on a single day, with the accident investigators arriving after the enactment. 

We were very lucky indeed that University of Nottingham Prof Carl McRae genrously acted as lead investigator for all three accident simulations in RoboTIPS. Carl is an authority on accident investigation in both aviation and heathcare. This meant that the process that Carl, together with a second volunteer investigator, followed asked the same questions that a real investigation would ask, namely: what happened, why did it happen, and how can we improve the system so that it doesn't happen again.

The full paper is on ArXiv here: https://arxiv.org/pdf/2411.14008v1

A simulated upper body exoskeleton accident and investigation

On Wednesday 21 February we ran the third of our RoboTIPS simulated accident scenarios in the Bristol Robotics Lab. This scenario focussed on an upper-body exoskeleton in an industrial environment.



Above left we see Dan working to move boxes, with the physical support of the wonderful Tribonix exoskeleton. On the right Dan has fallen to the floor, attended by his manager Monica and paramedic Ben. The simulation was carefully scripted and stage managed to ensure that none of the volunteers were hurt or, indeed, ever at risk.

 

Following the simulation the accident was investigated by lead investigator Carl and co-investigator Jack. Carl Macrae is a leading authority on accident investigation. Here we see Jack and Carl interviewing expert witness Appolinaire, observed by RoboTIPS project lead Marina Jirotka.

In addition to witness testimony our investigators were also able to examine Ethical Black Box data logs collected from the exoskeleton during the simulated accident.

The simulated accident scenario was a huge success. The various roles (not all of which are shown in the photos here) were acted brilliantly by our volunteers Dan Read, Ashwin Chandapur, Monica Monica, Surin Machaiah, Ben Allen and Dr Appolinaire Etoundi. And despite a complicated scenario which included human-human as well as human-robot interaction, our accident investigators Prof Carl Macrae and Jack Hughes were able to deduce, with reasonable accuracy, what happened and why. We are especially grateful to Romain Derval and Filip Hanus, co-founders of Tribonix, for both kindly agreeing to the use of their exoskeleton and generously working with RoboTIPS during the planning and enactment of this simulation.

The simulation was subject to Research Ethics Committee approval CATE-2324-218.

---

See also: 

Our first mock social robot accident and investigation

Robot Accident Investigation 

A Draft Open Standard for an Ethical Black Box

About 5 years ago we proposed that all robots should be fitted with the robot equivalent of an aircraft Flight Data Recorder to continuously record sensor and relevant internal status data. We call this an ethical black box (EBB). We argued that an ethical black box will play a key role in the processes of discovering why and how a robot caused an accident, and thus an essential part of establishing accountability and responsibility.

Since then, within the RoboTIPS project, we have developed and tested several model EBBs, including one for an e-puck robot that I wrote about in this blog, and another for the MIRO robot. With some experience under our belts, we have now drafted an Open Standard for the EBB for social robots - initially as a paper submitted to the International Conference on Robots Ethics and Standards. Let me now explain first why we need a standard, and second why it should be an open standard.

Why do we need a standard specification for an EBB? As we outline in our new paper, there are four reasons:

1. A standard approach to EBB implementation in social robots will greatly benefit accident and incident (near miss) investigations. 
2. An EBB will provide social robot designers and operators with data on robot use that can support both debugging and functional improvements to the robot. 
3. An EBB can be used to support robot ‘explainability’ functions to allow, for instance, the robot to answer ‘Why did you just do that?’ questions from its user. And,
4. a standard allows EBB implementations to be readily shared and adapted for different robots and, we hope, encourage manufacturers to develop and market general purpose robot EBBs.

And why should it be an Open Standard? Bruce Perens, author of The Open Source Definition, outlines a number of criteria an open standard must satisfy, including:

• “Availability: Open standards are available for all to read and implement.
  
• Maximize End-User Choice: Open Standards create a fair, competitive market for implementations of the standard.
• No Royalty: Open standards are free for all to implement, with no royalty or fee.
• No Discrimination: Open standards and the organizations that administer them do not favor one implementor over another for any reason other than the technical standards compliance of a vendor’s implementation.
• Extension or Subset: Implementations of open standards may be extended, or offered in subset form.”

These are *good* reasons.

The most famous and undoubtedly the most impactful Open Standards are those that specified Internet protocols, such as FTP and email. They were, and still are, called Requests for Comments (RFCs) to reflect the fact that they were - especially in the early years - drafts for revision. As a mark of respect we also regard our draft 0.1 Open Standard for an EBB for Social Robots, as an RFC. You can find draft 0.1 in Annex A of the paper on arXiv here.

Not only is this a first draft, it is also incomplete, covering only the specification of the data and its format, that should be saved in an EBB for social robots. Given that the EBB data specification is at the heart of the EBB standard, we feel that this is sufficient to be opened up for comments and feedback. We will continue to extend the specification, with subsequent versions also published on arXiv.

Please feel free to either submit comments to this blog post (best because everyone can see the comments), or by contacting me directly via email. All constructive comments that result in revisions to the standard will be acknowledged in the standard.

Our first mock social robot accident and investigation

Robot accidents are inevitable. These days the likelihood of serious accidents involving industrial robots is pretty low (but not zero), because such robots are generally inside safety cages. But a newer generation of social robots - robots designed to interact directly with people, including vulnerable elderly people or children - means that accidents are now much more likely. And if we also take into account ethical harms alongside physical harms, then the potential for accidents increases still further. Psychological harms include addiction, over trusting, or deception, and societal harms include privacy violations. For more on these ethical harms see my blog post outlining an ethical risk assessment of a smart robot teddy bear.

It has puzzled me for some years that there has been almost no research on robot accident investigation. In the RoboTIPS project we are addressing this deficit by developing both the technology - which we call an Ethical Black Box (EBB) - and the processes of robot accident investigation. One of the most exciting aspects of RoboTIPS is that we're running a series of mock, i.e. staged, social robot accidents in order to road test the EBB and investigation processes in as close to a real situation as is feasible in a research project. RoboTIPS started in March 2019, but then just as we were ready to trial our first mock accident the Covid pandemic hit, and closed down the lab.

So it was great that last week we finally managed to run the a pilot of our first (of three) mock accident scenarios. The scenario, based around an assisted living robot helping an elderly person to live independently, was sketched out in late 2019, and then - during the lockdown - rehearsed in a number of online events, including a podcast radio play for Oxford Sparks and CSI Robot during the UKRAS Festival of Robotics 2021.

Here is the scenario:

Imagine that your elderly mother, or grandmother, has an assisted living robot to help her live independently at home. The robot is capable of fetching her drinks, reminding her to take her medicine and keeping in touch with family. Then one afternoon you get a call from a neighbour who has called round and sees your grandmother collapsed on the floor. When the paramedics arrive they find the robot wandering around apparently aimlessly. One of its functions is to call for help if your grandmother stops moving, but it seems that the robot failed to do this

To enact this scenario we needed a number of volunteers: one to act as Rose - the subject of the accident, a second as the neighbour who discovers the accident and raises the alarm, a third as the paramedic who attends to Rose, a fourth who acts in the role of the cleaner and a fifth in the role of manager of the group of homes in which Rose lives. We also needed volunteers to act as members of the accident investigation team who are called in to try and discover what happened, why it happened and, if possible, what changes need to be made to how to ensure the accident doesn't happen again.

This is the mock accident taking place in the kitchen of our assisted living studio. Left shows the neighbour, acted by Paul, discovering Ross, acted by Alex, injured on the floor. (Note the chair on its side.) Right is the paramedic, role-played by Luc, attending to Ross. Meanwhile the Pepper robot is moving around somewhat aimlessly.

Our brilliant Research Fellow Dr Anouk van Maris, who organised the whole setup, persuaded five colleagues from the Bristol Robotics Lab. All were male, so Rose became Ross. Only one volunteer: Alex, who played the part of Ross, was fully briefed. The other four role played brilliantly and, although they were briefed on their roles, they were not told what was going to happened to Ross, or the part the Pepper robot played (or maybe didn't play) in the accident. Two colleagues from Oxford, Lars and Keri, kindly volunteered to act as the accident investigators. Lars and Keri also had no prior knowledge of the circumstances of the accident, and had to rely on (i) inspecting the robot and the scene of the accident, (ii) the data from the robot's EBB, and (iii) testimonies from Ross, the neighbour, the paramedic, the cleaner and the facility manager.

Here we see Lars interviewing Medhi, who acted as the house manager, while Ben, acting as the cleaner, waits to be interviewed. Inside the studio Keri is interviewing the neighbour and parademic.

So, what were the findings of our accident investigators? They did very well indeed. Close examination of the EBB data, alongside consideration of the (not always reliable) witness testimony enabled Lars and Keri to correctly deduce the role that the robot played in the accident. They were also able to make several recommendations on operational changes.  But I will not reveal their findings in detail here as we intend to run the same mock accident again soon with a different set of volunteers and - in case any of them should read this blog - I don't want to give the game away!

Acknowledgements

Very special thanks to Dr Anouk van Maris. Also Dr Pericle Salvini, who worked with Anouk in finalising the detail of the scenario and during the pilot itself. Also, huge thanks to BRL volunteers Dr Alex Smith, Dr Paul Bremner, Dr Luc Wijnen, Mehdi Sobhani and Dr Ben Ward-Cherrier. And last but not least a very big thank you to Dr Lars Kunze, Oxford Robotics Institute and Keri Grieman, Dept of Computer Science, Oxford.

From the left: Pericle, Ben, Lars, Alex, Keri, Medhi, Paul, Anouk, Luc, Lola and me. Pepper is looking nervously at Lola.

Ethics is the new Quality

This morning I took part in the first panel at the BSI conference The Digital World: Artificial Intelligence.  The subject of the panel was AI Governance and Ethics. My co-panelist was Emma Carmel, and we were expertly chaired by Katherine Holden.

Emma and I each gave short opening presentations prior to the Q&A. The title of my talk was Why is Ethical Governance in AI so hard? Something I've thought about alot in recent months.

Here are the slides exploring that question.

 

And here is what I said.

Early in 2018 I wrote a short blog post with the title Ethical Governance: what is it and who's doing it? Good ethical governance is important because in order for people to have confidence in their AI they need to know that it has been developed responsibly. I concluded my piece by asking for examples of good ethical governance. I had several replies, but none were nominating AI companies.

So. why is it that 3 years on we see some of the largest AI companies on the planet shooting themselves in the foot, ethically speaking? I’m not at all sure I can offer an answer but, in the next few minutes, I would like to explore the question: why is ethical governance in AI so hard? 

But from a new perspective. 

Slide 2

In the early 1970s I spent a few months labouring in a machine shop. The shop was chaotic and disorganised. It stank of machine oil and cigarette smoke, and the air was heavy with the coolant spray used to keep the lathe bits cool. It was dirty and dangerous, with piles of metal swarf cluttering the walkways. There seemed to be a minor injury every day.

Skip forward 40 years and machine shops look very different. 

Slide 3

So what happened? Those of you old enough will recall that while British design was world class – think of the British Leyland Mini, or the Jaguar XJ6 – our manufacturing fell far short. "By the mid 1970s British cars were shunned in Europe because of bad workmanship, unreliability, poor delivery dates and difficulties with spares. Japanese car manufacturers had been selling cars here since the mid 60s but it was in the 1970s that they began to make real headway. Japanese cars lacked the style and heritage of the average British car. What they did have was superb build quality and reliability" [1].

What happened was Total Quality Management. The order and cleanliness of modern machine shops like this one is a strong reflection of TQM practices. 

Slide 4

In the late 1970s manufacturing companies in the UK learned - many the hard way - that ‘quality’ is not something that can be introduced by appointing a quality inspector. Quality is not something that can be hired in.

This word cloud reflects the influence from Japan. The words Japan, Japanese and Kaizen – which roughly translates as continuous improvement – appear here. In TQM everyone shares the responsibility for quality. People at all levels of an organization participate in kaizen, from the CEO to assembly line workers and janitorial staff. Importantly suggestions from anyone, no matter who, are valued and taken equally seriously.

Slide 5

In 2018 my colleague Marina Jirotka and I published a paper on ethical governance in robotics and AI. In that paper we proposed 5 pillars of good ethical governance. The top four are:

• have an ethical code of conduct, 
• train everyone on ethics and responsible innovation,
• practice responsible innovation, and
• publish transparency reports.

The 5th pillar underpins these four and is perhaps the hardest: really believe in ethics.

Now a couple of months ago I looked again at these 5 pillars and realised that they parallel good practice in Total Quality Management: something I became very familiar with when I founded and ran a company in the mid 1980s [2].

Slide 6 

So, if we replace ethics with quality management, we see a set of key processes which exactly parallel our 5 pillars of good ethical governance, including the underpinning pillar: believe in total quality management.

I believe that good ethical governance needs the kind of corporate paradigm shift that was forced on UK manufacturing industry in the 1970s.

Slide 7

In a nutshell I think ethics is the new quality

Yes, setting up an ethics board or appointing an AI ethics officer can help, but on their own these are not enough. Like Quality, everyone needs to understand and contribute to ethics. Those contributions should be encouraged, valued and acted upon. Nobody should be fired for calling out unethical practices.

Until corporate AI understands this we will, I think, struggle to find companies that practice good ethical governance [3]. 

Quality cannot be ‘inspected in’, and nor can ethics.

Thank you.

---

Notes.

[1]    I'm quoting here from the excellent history of British Leyland by Ian Nicholls. 

[2]    My company did a huge amount of work for Motorola and - as a subcontractor - we became certified software suppliers within their six sigma quality management programme.

[3]    It was competitive pressure that forced manufacturing companies in the 1970s to up their game by embracing TQM. Depressingly the biggest AI companies face no such competitive pressures, which is why regulation is both necessary and inevitable.

The Grim Reality of Jobs in Robotics and AI

The reality is that AI is in fact generating a large number of jobs already. That is the good news. The bad news is that they are mostly - to put it bluntly - crap jobs. 

There are several categories of such jobs. 

At the benign end of the spectrum is the work of annotating images, i.e. looking at images and identifying features then labelling them. This is AI tagging. This work is simple and incredibly dull but important because it generates training data sets for machine learning systems. Those systems could be AIs for autonomous vehicles and the images are identifying bicycles, traffic lights etc. The jobs are low-skill low-pay and a huge international industry has grown up to allow the high tech companies to outsource this work to what have been called white collar sweatshops in China or developing countries. 

A more skilled version of this kind of job are translators who are required to ‘assist’ natural language translation systems who get stuck on a particular phrase or word.

And there is another category of such jobs that are positively dangerous: content moderators. These are again outsourced by companies like Facebook, to contractors who employ people to filter abusive, violent or illegal content. This can mean watching video clips and making a decision on whether the clip is acceptable or not (and apparently the rules are complex), over and over again, all day. Not surprisingly content moderators suffer terrible psychological trauma, and often leave the job burned out after a year or two. Publicly Facebook tells us this is important work, yet content moderators are paid a fraction of what staffers working on the company campus earn. So not that important.

But jobs created by AI and automation can also be physically dangerous. The problem with real robots, in warehouses for instance, is that like AIs they are not yet good enough to do everything in the (for the sake of argument) Amazon warehouse. So humans have to do the parts of the workflow that robots cannot yet do and - as we know from press reports - these humans are required to work super fast and behave, in fact, as if they are robots. And perhaps the most dehumanizing part of the job for such workers is that, like the content moderators (and for that matter Uber drivers or Deliveroo riders), their workflows are managed by algorithms, not humans.

We roboticists used to justifiably claim that robots would do jobs that are too dull, dirty and dangerous for humans. It is now clear that working as human assistants to robots and AIs in the 21st century is dull, and both physically and/or psychologically dangerous. One of the foundational promises of robotics has been broken. This makes me sad, and very angry.

The text above is a lightly edited version of my response to the Parliamentary Office of Science and Technology (POST) request for comments on a draft horizon scanning article. The final piece How technology is accelerating changes in the way we work was published a few weeks ago.