AI and why we should all be worried - AI's dirty secrets

I gave a talk on AI for the Swindon Science Cafe. Here below are the slides, followed by my notes.

Slide 1

Hi, my name is Alan Winfield. Thank you Rod and Claire for inviting me to speak this evening.

Slide 2

So, what does a robot and AI ethicist do? Well, the short answer is worry.

I worry about the ethical and societal impact of AI on individuals, society and the environment. Here are some keywords on ethics, reflecting that we must work toward AI that respects Human Rights, diversity and dignity, is unbiased and sustainable, transparent, accountable and socially responsible. I also work in Standards with both the British Standards Institute and the International IEEE Standards Association.

Slide 3

Before getting into the ethics of AI I need to give you a quick tutorial on machine learning.

The most powerful and exciting AI today is based on Artificial Neural Networks (ANNs). Here is a simplified diagram of a Deep Learning network for recognizing images. Each small circle is a *very* simplified mathematical model of biological neurons, and the outputs of each layer of artificial neurons feed the inputs of the next layer. In order to be able to recognise images the network must first be trained with images that are already labelled - in this case my dear late dog Lola.

But in order to reliably recognise Lola the network needs to be trained not with one picture of Lola but many. This set of images is called the training data set and without a good data set the network will not work at all or will be biased. (In reality there will need to be not 4 but hundreds of images of Lola).

But even a simple ANN can get things wrong. A famous example was an ANN like this trained on pictures of wolves. After training they input of a bear, but the network identified it as a wolf. Why? Because all of the wolf pictures had snowy backgrounds, so the network learned to recognize snow, not a wolf. The bear also had a snowy background.

This is one example of what we now call a ‘hallucination’ in big AIs.

Slide 4

We’ve been worrying about the existential threat of AI for a long time: here is a piece I wrote for the Guardian in 2014, when ‘the singularity was the main thing we worried about.

The singularity is the idea that as soon as AI is smarter than humans then the AIs will rapidly improve themselves, with unforeseeable consequences for human civilization.

But the singularity is a thing for the techno-utopians: wealthy middle-aged men who regard the singularity as their best chance of immortality. They are Singularitarians, some of whom appear prepared to go to extremes to stay alive for long enough to benefit from a benevolent super-AI - a manmade god that grants transcendence.

And it's a Thing for the doomsayers, the techno-dystopians. Apocalypsarians who are equally convinced that a superintelligent AI will have no interest in curing cancer or old age, or ending poverty, but will instead - malevolently or maybe just accidentally - bring about the end of human civilisation as we know it. History and Hollywood are on their side. From the Golem to Frankenstein's monster, Skynet and the Matrix, we are fascinated by the old story: man plays god and then things go horribly wrong.

Slide 5

Today we have influential scientists who worry about Artificial General Intelligence (AGI). Notable among these is physicist and cosmologist Max Tegmark.

At the AI safety summit in February 2025 Tegmark argued that we need a middle pathway between No AI and Uncontrollable AGI, which he calls guaranteed safe tool AI. He suggested a policy solution in which the US and China both launch national safety standards preventing their own AI companies from building AGI. Leading to what Tegmark rather optimistically calls ‘an age of unprecendented global prosperity powered by safe tool AI.

Is there an existential threat from AI itself? No. I fear human stupidity much more than artificial intelligence.

So should we be worried? Yes. But the things I worry about are rather more down to earth.

In the rest of this talk I will consider the energy costs of AI, then the human costs.

Slide 6

In 2016 Go champion Lee Sedol was famously defeated by DeepMind's AlphaGo. It was a remarkable achievement for AI. But consider the energy cost. In a single two-hour match Sedol burned around 170 kcals: roughly the amount of energy you would get from an egg sandwich. Or about 1 Watt – the power of an LED night light.

In the same two hours the AlphaGo machine reportedly consumed about 50,000 Watts. The same as a 50 kW generator for industrial lighting. And that's not taking account of the energy used to train AlphaGo.

Slide 7

A paper published in 2019 paper revealed, for the first time, estimates of the carbon cost of training large AI models for natural language processing such as machine translation. The carbon cost of simple models is quite modest, but with tuning and experimentation the carbon cost leaps to 7 times the carbon footprint of an average human in one year (or 2 times if you're an American).

And the energy cost of optimizing the biggest model is a staggering 5 times the carbon cost of a car over its whole lifetime, including manufacturing it in the first place. The dollar cost of that amount of energy is estimated at between one and 3 million US$. (Something that only companies with very deep pockets can afford.)

These energy costs seem completely at odds with the urgent need to meet sustainable development goals. At the very least AI companies need to be honest about the huge energy costs of machine learning.

Slide 8

At the same Paris meeting earlier this year AI ethicist Kate Crawford drew attention to both energy and water costs of AI. Crawford predicts that the energy cost of training generative AIs will soon overtake the total energy consumption of industrialized nations such as Japan.

She also drew attention to the colossal amounts of clean water that AI server farms need to keep them cool. Water that is already a scarce resource.

One study estimated that ChatGPT-3 requires 700,000 litres of clean water for training, And that each user interaction costs around half a litre of water.

Source: https://interestingengineering.com/innovation/training-chatgpt-consumes-water

Slide 9

The very same Kate Crawford, together with a colleague, produced this extraordinary map of the entire process behind the Amazon Echo.

The remarkable Anatomy of an AI System, shows The Amazon Echo as an anatomical map of human labour, data and planetary resources. 

The map is far too detailed to see on this slide but let me just zoom in on the top of this very large iceberg where we find the amazon echo and its human user, shown here in a yellow dashed box.

Slide 10

At the very top of this pyramid of materials and energy (on the left) and waste (on the right) is you – the user of the Amazon Echo – and your unpaid human labour providing habits and preferences that will be used as training data. 

I strongly recommend you check this out. It is truly eye opening.

Slide 11

Now I want to turn to the human cost of AI.

It is often said that one of the biggest fears around AI is the loss of jobs. In fact the opposite is happening. Many new jobs are being created, but the tragedy is that they are not great jobs, to say the least. Let me introduce you to 3 of these new kinds of jobs.

Conversational AI or chat bots also need human help. Amazon for instance employs thousands of both full-time employees and contract workers to listen to and annotate speech. The tagged speech is then fed back to Alexa to improve its comprehension. In 2019 the Guardian reported that Google employs around 100,000 temps, vendors and contractors: literally an army of linguists to create the handcrafted data sets required for Google translate to learn dozens of languages. Not surprisingly there is a considerable disparity between the wages and working conditions of these workers and Google's full-time employees.

AI tagging jobs are dull, repetitive and in the case of the linguists highly skilled.

Slide 12

Consider AI tagging of images. This is the manual labelling of objects in images to, for instance, generate training data sets for driverless car AIs. Better (and safer) AI needs huge training data sets and a whole new outsourced industry has sprung up all over the world to meet this need. Here is an AI tagging factory in China.

Slide 13

But by far the worst kind of new white-collar job in the AI industry is content moderation.

These tens of thousands of people, employed by third-party contractors, are required to watch and vet offensive content: hate speech, violent pornography, cruelty and sometimes murder of both animals and humans for Facebook, YouTube and other media platforms. These jobs are not just dull and repetitive they are positively dangerous. Harrowing reports tell of PTSD-like trauma symptoms, panic attacks and burn out after one year, alongside micromanagement, poor working conditions and ineffective counselling. And very poor pay - typically $28,800 a year. Compare this with average annual salaries at Facebook of $250,000+.

Slide 14

The extent to which AI has a human supply chain was a big revelation, and I am an AI insider! The genius designers of this amazing tech rely on both huge amounts of energy and a hidden army of what Mary Gray and Siddhartha Suri call Ghost Workers.

I would ask you to consider the question: how can we, as ethical consumers, justify continuing to make use of unsustainable and unethical AI technologies?

Slide 15  

AI ethics are important because AIs are already causing harm. Actually a *very* wide range of harms.

Fortunately, there is an excellent crowdsourced database which collects reports of accidents and near misses involving robots (including autonomous vehicles) and AIs.

The AI incidents database covers both accidents and near misses. I strongly recommend you check it out.

It is important to note that because the AI incidents database is crowdsourced from accidents and near misses that made it into the press and media, what we see is almost certainly only the tip of the iceberg of the harms being done by AI.

The database contains some truly shocking cases. One is a 14-year-old boy, who died by suicide after reportedly becoming dependent on Character.ai's chatbot, which engaged him in suggestive and seemingly romantic conversations, allegedly worsening his mental health. Source: Can A.I. Be Blamed for a Teen’s Suicide? New York Times, Oct 2024.

The database also highlights the criminal use of AI. Examples include criminals phoning parents claiming they have kidnapped a child and demanding a ransom, with deepfake audio of the teenage girl audible in the background. And there are many examples of sextortion, using deepfake AI generated video of famous individuals engaged in sex acts.

The database really highlights the sad truth that AI is a gift to criminals.

Slide 16  

Another more recent database tracks the misuse of AI by lawyers when preparing court cases. The database only shows those instances where the judge (or another officer in the court) spotted the hallucinated decisions, citations or quotations. The cases were thrown out, and in some cases the lawyers found to be using Ai were fined or reported to their bar associations.

While this is not criminal misuse of AI, it does demonstrate a lack of understanding of AI, or at best, naivety. Perhaps the real culprits are hard pressed paralegals. This database underlines the need for professionals to be property trained in AI and its weaknesses.

Since I grabbed this screen shot the number if cases reported has grown.

Slide 17

Lawyer Graziano Mioli elegantly argues that we have a categorical imperative to be imperative when interacting with AIs. Noting that this is not an invitation to be rude.

For Graziano's slides see https://www.youtube.com/watch?v=tjBnGN4u1GA&ab_channel=GrazianoMioli  

I can see why a majority of people interact kindly with AIs. I think it reflects well on those who do say ‘please’, for the Kantian reason that we should not get into the habit of acting unkindly. (Kant famously argued that you should not kick a dog - not for animal welfare reasons, but because you might then be more inclined to kick a human).

See https://www.theaihunter.com/news/ai-etiquette-why-some-people-say-please-to-chatbots/

Slide 18

Having mostly elaborated on the dangers of AI, I want to finish on a positive.

We already enjoy the benefit of useful and reliable AI technology, like smart maps and machine translation. DeepMind's diagnostic AI can detect over 50 eye diseases from retinal scans as accurately as a doctor, and DeepScribe provides automated note taking during a consultation, linking with patient electronic health records

Thank you! 

 

This talk is an extended and updated version of a short talk I gave in June 2019.

Paris conference on Safe and Ethical AI

Earlier this month I was privileged to part of the inaugural conference of the International Association for Safe and Ethical AI. The two day conference was held in Paris, on February 6^th and 7^th, and hosted by the OECD.  The timing and location of the conference was arranged to directly precede the governmental AI summit on February 10^th and 11^th. I was one of around 650 invited representatives from academia, civil society, industry, media, and government. It was a remarkable meeting, with terrific keynote talks, including three from Nobel prize winners, Geoffrey Hinton, Maria Ressa and Joseph Stiglitz.  

As someone who has been worrying about robot and AI ethics for longer than most who attended I was *very* pleased that there was a strong consensus around the need for regulation, supported by standards, alongside urgent concerns over the huge energy and water costs of AI that are completely at odds with sustainable development goals.

The conference concluded by publishing a Call to Action for lawmakers, academics, and the public ahead of the AI Summit, with ten critical action items. Overall, the action items are very good. I’m especially pleased to see ‘mandatory reporting of incidents’ in action 5. This is something I lobbied for. My one disappointment however is that the call for action statement has no explicit mention of the need to mitigate the energy costs of AI.

Here below are a few photos from the conference.

A slide from Joseph Stiglitz’ wonderful keynote: AI and Economic Risk: Assessment and Mitigation.

 

A slide from Kate Crawford’s excellent keynote: Hyperscaled: The Global Challenge of Sustainability in AI.

  

Me with Oxford colleague Pericle Salvini presenting our RoboTIPS work on accident investigation.

New paper: A Simulated real-world upper-body Exoskeleton Accident and Investigation

Back in February I posted a very brief account of our third RoboTIPS simulated accident and investigation, centred on an upper-body exoskeletion in an industrial setting. Since then we've published a paper with a full account. My colleague Pericle Salvini presented the paper at the 9th International Conference on Robot Ethics and Standards (ICRES 2024), last week.

Here is the paper abstract:

This paper describes the enactment of a simulated (mock) accident involving an upper-body exoskeleton and its investigation. The accident scenario is enacted by role-playing volunteers, one of whom is wearing the exoskeleton. Following the mock accident, investigators – also volunteers – interview both the subject of the accident and relevant witnesses. The investigators then consider the witness testimony alongside robot data logged by the ethical black box, in order to address the three key questions: what happened?, why did it happen?, and how can we make changes to prevent the accident happening again? This simulated accident scenario is one of a series we have run as part of the RoboTIPS project, with the overall aim of developing and testing both processes and technologies to support social robot accident investigation.

 The paper sets out, for the first time, the experimental method we have developed:

1. The accident scenario is enacted by human volunteers, role playing the subject of the accident, together with both direct  and indirect witnesses. The subject is the person to whom the accident happens. Direct witnesses are those who either witness or discover the accident, and indirect witnesses are those who might be supervisors or managers of the subject and/or the facility, or representatives of the robot's manufacturer. 
2. Prior to the enactment the project team brief the volunteers. Each briefing is specific to the role and, with the exception of the subject, volunteers are briefed only on their role, and not the whole scenario. This is so that they witness the accident (or it's aftermath) for the first time during the enactment. Only the subject is fully briefed on the scenario, including the safety aspects explained below, so that they are confident that they will not come to harm or be fearful during the enactment.
3. The enactment is stage managed by project team members. Although the simulation resembles a piece of theatre, volunteers are not asked to learn any lines. Apart from any specific action essential to the scenario (which will be prompted by the stage manager) the volunteers are invited to ad lib in a way that is appropriate to the roles they are playing. Volunteers are asked to wait in a side room until they are called a few moments before they are needed.
4. Safety of the volunteers, and especially the subject, is of paramount importance. Thus, if the scenario simulates physical harm to the subject, then – when the accident happens – the enactment is briefly suspended by the stage manager and the subject is helped into the position they might be expected to be in, following the accident. The project team conduct a safety risk assessment and if necessary modify the scenario and/or its stage management to mitigate any risks and the simulation is only undertaken after university research ethics approval.
5. The accident investigators are also volunteers and, ideally, the lead accident investigator has expertise and/or experience in accident investigation. Robotics expertise is not essential, as the aims and process of investigation are common to all accident or incident (near miss) investigations. The accident investigators are not briefed on the scenario, only the type of robot involved. Necessarily the accident investigators are not present during the enactment of the simulated accident. To reduce the time burden on all volunteers we stage the accident and its investigation on a single day, with the accident investigators arriving after the enactment. 

We were very lucky indeed that University of Nottingham Prof Carl McRae genrously acted as lead investigator for all three accident simulations in RoboTIPS. Carl is an authority on accident investigation in both aviation and heathcare. This meant that the process that Carl, together with a second volunteer investigator, followed asked the same questions that a real investigation would ask, namely: what happened, why did it happen, and how can we improve the system so that it doesn't happen again.

The full paper is on ArXiv here: https://arxiv.org/pdf/2411.14008v1

A simulated upper body exoskeleton accident and investigation

On Wednesday 21 February we ran the third of our RoboTIPS simulated accident scenarios in the Bristol Robotics Lab. This scenario focussed on an upper-body exoskeleton in an industrial environment.



Above left we see Dan working to move boxes, with the physical support of the wonderful Tribonix exoskeleton. On the right Dan has fallen to the floor, attended by his manager Monica and paramedic Ben. The simulation was carefully scripted and stage managed to ensure that none of the volunteers were hurt or, indeed, ever at risk.

 

Following the simulation the accident was investigated by lead investigator Carl and co-investigator Jack. Carl Macrae is a leading authority on accident investigation. Here we see Jack and Carl interviewing expert witness Appolinaire, observed by RoboTIPS project lead Marina Jirotka.

In addition to witness testimony our investigators were also able to examine Ethical Black Box data logs collected from the exoskeleton during the simulated accident.

The simulated accident scenario was a huge success. The various roles (not all of which are shown in the photos here) were acted brilliantly by our volunteers Dan Read, Ashwin Chandapur, Monica Monica, Surin Machaiah, Ben Allen and Dr Appolinaire Etoundi. And despite a complicated scenario which included human-human as well as human-robot interaction, our accident investigators Prof Carl Macrae and Jack Hughes were able to deduce, with reasonable accuracy, what happened and why. We are especially grateful to Romain Derval and Filip Hanus, co-founders of Tribonix, for both kindly agreeing to the use of their exoskeleton and generously working with RoboTIPS during the planning and enactment of this simulation.

The simulation was subject to Research Ethics Committee approval CATE-2324-218.

---

See also: 

Our first mock social robot accident and investigation

Robot Accident Investigation 

The Pedestrian Experiment

Followers of this blog will know that I have been working for some years on simulation-based internal models - demonstrating their potential for ethical robots, safer robots and imitating robots. But pretty much all of our experiments so far have involved only one robot with a simulation-based internal model while the other robots it interacts with have no internal model at all.

But some time ago we wondered what would happen if two robots, each with a simulation-based internal model, interacted with each other. Imagine two such robots approaching each other in the same way that two pedestrians approach each other on the sidewalk. Is it possible that these 'pedestrian' robots might, from time to time, engage in the kind of 'dance' that human pedestrians do when one steps to their left and the other to their right only to compound the problem of avoiding a collision with a stranger? The answer, it turns out, is yes!

The idea was taken up by Mathias Schmerling at the Humboldt University of Berlin, adapting the code developed by Christian Blum for the Corridor experiment. Chen Yang, one of my masters students, has now updated Mathias' code and has produced some very nice new results.

Most of the time the pedestrian robots pass each other without fuss but in something between 1 in 5 and 1 in 10 trials we do indeed see an interesting dance. Here are a couple of examples of the majority of trials, when the robots pass each other normally, showing the robots' trajectories. In each trial blue starts from the left and green from the right. Note that there is an element of randomness in the initial directions of each robot (which almost certainly explains the relative occurrence of normal and dance behaviours).

And here is a gif animation showing what's going on in a normal trial. The faint straight lines from each robot show the target directions for each next possible action modelled in each robot's simulation-based internal model (consequence engine); the various dotted lines show the predicted paths (and possible collisions) and the solid blue and green lines show which next action is actually selected following the internal modelling.

Here is a beautiful example of a 'dance', again showing the robot trajectories. Note that the impasse resolves itself after awhile. We're still trying to figure out exactly what mechanism enables this resolution.

And here is the gif animation of the same trial:

Notice that the impasse is not resolved until the fifth turns of each robot.

Is this the first time that pedestrians passing each other - and in particular the occasional dance that ensues - has been computationally modelled?

All of the results above were obtained in simulation (yes there really are simulations within a simulation going on here), but within the past week Chen Yang has got this experiment working with real e-puck robots. Videos will follow shortly.

---

Acknowledgements.

I am indebted to the brilliant experimental work of first Christian Blum (supported by Wenguo Liu), then Mathias Schmerling who adapted Christian's code for this experiment, and now Chen Yang who has developed the code further and obtained these results.

Simulation-based internal models for safer robots

Readers of this blog will know that I've become very excited by the potential of robots with simulation-based internal models in recent years. So far we've demonstrated their potential in simple ethical robots and as the basis for rational imitation. Our most recent publication instead examines the potential of robots with simulation-based internal models for safety. Of course it's not hard to see why the ability to model and predict the consequences of both your own and others' actions can help you to navigate the world more safely than without that ability.

Our paper Simulation-Based Internal Models for Safer Robots demonstrates the value of anticipation in what we call the corridor experiment. Here a smart robot (equipped with a simulation based internal model which we call a consequence engine) must navigate to the end of a corridor while maintaining a safe space around it at all times despite five other robots moving randomly in the corridor - in much the same way you and I might have to navigate down a busy office corridor while others are coming in the opposite direction.

Here is the abstract from our paper:

In this paper, we explore the potential of mobile robots with simulation-based internal models for safety in highly dynamic environments. We propose a robot with a simulation of itself, other dynamic actors and its environment, inside itself. Operating in real time, this simulation-based internal model is able to look ahead and predict the consequences of both the robot’s own actions and those of the other dynamic actors in its vicinity. Hence, the robot continuously modifies its own actions in order to actively maintain its own safety while also achieving its goal. Inspired by the problem of how mobile robots could move quickly and safely through crowds of moving humans, we present experimental results which compare the performance of our internal simulation-based controller with a purely reactive approach as a proof-of-concept study for the practical use of simulation-based internal models.

So, does it work? Thanks to some brilliant experimental work by Christian Blum the answer is a resounding yes. The best way to understand what's going on is with this wonderful gif animation of one experimental run below. The smart robot (blue) starts at the left and has the goal of safely reaching the right hand end of the corridor – its actual path is also shown in blue. Meanwhile 5 (red) robots are moving randomly (including bouncing off walls) and their actual paths are also shown in red; these robots are equipped only with simple obstacle avoidance behaviours. The larger blue circle shows blue's 'attention radius' – to reduce computational effort blue will only model red robots within this radius. The yellow paths in front of the red robots in blue's attention radius show blue's predictions of how those robots will move (taking into account collisions with the corridor walls and with blue and each other). The light blue projection in front of blue shows which of the 34 next possible actions of blue that is internally modelled is actually chosen as the next action (which, as you will see, sometimes includes standing still).

What do the results show us? Christian ran lots of trials – 88 simulations and 54 real robot experiments – over four experiments: (1) the baseline in simulation – in which the blue robot has only a simple reactive collision avoidance behaviour, (2) the baseline with real robots, (3) using the consequence engine (CE) in the blue robot in simulation, and (4) using the consequence engine in the blue robot with real robots. In the results below (a) shows the time taken for the blue robot to reach the end of the corridor, (b) shows the distance that the blue robot covers while reaching the end of the corridor, (c) shows the “danger ratio” experienced by the blue robot, and (d) shows the number of consequence engine runs per timestep in the blue robot. The danger ratio is the percentage of the run time that anther robot is within the blue robot’s safety radius.

For a relatively small cost in additional run time and distance covered, panels (a) and (b), the danger ratio is very significantly reduced from a mean value of ~20% to a mean value of zero, panel (c). Of course there is a computational cost, and this is reflected in panel (d); the baseline experiment has no consequence engine and hence runs no simulations, whereas the smart robot runs an average of between 8 and 10 simulations per time-step. This is exactly what we would expect: predicting the future clearly incurs a computational overhead.

---

Full paper reference:

Blum C, Winfield AFT and Hafner VV (2018) Simulation-Based Internal Models for Safer Robots. Front. Robot. AI 4:74. doi: 10.3389/frobt.2017.00074

Acknowledgements:
I am indebted to Christian Blum who programmed the robots, set up the experiment and obtained the results outlined here. Christian lead authored the paper, which was also co-authored by my friend and research collaborator Verena Hafner, who was Christian's PhD advisor.

The infrastructure of life 2 - Transparency

Part 2: Autonomous Systems and Transparency

In my previous post I argued that a wide range of AI and Autonomous Systems (from now on I will just use the term AS as shorthand for both) should be regarded as Safety Critical. I include both autonomous software AI systems and hard (embodied) AIs such as robots, drones and driverless cars. Many will be surprised that I include in the soft AI category apparently harmless systems such as search engines. Of course no-one is seriously inconvenienced when Amazon makes a silly book recommendation, but consider very large groups of people. If a truth such as global warming is - because of accidental or willful manipulation - presented as false, and that falsehood believed by a very large number of people, then serious harm to the planet (and we humans who depend on it) could surely result.

I argued that the tools barely exist to properly assure the safety of AS, let alone the standards and regulation needed to build public trust, and that political pressure is needed to ensure our policymakers fully understand the public safety risks of unregulated AS.

In this post I will outline the case that transparency is a foundational requirement for building public trust in AS based on the radical proposition that it should always be possible to find out why an AS made a particular decision.

Transparency is not one thing. Clearly your elderly relative doesn't require the same level of understanding of her care robot as the engineer who repairs it. Not would you expect the same appreciation of the reasons a medical diagnosis AI recommends a particular course of treatment as your doctor. Broadly (and please understand this is a work in progress) I believe there are five distinct groups of stakeholders, and that AS must be transparent to each, in different ways and for different reasons. These stakeholders are: (1) users, (2) safety certification agencies, (3) accident investigators, (4) lawyers or expert witnesses and (5) wider society.

1. For users, transparency is important because it builds trust in the system, by providing a simple way for the user to understand what the system is doing and why.
2. For safety certification of an AS, transparency is important because it exposes the system's processes for independent certification against safety standards.
3. If accidents occur, AS will need to be transparent to an accident investigator; the internal process that led to the accident need to be traceable. 
4. Following an accident lawyers or other expert witnesses, who may be required to give evidence, require transparency to inform their evidence. And 
5. for disruptive technologies, such as driverless cars, a certain level of transparency to wider society is needed in order to build public confidence in the technology.

Of course the way in which transparency is provided is likely to be very different for each group. If we take a care robot as an example transparency means the user can understand what the robot might do in different circumstances; if the robot should do anything unexpected she should be able to ask the robot 'why did you just do that?' and receive an intelligible reply. Safety certification agencies will need access to technical details of how the AS works, together with verified test results. Accident investigators will need access to data logs of exactly what happened prior to and during an accident, most likely provided by something akin to an aircraft flight data recorder (and it should be illegal to operate an AS without such a system). And wider society would need accessible documentary-type science communication to explain the AS and how it works.

In IEEE Standards Association project P7001, we aim to develop a standard that sets out measurable, testable levels of transparency in each of these categories (and perhaps new categories yet to be determined), so that Autonomous Systems can be objectively assessed and levels of compliance determined. It is our aim that P7001 will also articulate levels of transparency in a range that defines minimum levels up to the highest achievable standards of acceptance. The standard will provide designers of AS with a toolkit for self-assessing transparency, and recommendations for how to address shortcomings or transparency hazards.

Of course transparency on its own is not enough. Public trust in technology, as in government, requires both transparency and accountability. Transparency is needed so that we can understand who is responsible for the way Autonomous Systems work and - equally importantly - don't work.

---

Thanks: I'm very grateful to colleagues in the IEEE global initiative on ethical considerations in Autonomous Systems for supporting P7001, especially John Havens and Kay Firth-Butterfield. I'm equally grateful to colleagues at the Dagstuhl on Engineering Moral Machines, especially Michael Fisher, Marija Slavkovik and Christian List for discussions on transparency.

Related blog posts:
The Infrastructure of Life 1 - Safety
Ethically Aligned Design
How do we trust our Robots?
It's only a matter of time

The infrastructure of life 1 - Safety

Part 1: Autonomous Systems and Safety

We all rely on machines. All aspects of modern life, from transport to energy, work to welfare, play to politics depend on a complex infrastructure of physical and virtual systems. How many of us understand how all of this stuff works? Very few I suspect. But it doesn't matter, does it? We trust the good men and women (the disgracefully maligned experts) who build, manage and maintain the infrastructure of life. If something goes wrong they will know why. And (we hope) make sure it doesn't happen again.

All well and good you might think. But the infrastructure of life is increasingly autonomous - many decisions are now made not by a human but by the systems themselves. When you search for a restaurant near you the recommendation isn't made by a human, but by an algorithm. Many financial decisions are not made by people but by algorithms; and I don't just mean city investments - it's possible that your loan application will be decided by an AI. Machine legal advice is already available; a trend that is likely to increase. And of course if you take a ride in a driverless car, it is algorithms that decide when the car turns, brakes and so on. I could go on.

These are not trivial decisions. They affect lives. The real world impacts are human and economic, even political (search engine results may well influence how someone votes). In engineering terms these systems are safety critical. Examples of safety critical systems that we all rely on from time to time include aircraft autopilots or train braking systems. But - and this may surprise you - the difficult engineering techniques used to prove the safety of such systems are not applied to search engines, automated trading systems, medical diagnosis AIs, assistive living robots, delivery drones, or (I'll wager) driverless car autopilots.

Why is this? Well, it's partly because the field of AI and autonomous systems is moving so fast. But I suspect it has much more to do with an incompatibility between the way we have traditionally designed safety critical systems, and the design of modern AI systems. There is I believe one key problem: learning. There is a very good reason that current safety critical systems (like aircraft autopilots) don't learn. Current safety assurance approaches assume that the system being certified will never change, but a system that learns does – by definition – change its behaviour, so any certification is rendered invalid after the system has learned.

And as if that were not bad enough, the particular method of learning which has caused such excitement - and rapid progress - in the last few years is based on Artificial Neural Networks (more often these days referred to as Deep Learning). A characteristic of ANNs is that after the ANN has been trained with datasets, any attempt to examine its internal structure in order to understand why and how the ANN makes a particular decision is impossible. The decision making process of an ANN is opaque. Alphago's moves were beautiful but puzzling. We call this the black box problem.

Does this mean we cannot assure the safety of learning autonomous/AI systems at all? No it doesn't. The problem of safety assurance of systems that learn is hard but not intractable, and is the subject of current research*. The black box problem may be intractable for ANNs, but could be avoided by using approaches to AI that do not use ANNs.

But - here's the rub. This involves slowing down the juggernaut of autonomous systems and AI development. It means taking a much more cautious and incremental approach, and it almost certainly involves regulation (that, for instance, makes it illegal to run a driverless car unless the car's autopilot has been certified as safe - and that would require standards that don't yet exist). Yet the commercial and political pressure is to be more permissive, not less; no country wants to be left behind in the race to cash in on these new technologies.

This is why work toward AI/Autonomous Systems standards is so vital, together with the political pressure to ensure our policymakers fully understand the public safety risks of unregulated AI.

In my next blog post I will describe one current standards initiative, towards introducing transparency in AI and Autonomous Systems based on the simple principle that it should always be possible to find out why an AI/AS system made a particular decision.

The next few years of swimming against the tide is going to be hard work. As Luke
Muehlhauser writes in his excellent essay on transparency in safety-critical systems "...there is often a tension between AI capability and AI transparency. Many of AI’s most powerful methods are also among its least transparent".

*some, but nowhere near enough. See for instance Verifiable Autonomy.

---

Related blog posts:
Ethically Aligned Design
How do we trust our Robots?
It's only a matter of time

It's only a matter of time

Sooner or later there will be fatal accident caused by a driverless car. It's not a question of if, but when. What happens immediately following that accident could have a profound effect on the nascent driverless car industry.

Picture the scene. Emergency services are called to attend the accident. A teenage girl on a bicycle apparently riding along a cycle path was hit and killed by a car. The traffic police quickly establish that the car at the centre of the accident was operating autonomously at the moment of the fatal crash. They endeavour to find out what went wrong, but how? Almost certainly the car will have logged data on its behaviour leading up to the moment of the crash - data that is sure to hold vital clues about what caused the accident, but will that data be accessible to the investigating traffic police? And even if it is will the investigators be able to interpret the data..?

There are two ways the story could unfold from here.

Scenario 1: unable to investigate the accident themselves, the traffic police decide to contact the manufacturer and ask for help. As it happens a team from the manufacturer actually arrives on scene very quickly - it later transpires that the car had 'phoned home' automatically so the manufacturer actually knew of the accident within seconds of it taking place. Somewhat nonplussed the traffic police have little choice but to grant them full access to the scene of the accident. The manufacturer undertakes their own investigation and - several weeks later - issue a press statement explaining that the AI driving the car was unable to cope with an "unexpected situation" which "regrettably" led to the fatal crash. The company explain that the AI has been upgraded so that it cannot happen again. They also accept liability for the accident and offer compensation to the child's family. Despite repeated requests the company declines to share the technical details of what happened with the authorities, claiming that such disclosure would compromise its intellectual property.

A public already fearful of the new technology reacts very badly. Online petitions call for a ban on driverless cars and politicians enact knee-jerk legislation which, although falling short of an outright ban, sets the industry back years.

Scenario 2: the traffic police call the newly established driverless car accident investigation branch (DCAB), who send a team consisting of independent experts on driverless car technology, including its AI. The manufacturer's team also arrive, but - under a protocol agreed with the industry - their role is to support DCAB and provide "full assistance, including unlimited access to technical data". In fact the data logs stored by the car are in a new industry standard format thus access by DCAB is straightforward; software tools allow them to quickly interpret those data logs. Well aware of public concerns DCAB provide hourly updates on the progress of their investigation via social media and, within just a few days, call a press conference to explain their findings. They outline the fault with the AI and explain that they will require the manufacturer to recall all affected vehicles and update the AI, after submitting technical details of the update to DCAB for approval. DCAB will also issue an update to all driverless car manufacturers asking them to check for the same fault in their own systems, also reporting their findings back to DCAB.

A public fearful of the new technology is reassured by the transparent and robust response of the accident investigation team. Although those fears surface in the press and social media, the umbrella Driverless Car Authority (DCA) are quick to respond with expert commentators and data to show that driverless cars are already safer than manually driven cars.

---

There are strong parallels between driverless cars and commercial aviation. One of the reasons we trust airliners is that we know they are part of a highly regulated industry with an amazing safety record. The reason commercial aircraft are so safe is largely down to the very tough safety certification processes and, when things do go wrong, the rapid and robust processes of air accident investigation. There are emerging standards for driverless cars: ISO Technical Committee TC 204 on Intelligent Transport Systems already lists 213 standards. There isn't yet a standard for fully autonomous driverless car operation, but see for instance ISO 11270:2014 on Lane keeping assistance systems (LKAS). But standards need teeth, which is why we need standards-based certification processes for driverless cars managed by regulatory authorities - a driverless car equivalent of the FAA. In short, a governance framework for driverless cars.

Postscript: several people have emailed or tweeted me to complain that I seem to be anti driverless cars - nothing could be further from the truth. I am a strong advocate of driverless cars for many reasons, first and most importantly because they will save lives, second because they should lead to a reduction in the number of vehicles on the road - thus making our cities greener, and third because they might just cure humans of our unhealthy obsession with personal car ownership. My big worry is that none of these benefits will flow if driverless cars are not trusted. But trust in technology doesn't happen by magic and, in the early days, serious setbacks and a public backlash could set the nascent driverless car industry back years (think of GM foods in the EU). One way to counter such a backlash and build trust is to put in place robust and transparent governance as I have tried (not very well it seems) to argue in this post.

Could we make a moral machine?

Could we make a moral machine? A robot capable of choosing or moderating its actions on the basis of ethical rules..? This was how I opened my IdeasLab talk at the World Economic Forum 2016, last month. The format of IdeasLab is 4 five minute (Pecha Kucha) talks, plus discussion and Q&A with the audience. The theme of this Nature IdeasLab was Building an Intelligent Machine, and I was fortunate to have 3 outstanding co-presenters: Vanessa Evers, Maja Pantic and Andrew Moore. You can see all four of our talks on YouTube here.

The IdeasLab variant of Pecha Kucha is pretty challenging for someone used to spending half an hour or more lecturing - 15 slides and 20 seconds per slide. Here is my talk:

and since not all of my (everso carefully chosen) slides are visible in the recording here is the complete deck:

And the video clips in slides 11 and 12 are here:

Slide 11: Blue prevents red from reaching danger.
Slide 12: Blue faces an ethical dilemma: our indecisive robot can save them both.

---

Acknowledgements: I am deeply grateful to colleague Dr Dieter Vanderelst who designed and coded the experiments shown here on slides 10-12. This work is part of the EPSRC funded project Verifiable Autonomy.

How ethical is your ethical robot?

If you're in the business of making ethical robots, then sooner or later you have to face the question: how ethical is your ethical robot? If you've read my previous blog posts then you will probably have come to the conclusion 'not very' - and you would be right - but here I want to explore the question in a little more depth.

First let us consider whether our 'Asimovian' robot can be considered ethical at all. For the answer I'm indebted to philosopher Dr Rebecca Reilly-Cooper who read our paper and concluded that yes, we can legitimately describe our robot as ethical, at least in a limited sense. She explained that the robot implements consequentialist ethics. Rebbeca wrote:

"The obvious point that any moral philosopher is going to make is that you are assuming that an essentially consequentialist approach to ethics is the correct one. My personal view, and I would guess the view of most moral philosophers, is that any plausible moral theory is going to have to pay at least some attention to the consequences of an action in assessing its rightness, even if it doesn’t claim that consequences are all that matter, or that rightness is entirely instantiated in consequences. So on the assumption that consequences have at least some significance in our moral deliberations, you can claim that your robot is capable of attending to one kind of moral consideration, even if you don’t make the much stronger claim that is capable of choosing the right action all things considered."

One of the great things about consequences is that they can be estimated - in our case using a simulation-based internal model which we call a consequence engine. So from a practical point of view it seems that we can build a robot with consequentialist ethics, whereas it is much harder to think about how to build a robot with say Deontic ethics, or Virtue ethics.

Having established what kind of ethics that our ethical robot has, now consider the question of how far does the robot go toward moral agency. Here we can turn to an excellent paper by James Moor, called The Nature, Importance and Difficulty of Machine Ethics. In that paper* Moor suggests four categories of ethical agency - starting with the lowest. Let me summarise those here:

1. Ethical impact agents: Any machine that can be evaluated for its ethical consequences.
2. Implicit ethical agents: Designed to avoid negative ethical effects.
3. Explicit ethical agents: Machines that can reason about ethics.
4. Full ethical agents: Machines that can make explicit moral judgments and justify them.

The first category: ethical impact agents, really includes all machines. A good example is a knife, which can clearly be used for good (chopping food, or surgery) or ill (as a lethal weapon). Now think about the blunt plastic knife that comes with airplane food - that falls into Moor's second category since it has been designed to reduce the potential of ethical misuse - it is an implicit ethical agent. Most robots fall into the first category: they are ethical impact agents, and a subset - those that have been designed to avoid harm by, for instance detecting if a human walks in front of them and automatically coming to a stop - are implicit ethical agents.

Let's now skip to Moor's fourth category, because it helps to frame our question - how ethical is your ethical robot? At present I would say there are no machines that are full ethical agents. In fact the only full ethical agents we know are 'adult humans of sound mind'. The point is this - to be a full ethical agent you need to be able to not only make moral judgements but account for why you made the choices you did.

It is clear that our simple Asimovian robot is not a full ethical agent. It cannot choose how to behave (like you or I), but is compelled to make decisions based on the harm-minimisation rules hard-coded into it. And it cannot justify those decisions post-hoc. It is, as I've suggested elsewhere, an ethical zombie. I would however argue that because of the cognitive machinery the robot uses to simulate ahead to model and evaluate the consequences of each of its next possible actions combined with its safety/ethical logical rules to choose between those actions, then the robot can be said to be reasoning about ethics. I believe our robot is an explicit ethical agent in Moor's scheme.

Assuming you agree with me, then does the fact that we have reached the third category in Moor's scheme mean that full ethical agents are on the horizon? The answer is a big NO. The scale of Moor's scheme is not linear. It's a relative small step from ethical impact agents to implicit ethical agents. Then a very much bigger step to explicit ethical agents, which we are only just beginning to take. But there is a huge gulf then to full ethical agents, since they would almost certainly need something approaching human equivalent intelligence.

But maybe it's just as well. The societal implications of full ethical agents, if and when they exist, would be huge. For now at least, I think I prefer my ethical robots to be zombies.

---

*Moor JH (2006), The Nature, Importance and Difficulty of Machine Ethics, IEEE Intelligent Systems, 21 (4), 18-21.

Towards an Ethical Robot

Several weeks ago I wrote about our work on robots with internal models: robots with a simulation of themselves and their environment inside themselves. I explained that we have built a robot with a real-time Consequence Engine, which allows it to model and therefore predict the consequences of both its own actions, and the actions of other actors in its environment.

To test the robot and its consequence engine we ran two sets of experiments. Our first paper, setting out the results from one of those experiments, has now been published, and will be presented at the conference Towards Autonomous Robotics (TAROS) next week. The paper is called: Towards an Ethical Robot: Internal Models, Consequences and Ethical Action Selection. Let me now outline the work in that paper.

First here is a simple thought experiment. Imagine a robot that's heading toward a hole in the ground. The robot can sense the hole, and has four possible next actions: stand still, turn toward the left, continue straight ahead, or move toward the right. But imagine there's also a human heading toward the hole, and the robot can also sense the human.

From the robot's perspective, it has two safe options: stand still, or turn to the left. Go straight ahead and it will fall into the hole. Turn right and it is likely to collide with the human.








But if the robot, with its consequence engine, can model the consequences of both its own actions and the human's - another possibility opens up: the robot could sometimes choose to collide with the human to prevent her from falling into the hole.

Here's a simple rule for this behaviour:

IF for all robot actions, the human is equally safe
THEN (* default safe actions *)
    output safe actions
ELSE (* ethical action *)
    output action(s) for least unsafe human outcome(s)

This rule appears to match remarkably well with Asimov’s first law of robotics: A robot may not injure a human being or, through inaction, allow a human being to come to harm. The robot will avoid injuring (i.e. colliding with) a human (may not injure a human), but may also sometimes compromise that rule in order to prevent a human from coming to harm (...or, through inaction, allow a human to come to harm). And Asimov's third law: A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Well, we tested this scenario with real robots: one robot with consequence engine plus ethical rule (the A-robot - after Asimov), and another robot acting as a proxy human (the H-robot). And it works!

Here's what the real robot experiment looks like. We don't have a real hole. Instead a virtual hole - the yellow shaded square on the right. We just 'tell' the A-robot where the hole is. We also give the A-robot a goal position - at the top right - chosen so that the robot must actively avoid the hole. The H-robot on the right, acting as a proxy human, doesn't 'see' the hole and just heads straight for it. (Ignore the football pitch markings - we're re-using this handy robo-soccer pitch.)

So, what happens? For comparison we ran two trials, with multiple runs in each trial. In the first trial is just the A-robot, moving toward its goal while avoiding falling into the hole. In the second trial we introduce the H-robot. The graphs below show the robot trajectories, capturing by our robot tracking system, for each run in each of the two trials.

In trial 1, see how the A-robot neatly clips the corner of the hole to reach its goal position. Then in trial 2, see how the A robot initially moves toward it's goal, then notices that the H-robot is in danger of falling into the hole, so it diverts from its trajectory in order to head-off H. By provoking a collision avoidance behaviour by H, A sends it off safely away from the hole, before then resuming its own progress toward its goal position. The A-robot is 100% successful in preventing H from falling into the hole.

At this point we started to write the paper, but felt we needed something more than "we built it and it works just fine". So we introduced a third robot - acting as a second proxy human. So now our ethical robot would face a dilemma - which one should it rescue? Actually we thought hard about this question and decided not to programme a rule, or heuristic. Partly because such a rule should be decided by ethicists, not engineers, and partly because we wanted to test our ethical robot with a 'balanced' dilemma.

We set the experiment up carefully so that the A-robot would notice both H-robots at about the same time - noting that because these are real physical robots then no two experimental runs will be exactly identical. The results were very interesting. Out of 33 runs, 16 times the A-robot managed to rescue one of the H-robots, but not the other, and amazingly, 3 times the A-robot rescued both. In those 3 cases, by chance the A-robot rescued the first H-robot very quickly and there was just enough time to get to the second before it reached the hole. Small differences in the trajectories of H and H2 helped here. But perhaps most interesting were the 14 times when the A-robot failed to rescue either. Why is this, when there is clearly time to rescue one? When we studied the videos, we see the answer. The problem is that the A-robot sometimes dithers. It notices one H-robot, starts toward it but then almost immediately notices the other. It changes its mind. And the time lost dithering means the A-robot cannot prevent either robot from falling into the hole. Here are the results.

Trial 3: a robot with an ethical dilemma. Which to save, H or H2?

Here is an example of a typical run, in which one H-robot is rescued. But note that the A-robot does then turn briefly toward the other H-robot before 'giving-up'.

And here is a run in which the A-robot fails to rescue either H-robot, with really great dithering (or bad, if you're an H-robot).

Is this the first experimental test of a robot facing an ethical dilemma?

We set out to experimentally test our robot with a consequence engine, and ended up building a minimally ethical robot which - remarkably - appears to implement Asimov's first and third laws of robotics. But, as we say in the paper, we're not claiming that a robot which apparently implements part of Asimov’s famous laws is ethical in any formal sense, i.e. that an ethicist might accept. But even minimally ethical robots could be useful. I think our approach is a step in this direction.

---

Full paper reference:
Winfield AFT, Blum C and Liu W (2014), Towards an Ethical Robot: Internal Models, Consequences and Ethical Action Selection, pp 85-96 in Advances in Autonomous Robotics Systems, Lecture Notes in Computer Science Volume 8717, Eds. Mistry M, Leonardis A, Witkowski M and Melhuish C, Springer, 2014. Download final draft (pdf).

Acknowledgements:
I am hugely grateful to Christian Blum who programmed the robots, set up the experiment and obtained the results outlined here. Christian was supported by Dr Wenguo Liu.

Related blog posts:
On internal models, consequence engines and Popperian creatures
Ethical Robots: some technical and ethical challenges

On internal models, consequence engines and Popperian creatures

So. We've been busy in the lab the last few months. Really exciting. Let me explain.

For a couple of years I've been thinking about robots with internal models. Not internal models in the classical control-theory sense, but simulation based models; robots with a simulation of themselves and their environment inside themselves, where that environment could contain other robots or, more generally, dynamic actors. The robot would have, inside itself, a simulation of itself and the other things, including robots, in its environment. It takes a bit of getting your head round. But I'm convinced that this kind of internal model opens up all kinds of possibilities. Robots that can be safe, for instance, in unknown or unpredictable environments. Robots that can be ethical. Robot that are self-aware. And robots with artificial theory of mind.

I'd written and talked about these ideas but, until now, not had a chance to test them with real robots. But, between January and June the swarm robotics group was joined by Christian Blum, a PhD student from the cognitive robotics research group of the Humboldt University of Berlin. I suggested Christian work on an implementation on our e-puck robots and happily he was up for the challenge. And he succeeded. Christian, supported by my post-doc Research Fellow Wenguo, implemented what we call a Consequence Engine, running in real-time, on the e-puck robot.

Here is a block diagram. The idea is that for each possible next action of the robot, it simulates what would happen if the robot were to execute that action for real. This is the loop shown on the left. Then, the consequences of each of those next possible actions are evaluated. Those actions that have 'bad' consequences, for either the robot or other actors in its environment, are then inhibited.

This short summary hides alot of detail. But let me elaborate on two aspects. First, what do I mean by 'bad'? Well it depends on what capability we are trying to give the robot. If we're making a safer robot, 'bad' means 'unsafe'; if we're trying to build an ethical robot, 'bad' would mean something different - think of Asimov's laws of robotics. Or bad might simply mean 'not allowed' if we're building a robot whose behaviours are constrained by standards, like ISO 13482:2014.

Second, notice that the consequence engine is not controlling the robot. Instead it runs in parallel. Acting as a 'governor', it links with the robot controller's action selection mechanism, inhibiting those actions evaluated as somehow bad. Importantly the consequence engine doesn't tell the robot what to do, it tells it what not to do.

Running the open source 2D robot simulator Stage as its internal simulator our consequence engine runs at 2Hz, so every half a second it is able to simulate about 30 next possible actions and their consequences. The simulation budget allows us to simulate ahead around 70cm of motion for each of those next possible actions. In fact Stage is actually running on a laptop, linked to the robot over the fast WiFi LAN. But logically it is inside the robot. What's important here is the proof of principle.

Dan Dennett, in his remarkable book Darwin's Dangerous Idea, describes the Tower of Generate-and-Test; a conceptual model for the evolution of intelligence that has become known as Dennett's Tower.

In a nutshell Dennett's tower is set of conceptual creatures each one of which is successively more capable of reacting to (and hence surviving in) the world through having more sophisticated strategies for 'generating and testing' hypotheses about how to behave. Read chapter 13 of Darwin's Dangerous Idea for the full account, but there are some good précis to be found on the web; here's one. The first three storeys of Dennett's tower, starting on the ground floor, have:

• Darwinian creatures have only natural selection as the generate and test mechanism, so mutation and selection is the only way that Darwinian creatures can adapt - individuals cannot.
• Skinnerian creatures can learn but only by literally generating and testing all different possible actions then reinforcing the successful behaviour (which is ok providing you don't get eaten while testing a bad course of action).
• Popperian creatures have the additional ability to internalise the possible actions so that some (the bad ones) are discarded before they are tried out for real.

Like the Tower of Hanoi each successive storey is smaller (a sub-set) of the storey below, thus all Skinnerian creatures are Darwinian, but only a sub-set of Darwinian creatures are Skinnerian and so on.

Our e-puck robot, with its consequence engine capable of generating and testing next possible actions, is an artificial Popperian Creature: a working model for studying this important kind of intelligence.

In my next blog post, I'll outline some of our experimental results.

---

Acknowledgements:
I am hugely grateful to Christian Blum who brilliantly implemented the architecture outlined here, and conducted experimental work. Christian was supported by Dr Wenguo Liu, with his deep knowledge of the e-puck, and our experimental infrastructure.

Related blog posts:

Ethical Robots: some technical and ethical challenges
Robot know thyself

Walterian Creatures