Paris conference on Safe and Ethical AI

Earlier this month I was privileged to part of the inaugural conference of the International Association for Safe and Ethical AI. The two day conference was held in Paris, on February 6^th and 7^th, and hosted by the OECD.  The timing and location of the conference was arranged to directly precede the governmental AI summit on February 10^th and 11^th. I was one of around 650 invited representatives from academia, civil society, industry, media, and government. It was a remarkable meeting, with terrific keynote talks, including three from Nobel prize winners, Geoffrey Hinton, Maria Ressa and Joseph Stiglitz.  

As someone who has been worrying about robot and AI ethics for longer than most who attended I was *very* pleased that there was a strong consensus around the need for regulation, supported by standards, alongside urgent concerns over the huge energy and water costs of AI that are completely at odds with sustainable development goals.

The conference concluded by publishing a Call to Action for lawmakers, academics, and the public ahead of the AI Summit, with ten critical action items. Overall, the action items are very good. I’m especially pleased to see ‘mandatory reporting of incidents’ in action 5. This is something I lobbied for. My one disappointment however is that the call for action statement has no explicit mention of the need to mitigate the energy costs of AI.

Here below are a few photos from the conference.

A slide from Joseph Stiglitz’ wonderful keynote: AI and Economic Risk: Assessment and Mitigation.

 

A slide from Kate Crawford’s excellent keynote: Hyperscaled: The Global Challenge of Sustainability in AI.

  

Me with Oxford colleague Pericle Salvini presenting our RoboTIPS work on accident investigation.

New paper: A Simulated real-world upper-body Exoskeleton Accident and Investigation

Back in February I posted a very brief account of our third RoboTIPS simulated accident and investigation, centred on an upper-body exoskeletion in an industrial setting. Since then we've published a paper with a full account. My colleague Pericle Salvini presented the paper at the 9th International Conference on Robot Ethics and Standards (ICRES 2024), last week.

Here is the paper abstract:

This paper describes the enactment of a simulated (mock) accident involving an upper-body exoskeleton and its investigation. The accident scenario is enacted by role-playing volunteers, one of whom is wearing the exoskeleton. Following the mock accident, investigators – also volunteers – interview both the subject of the accident and relevant witnesses. The investigators then consider the witness testimony alongside robot data logged by the ethical black box, in order to address the three key questions: what happened?, why did it happen?, and how can we make changes to prevent the accident happening again? This simulated accident scenario is one of a series we have run as part of the RoboTIPS project, with the overall aim of developing and testing both processes and technologies to support social robot accident investigation.

 The paper sets out, for the first time, the experimental method we have developed:

1. The accident scenario is enacted by human volunteers, role playing the subject of the accident, together with both direct  and indirect witnesses. The subject is the person to whom the accident happens. Direct witnesses are those who either witness or discover the accident, and indirect witnesses are those who might be supervisors or managers of the subject and/or the facility, or representatives of the robot's manufacturer. 
2. Prior to the enactment the project team brief the volunteers. Each briefing is specific to the role and, with the exception of the subject, volunteers are briefed only on their role, and not the whole scenario. This is so that they witness the accident (or it's aftermath) for the first time during the enactment. Only the subject is fully briefed on the scenario, including the safety aspects explained below, so that they are confident that they will not come to harm or be fearful during the enactment.
3. The enactment is stage managed by project team members. Although the simulation resembles a piece of theatre, volunteers are not asked to learn any lines. Apart from any specific action essential to the scenario (which will be prompted by the stage manager) the volunteers are invited to ad lib in a way that is appropriate to the roles they are playing. Volunteers are asked to wait in a side room until they are called a few moments before they are needed.
4. Safety of the volunteers, and especially the subject, is of paramount importance. Thus, if the scenario simulates physical harm to the subject, then – when the accident happens – the enactment is briefly suspended by the stage manager and the subject is helped into the position they might be expected to be in, following the accident. The project team conduct a safety risk assessment and if necessary modify the scenario and/or its stage management to mitigate any risks and the simulation is only undertaken after university research ethics approval.
5. The accident investigators are also volunteers and, ideally, the lead accident investigator has expertise and/or experience in accident investigation. Robotics expertise is not essential, as the aims and process of investigation are common to all accident or incident (near miss) investigations. The accident investigators are not briefed on the scenario, only the type of robot involved. Necessarily the accident investigators are not present during the enactment of the simulated accident. To reduce the time burden on all volunteers we stage the accident and its investigation on a single day, with the accident investigators arriving after the enactment. 

We were very lucky indeed that University of Nottingham Prof Carl McRae genrously acted as lead investigator for all three accident simulations in RoboTIPS. Carl is an authority on accident investigation in both aviation and heathcare. This meant that the process that Carl, together with a second volunteer investigator, followed asked the same questions that a real investigation would ask, namely: what happened, why did it happen, and how can we improve the system so that it doesn't happen again.

The full paper is on ArXiv here: https://arxiv.org/pdf/2411.14008v1

A simulated upper body exoskeleton accident and investigation

On Wednesday 21 February we ran the third of our RoboTIPS simulated accident scenarios in the Bristol Robotics Lab. This scenario focussed on an upper-body exoskeleton in an industrial environment.



Above left we see Dan working to move boxes, with the physical support of the wonderful Tribonix exoskeleton. On the right Dan has fallen to the floor, attended by his manager Monica and paramedic Ben. The simulation was carefully scripted and stage managed to ensure that none of the volunteers were hurt or, indeed, ever at risk.

 

Following the simulation the accident was investigated by lead investigator Carl and co-investigator Jack. Carl Macrae is a leading authority on accident investigation. Here we see Jack and Carl interviewing expert witness Appolinaire, observed by RoboTIPS project lead Marina Jirotka.

In addition to witness testimony our investigators were also able to examine Ethical Black Box data logs collected from the exoskeleton during the simulated accident.

The simulated accident scenario was a huge success. The various roles (not all of which are shown in the photos here) were acted brilliantly by our volunteers Dan Read, Ashwin Chandapur, Monica Monica, Surin Machaiah, Ben Allen and Dr Appolinaire Etoundi. And despite a complicated scenario which included human-human as well as human-robot interaction, our accident investigators Prof Carl Macrae and Jack Hughes were able to deduce, with reasonable accuracy, what happened and why. We are especially grateful to Romain Derval and Filip Hanus, co-founders of Tribonix, for both kindly agreeing to the use of their exoskeleton and generously working with RoboTIPS during the planning and enactment of this simulation.

The simulation was subject to Research Ethics Committee approval CATE-2324-218.

---

See also: 

Our first mock social robot accident and investigation

Robot Accident Investigation 

A Draft Open Standard for an Ethical Black Box

About 5 years ago we proposed that all robots should be fitted with the robot equivalent of an aircraft Flight Data Recorder to continuously record sensor and relevant internal status data. We call this an ethical black box (EBB). We argued that an ethical black box will play a key role in the processes of discovering why and how a robot caused an accident, and thus an essential part of establishing accountability and responsibility.

Since then, within the RoboTIPS project, we have developed and tested several model EBBs, including one for an e-puck robot that I wrote about in this blog, and another for the MIRO robot. With some experience under our belts, we have now drafted an Open Standard for the EBB for social robots - initially as a paper submitted to the International Conference on Robots Ethics and Standards. Let me now explain first why we need a standard, and second why it should be an open standard.

Why do we need a standard specification for an EBB? As we outline in our new paper, there are four reasons:

1. A standard approach to EBB implementation in social robots will greatly benefit accident and incident (near miss) investigations. 
2. An EBB will provide social robot designers and operators with data on robot use that can support both debugging and functional improvements to the robot. 
3. An EBB can be used to support robot ‘explainability’ functions to allow, for instance, the robot to answer ‘Why did you just do that?’ questions from its user. And,
4. a standard allows EBB implementations to be readily shared and adapted for different robots and, we hope, encourage manufacturers to develop and market general purpose robot EBBs.

And why should it be an Open Standard? Bruce Perens, author of The Open Source Definition, outlines a number of criteria an open standard must satisfy, including:

• “Availability: Open standards are available for all to read and implement.
  
• Maximize End-User Choice: Open Standards create a fair, competitive market for implementations of the standard.
• No Royalty: Open standards are free for all to implement, with no royalty or fee.
• No Discrimination: Open standards and the organizations that administer them do not favor one implementor over another for any reason other than the technical standards compliance of a vendor’s implementation.
• Extension or Subset: Implementations of open standards may be extended, or offered in subset form.”

These are *good* reasons.

The most famous and undoubtedly the most impactful Open Standards are those that specified Internet protocols, such as FTP and email. They were, and still are, called Requests for Comments (RFCs) to reflect the fact that they were - especially in the early years - drafts for revision. As a mark of respect we also regard our draft 0.1 Open Standard for an EBB for Social Robots, as an RFC. You can find draft 0.1 in Annex A of the paper on arXiv here.

Not only is this a first draft, it is also incomplete, covering only the specification of the data and its format, that should be saved in an EBB for social robots. Given that the EBB data specification is at the heart of the EBB standard, we feel that this is sufficient to be opened up for comments and feedback. We will continue to extend the specification, with subsequent versions also published on arXiv.

Please feel free to either submit comments to this blog post (best because everyone can see the comments), or by contacting me directly via email. All constructive comments that result in revisions to the standard will be acknowledged in the standard.

Our first mock social robot accident and investigation

Robot accidents are inevitable. These days the likelihood of serious accidents involving industrial robots is pretty low (but not zero), because such robots are generally inside safety cages. But a newer generation of social robots - robots designed to interact directly with people, including vulnerable elderly people or children - means that accidents are now much more likely. And if we also take into account ethical harms alongside physical harms, then the potential for accidents increases still further. Psychological harms include addiction, over trusting, or deception, and societal harms include privacy violations. For more on these ethical harms see my blog post outlining an ethical risk assessment of a smart robot teddy bear.

It has puzzled me for some years that there has been almost no research on robot accident investigation. In the RoboTIPS project we are addressing this deficit by developing both the technology - which we call an Ethical Black Box (EBB) - and the processes of robot accident investigation. One of the most exciting aspects of RoboTIPS is that we're running a series of mock, i.e. staged, social robot accidents in order to road test the EBB and investigation processes in as close to a real situation as is feasible in a research project. RoboTIPS started in March 2019, but then just as we were ready to trial our first mock accident the Covid pandemic hit, and closed down the lab.

So it was great that last week we finally managed to run the a pilot of our first (of three) mock accident scenarios. The scenario, based around an assisted living robot helping an elderly person to live independently, was sketched out in late 2019, and then - during the lockdown - rehearsed in a number of online events, including a podcast radio play for Oxford Sparks and CSI Robot during the UKRAS Festival of Robotics 2021.

Here is the scenario:

Imagine that your elderly mother, or grandmother, has an assisted living robot to help her live independently at home. The robot is capable of fetching her drinks, reminding her to take her medicine and keeping in touch with family. Then one afternoon you get a call from a neighbour who has called round and sees your grandmother collapsed on the floor. When the paramedics arrive they find the robot wandering around apparently aimlessly. One of its functions is to call for help if your grandmother stops moving, but it seems that the robot failed to do this

To enact this scenario we needed a number of volunteers: one to act as Rose - the subject of the accident, a second as the neighbour who discovers the accident and raises the alarm, a third as the paramedic who attends to Rose, a fourth who acts in the role of the cleaner and a fifth in the role of manager of the group of homes in which Rose lives. We also needed volunteers to act as members of the accident investigation team who are called in to try and discover what happened, why it happened and, if possible, what changes need to be made to how to ensure the accident doesn't happen again.

This is the mock accident taking place in the kitchen of our assisted living studio. Left shows the neighbour, acted by Paul, discovering Ross, acted by Alex, injured on the floor. (Note the chair on its side.) Right is the paramedic, role-played by Luc, attending to Ross. Meanwhile the Pepper robot is moving around somewhat aimlessly.

Our brilliant Research Fellow Dr Anouk van Maris, who organised the whole setup, persuaded five colleagues from the Bristol Robotics Lab. All were male, so Rose became Ross. Only one volunteer: Alex, who played the part of Ross, was fully briefed. The other four role played brilliantly and, although they were briefed on their roles, they were not told what was going to happened to Ross, or the part the Pepper robot played (or maybe didn't play) in the accident. Two colleagues from Oxford, Lars and Keri, kindly volunteered to act as the accident investigators. Lars and Keri also had no prior knowledge of the circumstances of the accident, and had to rely on (i) inspecting the robot and the scene of the accident, (ii) the data from the robot's EBB, and (iii) testimonies from Ross, the neighbour, the paramedic, the cleaner and the facility manager.

Here we see Lars interviewing Medhi, who acted as the house manager, while Ben, acting as the cleaner, waits to be interviewed. Inside the studio Keri is interviewing the neighbour and parademic.

So, what were the findings of our accident investigators? They did very well indeed. Close examination of the EBB data, alongside consideration of the (not always reliable) witness testimony enabled Lars and Keri to correctly deduce the role that the robot played in the accident. They were also able to make several recommendations on operational changes.  But I will not reveal their findings in detail here as we intend to run the same mock accident again soon with a different set of volunteers and - in case any of them should read this blog - I don't want to give the game away!

Acknowledgements

Very special thanks to Dr Anouk van Maris. Also Dr Pericle Salvini, who worked with Anouk in finalising the detail of the scenario and during the pilot itself. Also, huge thanks to BRL volunteers Dr Alex Smith, Dr Paul Bremner, Dr Luc Wijnen, Mehdi Sobhani and Dr Ben Ward-Cherrier. And last but not least a very big thank you to Dr Lars Kunze, Oxford Robotics Institute and Keri Grieman, Dept of Computer Science, Oxford.

From the left: Pericle, Ben, Lars, Alex, Keri, Medhi, Paul, Anouk, Luc, Lola and me. Pepper is looking nervously at Lola.

Back to Robot Coding part 3: testing the EBB

In part 2 a few weeks ago I outlined a Python implementation of the ethical black box. I described the key data structure - a dictionary which serves as both specification for the type of robot, and the data structure used to deliver live data to the EBB. I also mentioned the other key robot specific code: 

# Get data from the robot and store it in data structure spec

def getRobotData(spec):

Having reached this point I needed a robot - and a way of communicating with it - so that I could both write getRobotData(spec) and test the EBB. But how to do this? I'm working from home during lockdown, and my e-puck robots are all in the lab. Then I remembered that the excellent robot simulator V-REP (now called CoppeliaSim) has a pretty good e-puck model and some nice demo scenes. V-REP also offers multiple ways of communicating between simulated robots and external programs (see here). One of them - TCP/IP sockets - appeals to me as I've written sockets code many times, for both real-world and research applications.  Then a stroke of luck: I found that a team at Ensta-Bretagne had written a simple demo which shows how to connect a Python program to a robot in V-REP, using sockets. So, first I got that demo running and figured out how it works, then used the same approach for a simulated e-puck and the EBB. Here is a video capture of the working demo.

So, what's going on in the demo? The visible simulation views in the V-REP window show an e-puck robot following a black line which is blocked by both a potted plant and an obstacle constructed from 3 cylinders. The robot has two behaviours: line following and wall following. The EBB requests data from the e-puck robot once per second, and you can see those data in the Python shell window. Reading from left to right you will see first the EBB date and time stamp, then robot time botT, then the 3 line following sensors lfSe, followed by the 8 infra red proximity sensors irSe. The final two fields show the joint (i.e. wheel) angles jntA, in degrees, then the motor commands jntD. By watching these values as the robot follows its line and negotiates the two obstacles you can see how the line and infra red sensor values change, resulting in updated motor commands.

Here is the code - which is custom written both for this robot and the means of communicating with it - for requesting data from the robot.

# Get data from the robot and store it in spec[]

# while returning one of the following result codes

ROBOT_DATA_OK = 0

CANNOT_CONNECT = 1

SOCKET_ERROR = 2

BAD_DATA = 3

def getRobotData(spec):

    # This function connects, via TCP/IP to an ePuck robot in V-REP

    # create a TCP/IP socket and connect it to the simulated robot

    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    try:

        sock.connect(server_address_port)

    except:

        return CANNOT_CONNECT

    sock.settimeout(0.1) # set connection timeout

    

    # pack a dummy packet that will provoke data in response

    #   this is, in effect, a 'ping' to ask for a data record

    strSend = struct.pack('fff',1.0,1.0,1.0)

    sock.sendall(strSend) # and send it to V-REP

    # wait for data back from V-REP

    #   expect a packet with 1 time, 2 joints, 2 motors,   

    #   3 line sensors and 8 irSensors. All floats because V-REP

    #   total packet size = 16 x 4 = 64 bytes

    data = b''

    nch_rx = 64 # expect this many bytes from  V-REP 

    try:

        while len(data) < nch_rx:

            data += sock.recv(nch_rx)

    except:

        sock.close()

        return SOCKET_ERROR

    # unpack the received data

    if len(data) == nch_rx:

        # V-REP packs and unpacks in floats only so...

        vrx = struct.unpack('ffffffffffffffff',data)

        # now move data from vrx[] into spec[], while rounding floats

        spec["botTime"] = [ round(vrx[0],2) ] 

        spec["jntDemands"] = [ round(vrx[1],2), round(vrx[2],2) ]

        spec["jntAngles"] = [ round(vrx[3]*180.0/math.pi,2)

                              round(vrx[4]*180.0/math.pi,2) ]

        spec["lfSensors"] = [ round(vrx[5],2), 

                              round(vrx[6],2), round(vrx[7],2) ]

        for i in range(8):

            spec["irSensors"][i] = round(vrx[8+i],3)       

        result = ROBOT_DATA_OK

    else:       

        result = BAD_DATA

    sock.close()

    return result

The structure of this function is very simple: first create a socket then open it, then make a dummy packet and send it to V-REP to request EBB data from the robot. Then, when a data packet arrives, unpack it into spec, then close the socket before returning. The most complex part of the code is data wrangling.

Would a real EBB collect data in this way? Well if the EBB is embedded in the robot then probably not. Communication between the robot controller and the EBB might be via ROS messages, or even more directly, by - for instance - allowing the EBB code to access a shared memory space which contains the robot's sensor inputs, command outputs and decisions. But an external EBB, either running on a local server or in the cloud, would most likely use TCP/IP to communicate with the robot, so getRobotData() would look very much like the example here. 

Back to Robot Coding part 2: the ethical black box

In the last few days I started some serious coding. The first for 20 years, in fact, when I built the software for the BRL LinuxBots. (The coding I did six months ago doesn't really count as I was only writing or modifying small fragments of Python).

My coding project is to start building an ethical black box (EBB), or to be more accurate, a module that will allow a software EBB to be incorporated into a robot. Conceptually the EBB is very simple, it is a data logger - the robot equivalent of an aircraft Flight Data Recorder, or an automotive Event Data Recorder. Nearly five years ago I made the case, with Marina Jirotka, that all robots (and AIs) should be fitted with an EBB as standard. Our argument is very simple: without an EBB, it will be more or less impossible to investigate robot accidents, or near-misses, and in a recent paper on Robot Accident Investigation we argue that with the increasing use of social robots accidents are inevitable and will need to be investigated. 

Developing and demonstrating the EBB is a foundational part of our 5-year EPSRC funded project RoboTIPS, so it's great to be doing some hands-on practical research. Something I've not done for awhile.

Here is a block diagram showing the EBB and its relationship with a robot controller.

As shown here the data flows from the robot controller to the EBB are strictly one way. The EBB cannot and must not interfere with the operation of the robot. Coding an EBB for a particular robot would be straightforward, but I have set a tougher goal: a generic EBB module (i.e. library of functions) that would - with some inevitable customisation - apply to any robot. And I set myself the additional challenge of coding in Python, making use of skills learned from the excellent online Codecademy Python 2 course.

There are two elements of the EBB that must be customised for a particular robot. The first is the data structure used to fetch and save the sensor, actuator and decision data in the diagram above. Here is an example from my first stab at an EBB framework, using the Python dictionary structure:

# This dictionary structure serves as both 

# 1 specification of the type of robot, and each data field that

#   will be logged for this robot, &

# 2 the data structure we use to deliver live data to the EBB

# for this model let us create a minimal spec for an ePuck robot

epuckSpec = {

    # the first field *always* identifies the type of robot plus            # version and serial nos

    "robot" : ["ePuck", "v1", "SN123456"],

    # the remaining fields are data we will log, 

    # starting with the motors

    # ..of which the ePuck has just 2: left and right

    "motors" : [0,0],

    # then 8 infra red sensors

    "irSensors" : [0,0,0,0,0,0,0,0],

    # ..note the ePuck has more sensors: accelerometer, camera etc, 

    # but this will do for now

    # ePuck battery level

    "batteryLevel" : [0],

    # then 1 decision code - i.e. what the robot is doing now

    # what these codes mean will be specific to both the robot 

    # and the application

    "decisionCode" : [0]

    }

Whether a dictionary is the best way of doing this I'm not 100% sure, being new to Python (any thoughts from experienced Pythonistas welcome).

The idea is that all robot EBBs will need to define a data structure like this. All must contain the first field "robot", which names the robot's type, its version number and serial number. Then the following fields must use keywords from a standard menu, as needed. As shown in this example each keyword is followed by a list of placeholder values - in which the number of values in the list reflects the specification of the actual robot. The ePuck robot, for instance, has 2 motors and 8 infra-red sensors. 

The final field in the data structure is "decisionCode". The values stored in this field would be both robot and applications specific; for the ePuck robot these might be 1 = 'stop', 2 = 'turn left', 3 = 'turn right' and so on. We could add another value for a parameter, so the robot might decide for instance to turn left 40 degrees, so "decisionCode" : [2,40]. We could also add a 'reason' field, which would save the high-level reason for the decision, as in "decisionCode" : [2,40,"avoid obstacle right"] noting that the decision field could be a string as shown here, or a numeric code.

As I hope I have shown here the design of this data structure and its fields is at the heart of the EBB.

The second element of the EBB library that must be written for the particular robot and application, is the function which fetches data from the robot

# Get data from the robot and store it in data structure spec

def getRobotData(spec):

    

How this function is implemented will vary hugely between robots and robot applications. For our Linux enhanced ePucks with WiFi connections this is likely to be via a TCP/IP client-server, with the server running on the robot, sending data following a request from the client  getRobotData(ePuckspec) For simpler setups in which the EBB module is folded into the robot controller then accessing the required data within getRobotData() should be very straightforward.

The generic part of the EBB module will define the class EBB, with methods for both initialising the EBB and saving a new data record to the EBB. I will cover that in another blog post.

Before closing let me add that it is our intention to publish the specification of the EBB, together with the model EBB code, once it had been fully tested, as open source.

Any comments or feedback would be much appreciated.

RoboTED: a case study in Ethical Risk Assessment

A few weeks ago I gave a short paper* at the excellent International Conference on Robot Ethics and Standards (ICRES 2020), outlining a case study in Ethical Risk Assessment - see our paper here. Our chosen case study is a robot teddy bear, inspired by one of my favourite movie robots: Teddy, in A.I. Artificial Intelligence.

Although Ethical Risk Assessment (ERA) is not new - it is after all what research ethics committees do - the idea of extending traditional risk assessment, as practised by safety engineers, to cover ethical risks is new. ERA is I believe one of the most powerful tools available to the responsible roboticist, and happily we already have a published standard setting out a guideline on ERA for robotics in BS 8611, published in 2016.

Before looking at the ERA, we need to summarise the specification of our fictional robot teddy bear: RoboTed. First, RoboTed is based on the following technology:

• RoboTed is an Internet (WiFi) connected device, 
• RoboTed has cloud-based speech recognition and conversational AI (chatbot) and local speech synthesis,
• RoboTed’s eyes are functional cameras allowing RoboTed to recognise faces,
• RoboTed has motorised arms and legs to provide it with limited baby-like movement and locomotion.

And second RoboTed is designed to:

• Recognise its owner, learning their face and name and turning its face toward the child.
• Respond to physical play such as hugs and tickles.
• Tell stories, while allowing a child to interrupt the story to ask questions or ask for sections to be repeated.
• Sing songs, while encouraging the child to sing along and learn the song.
• Act as a child minder, allowing parents to both remotely listen, watch and speak via RoboTed.

The tables below summarise the ERA of RoboTED for (1) psychological, (2) privacy & transparency and (3) environmental risks. Each table has 4 columns, for the hazard, risk, level of risk (high, medium or low) and actions to mitigate the risk. BS8611 defines an ethical risk as the “probability of ethical harm occurring from the frequency and severity of exposure to a hazard”; an ethical hazard as “a potential source of ethical harm”, and an ethical harm as “anything likely to compromise psychological and/or societal and environmental well-being".

(1) Psychological Risks

(2) Security and Transparency Risks

(3) Environmental Risks

For a more detailed commentary on each of these tables see our full paper - which also, for completeness, covers physical (safety) risks.

And here are the slides from my short ICRES 2020 presentation:

Through this fictional case study we argue we have demonstrated the value of ethical risk assessment. Our RoboTed ERA has shown that attention to ethical risks can

• suggest new functions, such as “RoboTed needs to sleep now”,
• draw attention to how designs can be modified to mitigate some risks, 
• highlight the need for user engagement, and
• reject some product functionality as too risky.

But ERA is not guaranteed to expose all ethical risks. It is a subjective process which will only be successful if the risk assessment team are prepared to think both critically and creatively about the question: what could go wrong? As Shannon Vallor and her colleagues write in their excellent Ethics in Tech Practice toolkit design teams must develop the “habit of exercising the skill of moral imagination to see how an ethical failure of the project might easily happen, and to understand the preventable causes so that they can be mitigated or avoided”.

 

*Which won the conference best paper prize!

"Why Did You Just Do That?" Explainability and Artificial Theory of Mind for Social Robots

This week I have been attending (virtually) the excellent RoboPhilosophy conference, and this morning gave a plenary talk "Why did you just do that?" Here is the abstract:

An important aspect of transparency is enabling a user to understand what a robot might do in different circumstances. An elderly person might be very unsure about robots, so it is important that her assisted living robot is helpful, predictable – never does anything that puzzles or frightens her – and above all safe. It should be easy for her to learn what the robot does and why, in different circumstances, so that she can build a mental model of her robot. An intuitive approach would be for the robot to be able to explain itself, in natural language, in response to spoken requests such as “Robot, why did you just do that?” or “Robot, what would you do if I fell down?” In this talk I will outline current work, within project RoboTIPS, to apply recent research on artificial theory of mind to the challenge of providing social robots with the ability to explain themselves. 

And here are the slides:

Here are links to the movies:

On slide 16,

on slide 20,

on slide 21, and

on slide 22.

And here are the papers referenced in the talk, with links:

1. Jobin, A., Ienca, M. & Vayena, E. (2019) The global landscape of AI ethics guidelines. Nat Mach Intell 1, 389–399
2. Winfield, A. Ethical standards in robotics and AI. Nature Electronics 2, 46–48 (2019).  Pre-print here.
3. Winfield, A. F. (2018) Experiments in Artificial Theory of Mind: from safety to story telling. Front. Robot. AI 5:75.
4. Blum, C., Winfield, A. F. and Hafner, V. V. (2018) Simulation-based internal models for safer robots. Frontiers in Robotics and AI, 4 (74). pp. 1-17.
5. Vanderelst, D. and Winfield, A. F. (2018) An architecture for ethical robots inspired by the simulation theory of cognition. Cognitive Systems Research, 48. pp. 56-66.
6. Winfield AFT (2018) When Robots Tell Each Other Stories: The Emergence of Artificial Fiction. In: Walsh R., Stepney S. (eds) Narrating Complexity. Springer, Cham. Preprint here.
7. Winfield, AF and Jirotka, M. (2017) The case for an ethical black box. In: Gao, Y. et al, eds. (2017) Towards Autonomous Robot Systems. LNCS 10454, pp. 262-273, Springer. Preprint here.
8. Winfield AFT, Katie Winkle, Helena Webb, Ulrik Lyngs, Marina Jirotka and Carl Macrae, Robot Accident Investigation: a case study in Responsible Robotics, chapter submitted to RoboSoft.

and mentioned in the Q&A:

1. Winfield, AF, K. Michael, J. Pitt and V. Evers (2019) Machine Ethics: The Design and Governance of Ethical AI and Autonomous Systems [Scanning the Issue], in Proceedings of the IEEE, vol. 107, no. 3, pp. 509-517.
2. Vanderelst, D. and Winfield, A. (2018), The Dark Side of Ethical Robots, AIES '18: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society Dec 2018 Pages 317–322. 

Back to robot coding part 1: hello world

One of the many things I promised myself when I retired nearly two years ago was to get back to some coding. Why? Two reasons: one is that writing and debugging code is hugely satisfying - for those like me not smart enough to do pure maths or theoretical physics - it's the closest you can get to working with pure mind stuff. But the second is that I want to prototype a number of ideas in cognitive robots which tie together work in artificial theory of mind and the ethical black box, with old ideas on how robots telling each other stories and new ideas on how social robots might be able to explain themselves in response to questions like "Robot: what would you do if I forget to take my medicine?"

But before starting to work on the robot (a NAO) I first needed to learn Python, so completed most of the Codecadamy's excellent Learn Python 2 course over the last few weeks. I have to admit that I started learning Python with big misgivings over the language. I especially don't like the way Python plays fast and loose with variable types, allowing you to arbitrarily assign a thing (integer, float, string, etc) to a variable and then assign a different kind of thing to the same variable; very different to the strongly typed languages I have used since student days: Algol 60, Algol 68, Pascal and C. However, there are things I do like: the use of indentation as part of the syntax for instance, and lots of nice built in functions like range(), so x = range(0,10) puts a list ('array' in old money) of integers from 0 to 9 in x. 

So, having got my head around Python I finally made a start with the robot on Thursday last week. I didn't get far and it was *very* frustrating. 

Act 1: setting up on my Mac

Attempting to set things up on my elderly Mac air was a bad mistake which sent me spiralling down a rabbit hole of problems. The first thing you have to do is download and unzip the NAO API, called naoqi, from Aldebaran. The same web page then suggests you simply try to import naoqi from within Python, and if there are no errors all's well.

As soon as I got the export path commands right,  import naoqi resulted in the following error

...

Reason: unsafe use of relative rpath libboost_python.dylib in /Users/alansair/Desktop/naoqi/pynaoqi-python2.7-2.1.4.13-mac64/_qi.so with restricted binary

According to stack overflow this problem is caused by Mac OSX system integrity protection (SIP). 

Then (somewhat nervously) I tried turning SIP off, as instructed here.

But import naoqi still gives a different error. Perhaps its because my Python is in the wrong place, the Aldebaran page says it must be at /usr/local/bin/python (the default on the mac is /usr/bin. Ok so I So, reinstall python 2.7 from Python.org  so that it is in /usr/local/bin/python. But now I get another error message:

>> import naoqi

Fatal Python error: PyThreadState_Get: no current thread

Abort trap: 6

A quick search and I read: "this error shows up when a module tries to use a python library that is different than the one the interpreter uses, that is, when you mix two different pythons. I would run otool -L <dyld> on each of the dynamic libraries in the list of Binary Images, and see which ones is linked to the system Python."

At which point I admitted defeat.

Act 2: setting up on my Linux machine

Once I had established that the Python on my Linux machine was also the required version 2.7, I then downloaded and unzipped the NAO API, this time for Linux.

This time I was able to import naoqi with no errors, and within just a few minutes ran my first NAO program: hello world. 

from naoqi import ALProxy

tts = ALProxy("ALTextToSpeech", "164.168.0.17", 9559)

tts.say("Hello, world!")

whereupon my NAO robot spoke the words "Hello world". Success!

Robot Accident Investigation

Yesterday I gave an talk at the ICRA 2020 workshop Against Robot Dystopias. The workshop should have been in Paris but - like most academic meetings during the lockdown - was held online. In the zoom chat window toward the end of the workshop many of us were wistfully imagining continued discussions in a Parisian bar over a few glasses of wine. Next year I hope. The workshop was excellent and all of the talks should be online soon.

My talk was an extended version of last year's talk for AI@Oxford What could possibly go wrong. With results from our new paper Robot Accident Investigation, the talk outlines a fictional investigation of a fictional robot accident. We had hoped to stage the mock accident, in the lab, with human volunteers and report a real investigation (of a mock accident) but the lockdown put paid to that too. So we have had to use our imagination and construct - I hope plausibly - the process and findings of the accident investigation.

Here is the abstract of our paper.

Robot accidents are inevitable. Although rare, they have been happening since assembly-line robots were first introduced in the 1960s. But a new generation of social robots are now becoming commonplace. Often with sophisticated embedded artificial intelligence (AI) social robots might be deployed as care robots to assist elderly or disabled people to live independently. Smart robot toys offer a compelling interactive play experience for children and increasingly capable autonomous vehicles (AVs) the promise of hands-free personal transport and fully autonomous taxis. Unlike industrial robots which are deployed in safety cages, social robots are designed to operate in human environments and interact closely with humans; the likelihood of robot accidents is therefore much greater for social robots than industrial robots. This paper sets out a draft framework for social robot accident investigation; a framework which proposes both the technology and processes that would allow social robot accidents to be investigated with no less rigour than we expect of air or rail accident investigations. The paper also places accident investigation within the practice of responsible robotics, and makes the case that social robotics without accident investigation would be no less irresponsible than aviation without air accident investigation.

And the slides from yesterday's talk:

---

Special thanks to project colleagues and co-authors: Prof Marina Jirotka, Prof Carl Macrae, Dr Helena Webb, Dr Ulrik Lyngs and Katie Winkle.

What's the worst that could happen? Why we need robot/AI accident investigation.

Robots. What could possibly go wrong?

Imagine that your elderly mother, or grandmother, has an assisted living robot to help her live independently at home. The robot is capable of fetching her drinks, reminding her to take her medicine and keeping in touch with family. Then one afternoon you get a call from a neighbour who has called round and sees your grandmother collapsed on the floor. When the paramedics arrive they find the robot wandering around apparently aimlessly. One of its functions is to call for help if your grandmother stops moving, but it seems that the robot failed to do this. 

Fortunately your grandmother recovers but the doctors find bruising on her legs, consistent with the robot running into them. Not surprisingly you want to know what happened: did the robot cause the accident? Or maybe it didn't but made matters worse, and why did it fail to raise the alarm? 

Although this is a fictional scenario it could happen today. If it did you would be totally reliant on the goodwill of the robot manufacturer to discover what went wrong. Even then you might not get the answers you seek; it's entirely possible the robot and the company that made it are just not equipped with the tools and processes to undertake an investigation.

Right now there are no established processes for robot accident investigation. 

Of course accidents happen, and that just as true for robots as any other machinery [1].

Finding statistics is tough. But this web page shows serious accidents with industrial robots in the US since the mid 1980s. Driverless car fatalities of course make the headlines. There have been five (that we know about) since 2016. But we have next to no data on accidents in human robot interaction (HRI); that is for robots designed to interact directly with humans. Here is one - a security robot - that happened to be reported.

But a Responsible Roboticist must be interested in *all* accidents, whether serious or not. We should also be very interested in near misses; these are taken *very* seriously in aviation [2], and there is good evidence that reporting near misses improves safety.

So I am very excited to introduce our 5-year EPSRC funded project RoboTIPS – responsible robots for the digital economy. Led by Professor Marina Jirotka at the University of Oxford, we believe RoboTIPS to be the first project with the aim of systematically studying the question of how to investigate accidents with social robots.

So what are we doing in RoboTIPS..?

First we will look at the technology needed to support accident investigation.

In a paper published 2 years ago Marina and I argued the case for an Ethical Black Box (EBB) [3]. Our proposition is very simple: that all robots (and some AIs) should be equipped by law with a standard device which continuously records a time stamped log of the internal state of the system, key decisions, and sampled input or sensor data (in effect the robot equivalent of an aircraft flight data recorder). Without such a device finding out what the robot was doing, and why, in the moments leading up to an accident is more or less impossible. In RoboTIPS we will be developing and testing a model EBB for social robots.

But accident investigation is a human process of discovery and reconstruction. So in this project we will be designing and running three staged (mock) accidents, each covering a different application domain: 

• assisted living robots, 
• educational (toy) robots, and 
• driverless cars.

In these scenarios we will be using real robots and will be seeking human volunteers to act in three roles, as the 

• subject(s) of the accident, 
• witnesses to the accident, and as 
• members of the accident investigation team.

Thus we aim to develop and demonstrate both technologies and processes (and ultimately policy recommendations) for robot accident investigation. And the whole project will be conducted within the framework of Responsible Research and Innovation; it will, in effect, be a case study in Responsible Robotics.

---

The text above is the script for a very short (10 minute) TED-style talk I gave at the conference AI@Oxford today in the Impact of Trust in AI session, and here below are the slides.

---

References:

[1] Dhillon BS (1991) Robot Accidents. In: Robot Reliability and Safety. Springer, New York, NY
[2] Macrae C (2014) Close Calls: Managing risk and resilience in Airline flight safety, Palgrave macmillan.
[3] Winfield AFT and Jirotka M (2017) The Case for an Ethical Black Box. In: Gao Y, Fallah S, Jin Y, Lekakou C (eds) Towards Autonomous Robotic Systems. TAROS 2017. Lecture Notes in Computer Science, vol 10454. Springer, Cham.