When Robots Tell Each Other Stories: The Emergence of Artificial Fiction

When I wrote about story-telling robots nearly 7 years ago I had no idea how we could actually build robots that can tell each other stories. Now I believe I do, and my paper setting out how has just been published in a new volume called Narrating Complexity. You can find a pdf online here.

The book emerged from a hugely interesting series of workshops, led by Richard Walsh and Susan Stepney, which brought together several humanities disciplines including narratology, with complexity scientists, systems biologists and a roboticist (me). It was at one of those workshops that I realised that simulation-based internal models - the focus of much of my recent work - could form the basis for story-telling.

To recap: a simulation-based internal model is a computer simulation of a robot and its environment, including other robots, inside itself. Like animals all robots have a set of next possible actions, but unlike animals (and especially humans) robots have only a small repertoire of actions. With an internal model a robot can predict what might happen (in its immediate future) for each of those next possible actions. I call this model a consequence engine because it gives the robot a powerful way of predicting the consequences of its actions, for both itself and other robots.

So, how can we use the consequence engine to make story-telling robots?

When the robot runs its consequence engine it is asking itself a 'what if' question; 'what if I turned left?' or, 'what if I just stand here?'. Some researchers have called a simulation-based internal model a 'functional imagination' and it's not a bad metaphor. Our robot 'imagines' what might happen in different circumstances. And when the robot has imagined something it has a kind of internal narrative: 'if I turn left I will likely crash into the wall'. In a way the robot is telling itself a story about something that might happen. In Dennett's conceptual Tower-of-Generate-and-Test the robot is a Popperian creature.

Now consider the possibility that the robot converts that internal narrative into speech, and literally speaks it out loud. With current speech synthesis technology that should be relatively easy to do. Here is a diagram showing this.

The blue box on the left is a simplified version of the consequence engine; it's the cognitive machinery that allows the robot to predict the consequences of a particular action. For an outline of how it works there's a description in the paper.

Another robot (B) is equipped with exactly the same cognitive machinery as robot A, and - as shown below robot B listens to robot A's 'story' (using speech recognition), interprets that story as an action and a consequence, which it 'runs' in its consequence engine. In effect robot B 'imagines' robot A's story. It 'imagines' turning left and crashing into the wall - even though it might not be standing near a wall to its left.

The new idea here is that the listener robot (B) converts the story it has heard into a 'what if' question, then 'runs' it in its own consequence engine. In a sense A has invited B to imagine itself in A's shoes. Although compared with the stories we humans tell each other, A's story is trivial, it does I suggest have all the key elements. And of course A and B are not limited to fictional stories: A could - just as easily - recount something that has actually happened to it, like 'I turned right to avoid crashing into the wall'.

You may be wondering 'ok but where is the meaning? Surely B cannot really understand A's simple stories..?' Here I am going to stick my neck out and suggest that the process of re-imagining is what understanding is. Of course you and I can imagine a vast range of things, including situations that no human has ever (or perhaps could ever) experience; Roy Batty's famous line "I've seen things you people wouldn't believe. Attack ships on fire off the shoulder of Orion..." comes to mind.

In contrast our robots have a profoundly limited imagination; their world (both real and imagined) contains only the objects and hazards of their immediate environment and they are capable only of imagining next possible actions and the immediate consequences of those actions. And that limited imagination does have the simple physics of collisions built in (providing the robot with a kind of common sense). But I contend that - within the constraints of that very limited imagination - our robots can properly be said to 'understand' each other.

But perhaps I'm getting ahead of myself, given that we haven't actually run the experiments yet.

Swarm robotics at the Science Museum

Just spent an awesomely busy weekend at the Science Museum, demonstrating Swarm Robotics. We were here as part of the Robotville exhibition, and - on the wider stage - European Robotics Week. I say we because it was a team effort, led by my PhD student Paul O'Dowd who heroically manned the exhibit all four days, and supported also by postdoc Dr Wenguo Liu. Here is a gallery of pictures from Robotville on the science museum blog, and some more pictures here (photos by Patu Tifinger):

Although exhausting, it was at the same time uplifting. We had a crowd of very interested families and children the whole time - in fact the organisers tell me that Robotville had just short of 8000 visitors over the 4 days of the exhibition. What was really nice was that the whole exhibition was hands-on, and our sturdy e-puck robots - at pretty much eye-level for 5-year olds, attracted lots of small hands interacting with the swarm. A bit like putting your hand into an ants nest (although I doubt the kids would have been so keen on that.)

Let me explain what the robots were doing. Paul had programmed two different demonstrations, one with fixed behaviours and the other with learning.

For the fixed behaviour demo the e-puck robots were programmed with the following low-level behaviours:

1. Short-range avoidance. If a robot gets too close to another robot or an obstacle then it turns away to avoid it.
2. Longer-range attraction. If a robot can sense other robots nearby but gets too far from the flock, then it turns back toward the flock. And while in a flock, move slowly.
3. If a robot loses the flock then it speeds up and wanders at random in an effort to regain the flock (i.e. another robot).
4. While in a flock, each robot will communicate (via infra-red) its estimate of the position of an external light source to nearby robots in the flock. While communicating the robot flashes its green body LED.
5. Also while in a flock, each robot will turn toward the 'consensus' direction of the external light source.

The net effect of these low-level behaviours is that the robots will both stay together as a swarm (or flock), and over time, move as a swarm toward the external light source. Both of these swarm-level behaviours are emergent because they result from the low-level robot-robot and robot-environment interactions. While the flocking behaviour is evident in just a few minutes, the overall swarm movement toward the external light source is less obvious. In reality even the flocking behaviour appears chaotic, with robots losing each other, and leaving the flock, or several mini-flocks forming. The reason for this is that all of the low-level behaviours make use of the e-puck robots' multi-purpose Infra-red sensors, and the environment is noisy; in other words because we don't have carefully controlled lighting there is lots of ambient IR light constantly confusing the robots.

The learning demo is a little more complex and makes use of an embedded evolutionary algorithm, actually running within the e-puck robots, so that - over time - the robots learn how to flock. This demo is based on Paul's experimental work, which I described in some detail in an earlier blog post, so I won't go into detail here. It's the robots with the yellow hats in the lower picture above. What's interesting to observe is that initially, the robots are hopeless - constantly crashing into each other or the arena walls, but noticeably over 30 minutes or so we can see the robots learn to control themselves, using information from their sensors. The weird thing here is that, every minute or so, each robot's control software is replaced by a great-great-grand child of itself. The robot's body is not evolving, but invisibly its controller is evolving, so that later generations of controller are more capable.

The magical moment of the two days was when one young lad - maybe 12 years old, who very clearly understood everything straight away and seemed to intuit things I hadn't explained - stayed nearly an hour explaining and demonstrating to other children. Priceless.

Encyclopaedias and Emergence in Warwick

Here at the University of Warwick this week for the European Conference on Complex Systems.

Unexpectedly, Springer also used the conference to launch their excellent new Encyclopaedia of Complexity and Systems Science. As an author of one of the articles in the encyclopaedia - on Foraging Robots - it was great to see all 11 volumes and my article, in print, for the first time. Editor in chief Bob Meyers did the formal launch last night and (perhaps not suprisingly) there were four or five contributors here in Warwick. Bob called a couple of us out of the audience to say a few words, which was great. I made the points that complexity science is a wonderful unifier of multiple disciplines - everything from cell biology to economics - and that the grand challenge is to find unifying principles of emergence and self-organisation.

Today is the EmergeNET3 workshop. James Crutchfield gave a terrific invited talk about emergence, which he defines as a change in a system's causal architecture. He outlined his computational mechanics framework for analysing emerging patterns, and gave examples from cellular automata. Very interesting, but I was left wondering if Jim's framework would transfer well from the ideal grid-world of cellular automata, to the continuous time and space of swarm robotics, with rather more complex agent behaviours, real-world physics and noise. I suspect not.

Artificial Culture in Prague

I'm here at the brilliant European Union conference Science beyond Fiction, and yesterday gave my talk in the session on Collective Robotics: adaptivity, co-evolution, robot societies. I was pretty nervous because (a) this is my first talk on the Artificial Culture project to a international audience of senior researchers and (b) the project is still in its early development stages so we don't yet have any results. However, I'm pleased to say the talk went down well and I had some great questions - followed by conversations late into the evening.

Here is a movie of my presentation slides:


One of the questions was about robot imitation: are the robots learning to imitate, or have we pre-programmed them with imitation? My answer was that we have hand-coded imitation, in other words, our robots are endowed with an imitation instinct. You have to start somewhere, I argued, and this seems a good place to start and will initially allow us to study meme-evolution in our robot society in isolation from robot adaptation. While my questioners agreed, they also suggested that the evolution of imitation would also be really interesting, and encouraged us to - in effect - turn the evolutionary clock a little further back in our robot model of the emergence of culture.

Here are all of my blog posts on this project so far.

Emergence in Glasgow

Just returned from the excellent 2nd EmergeNet meeting, in Glasgow. EmergeNet is an EPSRC funded network of projects and people linked by an interest in emergence. As the Wikipedia article states, the phenomenon of emergence has been known about for a long time, but it still defies a proper scientific definition. In other words a definition that allows you to look at some complex phenomena and say yes, this is true emergence, but that isn't, and to measure the strength of the emergence (if indeed that is possible).

The reason a rigourous definition of emergence is important is that we can now contemplate designing complex systems that exploit emergence. A swarm robotics system is, for instance, a designed system which relies on emergence but - within the framework of complexity science - many other systems, from molecular to economic, would benefit from a deep understanding of emergence.

There were some truly excellent talks at EmergeNet2 - I'll add a link here when the presentations are online. But from one of those talks here is a link to an astonishing YouTube video from EmergeNet leader Lee Cronin and his team, showing (if I understand it correctly) controlled inorganic crystalline growth of molecular tubes - which looks remarkably organic.