When Robots Tell Each Other Stories: The Emergence of Artificial Fiction

When I wrote about story-telling robots nearly 7 years ago I had no idea how we could actually build robots that can tell each other stories. Now I believe I do, and my paper setting out how has just been published in a new volume called Narrating Complexity. You can find a pdf online here.

The book emerged from a hugely interesting series of workshops, led by Richard Walsh and Susan Stepney, which brought together several humanities disciplines including narratology, with complexity scientists, systems biologists and a roboticist (me). It was at one of those workshops that I realised that simulation-based internal models - the focus of much of my recent work - could form the basis for story-telling.

To recap: a simulation-based internal model is a computer simulation of a robot and its environment, including other robots, inside itself. Like animals all robots have a set of next possible actions, but unlike animals (and especially humans) robots have only a small repertoire of actions. With an internal model a robot can predict what might happen (in its immediate future) for each of those next possible actions. I call this model a consequence engine because it gives the robot a powerful way of predicting the consequences of its actions, for both itself and other robots.

So, how can we use the consequence engine to make story-telling robots?

When the robot runs its consequence engine it is asking itself a 'what if' question; 'what if I turned left?' or, 'what if I just stand here?'. Some researchers have called a simulation-based internal model a 'functional imagination' and it's not a bad metaphor. Our robot 'imagines' what might happen in different circumstances. And when the robot has imagined something it has a kind of internal narrative: 'if I turn left I will likely crash into the wall'. In a way the robot is telling itself a story about something that might happen. In Dennett's conceptual Tower-of-Generate-and-Test the robot is a Popperian creature.

Now consider the possibility that the robot converts that internal narrative into speech, and literally speaks it out loud. With current speech synthesis technology that should be relatively easy to do. Here is a diagram showing this.

The blue box on the left is a simplified version of the consequence engine; it's the cognitive machinery that allows the robot to predict the consequences of a particular action. For an outline of how it works there's a description in the paper.

Another robot (B) is equipped with exactly the same cognitive machinery as robot A, and - as shown below robot B listens to robot A's 'story' (using speech recognition), interprets that story as an action and a consequence, which it 'runs' in its consequence engine. In effect robot B 'imagines' robot A's story. It 'imagines' turning left and crashing into the wall - even though it might not be standing near a wall to its left.

The new idea here is that the listener robot (B) converts the story it has heard into a 'what if' question, then 'runs' it in its own consequence engine. In a sense A has invited B to imagine itself in A's shoes. Although compared with the stories we humans tell each other, A's story is trivial, it does I suggest have all the key elements. And of course A and B are not limited to fictional stories: A could - just as easily - recount something that has actually happened to it, like 'I turned right to avoid crashing into the wall'.

You may be wondering 'ok but where is the meaning? Surely B cannot really understand A's simple stories..?' Here I am going to stick my neck out and suggest that the process of re-imagining is what understanding is. Of course you and I can imagine a vast range of things, including situations that no human has ever (or perhaps could ever) experience; Roy Batty's famous line "I've seen things you people wouldn't believe. Attack ships on fire off the shoulder of Orion..." comes to mind.

In contrast our robots have a profoundly limited imagination; their world (both real and imagined) contains only the objects and hazards of their immediate environment and they are capable only of imagining next possible actions and the immediate consequences of those actions. And that limited imagination does have the simple physics of collisions built in (providing the robot with a kind of common sense). But I contend that - within the constraints of that very limited imagination - our robots can properly be said to 'understand' each other.

But perhaps I'm getting ahead of myself, given that we haven't actually run the experiments yet.

Three stories about Robot Stories

Here are the slides I gave yesterday morning as member of panel Sci-Fi Dreams: How visions of the future are shaping the development of intelligent technology, at the Centre for the Future of Intelligence 2017 conference. I presented three short stories about robot stories.




Slide 2:
The FP7 TRUCE Project invited a number of scientists - mostly within the field of Artificial Life - to suggest ideas for short stories. Those stories were then sent to a panel of writers, who chose one of the stories. I submitted an idea called The feeling of what it is like to be a robot and was delighted when Lucy Caldwell contacted me. Following a visit to the lab Lucy drafted a beautiful story called The Familiar which - following some iteration - appeared in the collected volume Beta Life.

Slide 3:
More recently the EU Human Brain Project Foresight Lab brought three Sci Fi writers - Allen Ashley, Jule Owen and Stephen Oram - to visit the lab. Inspired by what they saw they then wrote three wonderful short stories, which were read at the 2016 Bristol Literature Festival. The readings were followed by a panel discussion which included myself and BRL colleagues Antonia Tzemanaki and Marta Palau Franco. The three stories are published in the volume Versions of the Future. Stephen Oram went on to publish a collection called Eating Robots.

Slide 4:
My first two stories were about people telling stories about robots. Now I turn to the possibility of robots themselves telling stories. Some years ago I speculated on the idea on the idea of robots telling each other stories (directly inspired by a conversation with Richard Gregory). That idea has now turned into a current project, with the aim of building an embodied computational model of storytelling. For a full description see this paper, currently in press.

Memories of Skyrim

From time to time I like to visit Skyrim. I've been going there for about 2 years - completed the main quest a year or so ago, and since then go back to undertake a side quest or, more often that not, just wander around admiring the scenery. Of course my character is now reasonably levelled-up so wandering around is not quite as perilous as it used to be; interesting rather than terrifying.

But the thing I've noticed in recent months is that I have memories of being in places in Skyrim that, subjectively, feel completely indistinguishable from memories of being in real places in the real world. In other words the quality and character of the memory - the sense of having really been there - is no different when I recall, say, standing on the porch of Dragonsreach Hall looking South toward Throat of the World mountain, as when I remember looking down toward the Cumberland basin from the south footpath of the Clifton suspension bridge.

For me this is a new experience. I've been playing (and sometimes coding) video games since that meant batting a pixelated ball from one side of a low-res monochrome screen to the other on home built 8-bit micros in the 1980s. I have fond memories of playing the first generation Alone in the Dark ~1993 on a 386 PC, with my 5 year old son - but my memories are of the experience of playing the game with Tom, not of actually being in that haunted house. More recent games on the Xbox 360, with graphics I would have had difficulty imagining 20 years ago, have not had the same effect of creating such compellingly real memories for me.

So what is it about Skyrim that is making these memories feel so real? I think there are several factors. The first is that the scenery is so breathtakingly beautiful, which means you really do want to just stop and stare for awhile. Second, and equally important I think, is that this is not some imagined alien landscape. It is decidedly Earth - a cold Northerly Earth certainly, but the fells and mountains, the lakes and forests, the grasses and especially trees, are realised so accurately you can identify whether it's a birch or an oak. Third, the landscape is in constant motion - so the grasses sway in the wind, the brooks gurgle and splash and insects flutter. Wait a little longer and you realise the day is passing from afternoon to dusk, the sky turns golden in the sunset, then to night. A star spangled and moon-crossed night sky then rewards the patient (and the brave - this is a wild and dangerous place), following by a glorious sunrise. And there is weather too. Rain, which splashes delightfully on the lake, occasional thunderstorms (learn the power and you can call them up!), and snow blizzarding in the mountains. In case you can't go there yourself watch this YouTube movie: Skyrim - landscapes and scenery.

All of these are I think important cues in making the experience, and hence the memory, feel so real. But I think there is another factor, which is that my journeying through Skyrim is part of a narrative which connects places with events, quests and discoveries. So the memorable places are those I have arrived at following some perilous and occasionally epic trail, with multiple trials on the journey, Or they are places I have discovered offer safety and refuge; places to return to after days questing in the wilderness. Perhaps the depth and intensity of the experienced narrative somehow makes up for the limitations of the sensory experience? With only 2D vision on a flat TV screen and stereo audio, and nothing at all to stimulate the rest of the considerable human sensorium, it seems incredible that such a weakly immersive experience - compared to being in real places - can create subjectively comparable memories.

Since thinking about these memories of Skyrim, I've wondered if these are technically false memories. Am I experiencing so called False Memory Syndrome? Although FSM is controversial what is beyond doubt is the extraordinary suggestibility and unreliability of human memory. I was astonished by the unreliability of memory I witnessed at first hand two years ago while on Jury service. But false memories are memories of events that never happened, but are strongly believed. On reflection I think my memories of Skyrim do not fall into this category. The events and places occurred in the virtual rather than the real, but they, and my experience of them, really happened.

So if immersive video games technology has reached the point that it can create, for the gamer, memories of places and events which feel no different to memories of places and events in the real world, is this a bad thing? And, as the technology improves to make the experience more immersive, involving more senses, will we find ourselves unable to distinguish between the virtual and the real - confusing memories of one with the other? I think the answer is no. After all we each create a personal narrative - the remembered story of our lives - and I don't think it matters whether the events that make up that story happen in the virtual or the real. I think we're just as able to recall the difference between a trip to Rhyll and Rome, as between Skegness and Skyrim. And if, with advancing years or just because that's the way we are, we start to confuse these memories I don't think we're any more likely to confuse the virtual and the real, as we are the real and the real. Nor do I think the degree of immersion in the virtual matters as far as memories of being there is concerned*. After all, being in the real world is a fully immersive experience. Even if, and when, we can climb into a full-body immersive gaming rig, like those of Ernest Cline's brilliant Ready Player One, we will still only have an experience equal to that of the real world. So why should those experiences be remembered any differently to those in the real?

Ok. I've persuaded myself there isn't a problem. Time for another trip to Skyrim.

---

*There is of course another quite different concern - to do with how much more addictive the experience will become.Will we neglect the real - and ourselves, like Larry Niven's wireheads?

When robots start telling each other stories...

About 6 years ago the late amazing Richard Gregory said to me, with a twinkle in his eye, "when your robots start telling each other stories, then you'll really be onto something". It was a remark with much deeper significance than I realised at the time.

Richard planted a seed that's been growing since. What I didn't fully appreciate then, but do now, is the profound importance of narrative. More than we perhaps imagine. Narrative is, I suspect, a fundamental property of both human societies and individual human beings. It may even be a universal property of all advanced societies of sentient social beings. Let me try and justify this outlandish claim. First, take human societies. We humans love to tell each other stories. Whether our stories are epic poems, love songs; stories told with sound (music), or movement (dance), or with stuff (sculpture or art). Stories about what we did today, or on our holidays, stories made with images (photos, or movies); true stories or fantasies, or stories about the Universe that strive to be true (science), or very formal abstract stories told with mathematics, stories are everywhere. Arguably human culture is mostly stories.

Since humans started remembering stories and passing them on orally, and more recently with writing, we have had history: the more-or-less-true grand stories of human civilisation. Even the many artefacts of our civilisation are kinds of stories. They are embodied stories, which narrate the process by which they were designed and made; the plans and drawings which we use to formally record those designs are literally stories which tell how to arrange and join materials in space to fashion the artefact. Project plans are narratives of a different kind: they tell the story of the future steps that must be taken to achieve a goal. Computer programs are stories too. Except that they contain multiple narratives (bifurcated with branches and reiterated with loops), whose paths are determined by input data, which are related over and over at blinding speed within the computer. 

Now consider individual humans. There is a persuasive view in psychology that each of us owes our identity, our sense of self, to our personal life stories. The physical stuff that makes us, the cells of our body, are regenerated and replaced continuously, so that there's very little of you that existed 5 years ago. (I just realised the fillings in my teeth are probably the oldest part of me!) Yet you are still you. You feel like the same you 10, 20 or in my case 50 years ago - since I first became self-aware. I think that it's the lived and remembered personal narrative of our lives that provides us with the feeling, the illusion if you like, of a persistent self. This is I think why degenerative brain diseases are so terrifying. They appear to eat away that personal narrative so devastatingly that the person is ultimately lost, even while their physical body continues living.

So I was tremendously excited to be invited to a cross-disciplinary workshop on Narrative and Complex Systems at the York Centre for Complex Systems Analysis a couple of weeks ago, co-organised by York Professors of English Richard Walsh, and Computer Science Susan Stepney. For the first time I found myself in a forum in which I could share and debate ideas on narrative.

In preparing for the workshop I realised that perhaps the idea of robots telling each other stories isn't as far fetched as it first appears. Think about a simple robot, like the e-puck. What does the story of its life consist of? Well, it is the complete history of all of the movements, including turns, etc, punctuated by interactions with its environment. Because the robot and its set of behaviours is simple, then those interactions are pretty simple too. It occurred to me that it is perfectly possible for a robot to remember everything that has ever happened to it. Now place a number of these robots together, in a simple 'society' of robots, and provide them with the mechanism to exchange 'life stories' (or more likely, fragments of life stories). This mechanism is something we already developed in the Artificial Culture project - it is social learning by imitation. These robots would be telling each other stories.

But, I hear you ask, would these stories have any meaning? Well, to start with I think we must abandon the notion that they would necessarily mean anything to us humans. After all, these are robots telling each other stories. Ok, so would the stories mean anything to the robots themselves, especially robots with limited 'cognition'? Now we are in the interesting territory of semiotics, or - to be more accurate - robosemiotics. What, for instance, would one robot's story signify to another? That signification would I think be the meaning. But I think to go any further we would need to do the robot experiment I have outlined here.

And what would be the point of my proposed robot experiment? It is, I suggest, this:

to explore, with an abstract but embodied model, the relationship between the narrative self and shared narrative, i.e. culture.

By doing this experiment would we be, as Richard Gregory suggested, really onto something?