Interacting Under Canvas

The joy of being under canvas

I’m just back from a short camping trip and reflecting on how exciting it is to live under canvas. There is a visceral thrill to being in a tent as the thin fabric leaks noise, light, heat and shadows. Laying awake in the dark you become aware of nearby voices, the sound of rain pattering on the roof, the tent shaking in the breeze. It’s the perfect time to imagine a world of stories that might be happening outside.

Festival camping in Britain (there’s rain on the way!)

The Storytent

This fascination with tents has inspired several projects at the Mixed Reality Lab. The Storytent aimed to create an intimate and exciting interactive storytelling environment for children. We folded a projection screen into the shape of an A-frame tent and projected synchronized graphics onto both sides, as well as sound, to create a mini-immersive environment. We experimented with various ways of interacting with the tent: RFID readers placed at its ends recognized the comings and goings of (tagged) occupants and their possessions; a touch-screen map transported the tent to new locations in the virtual world; while shining flashlights onto the tent manipulated virtual objects as shown in this short video. This final technique used bespoke computer vision software for identifying and tracking flashlight beams.

Using flashlights to interact in the Storytent

We deployed the Storytent at Nottingham Castle, the ancient site of many thrilling and gruesome stories: Richard the Lionheart arrived there to confront King John after crusading (but he spoke French so the locals wouldn’t let him in); Richard III rode out from there to his death at Bosworth Field (they’ve recently dug him up again in nearby Leicester); Charles I raised his standard there to declare the English Civil War and summon an army (but no-one turned up so he went to Oxford instead); the locals burned down the Castle during the Corn Law riots (you may be getting a sense of Nottingham’s attitude to authority by now) and of course, Robin Hood got up to all sorts of adventures there (and yes he absolutely did exist).

Historical digressions aside, what better place to experience these stories – which children did by exploring the castle grounds and filling in paper clues (which were tagged with RFID) before taking them into the Storytent and using them to trigger the replay of stories.


If the Storytent strives for excitement, then our second interactive tent aims for the opposite: meditative relaxation. ExoBuilding  has been created by Holger Schnädelbach, Alex Irune, Dave Kirk, Kevin Glover and Patrick Brundell as an early prototype of a built structure that reacts physically to its occupants’ activities – an idea that they call ‘adaptive architecture’.

Exobuilding takes the form of a tent that flexes and moves in direct response to an occupant’s respiration  while also sonifying their heartbeat. Early experiments  revealed that this form of biofeedback triggers changes in participants’ physiology, leading to lower respiration rates and higher respiration amplitudes, respiration to heart rate coherence and lower frequency heart rate variability, and causing some people to report feeling more relaxed. The team suggests that there is potential for use as a biofeedback device.

Exobuilding flexes in response to breathing

Tents as interfaces

These tents have some distinctive and unusual properties when considered as computer interfaces.

They surround and enclose their occupants to form a form an immersive display, similar in principle to a virtual reality CAVE, but very different in practice due to their personal scale. Of course, it can be tricky to read high-resolution graphics on a screen that is so close to your eyes, or to move, gesture and look around in order to interact when sitting in a cramped tent. On the other hand, the snugness of the tent lends a sense of intimacy to storytelling, while isolation may help with relaxation. Also, not being able to turn or look around quickly can bring suspense to storytelling, emphasising the feeling that something may be approaching from behind.

Unlike larger immersive displays, tents have both insides and outsides. This allows for a separation between those who are immersed and interacting inside, for example a child engaging with a story, and a wider audience that remains outside the tent but can still see what is happening, for example parents or perhaps other children who are waiting for a turn. A tent is therefore an example of a spectator interface; an interface that is deliberately designed to reveal some, although not all, aspects of interaction to an audience.

As with the fabric of a regular tent, the screens of our interactive tents are porous membranes, leaking sound and light in both directions. This opens up opportunities for playful interactions between those inside and those outside such as whispering, casting shadows and even shaking the tent (great fun for parents!).

Finally, as the Exobuilding shows, they can flex and move, changing shape and form under computer control, introducing a sense of physicality and even synchronising with an occupant’s physiological responses.

Interfaces as tents?

In turn, these interactive tents illustrate a wider interaction design principle. Might it be useful to think of all interfaces as being ‘tents’, by which I mean as permeable boundaries that connect different worlds or spaces. My colleague Boriana Koleva first explored this idea during her PhD research and referred to such interfaces as ‘mixed reality boundaries’.

So the Storytent is a permeable boundary that connects three spaces, the physical spaces of inside and outside, and a virtual world. It both separates these spaces, providing a degree of isolation and intimacy, but also allows some information and interactions to flow between them, even including participants who can traverse the boundary, for example entering tent from the outside or entering the virtual world.

Is it useful to conceive of other interfaces as being tent-like permeable boundaries? Might games consoles be designed around the idea that players peer into or enter a virtual world while others look back out at them (especially now they routinely come equipped with cameras such as the Kinect)? Should communication tools such as Skype be reimagined as boundaries between physical spaces that might afford different kinds of isolation and permeability? Might even the familiar web browser be reconceived a permeable boundary to the Web, one through which we look while others look back out at us, tracking our movements and actions?

There may be something to be said for seeing these interfaces as permeable membranes, encouraging us to reflect on how information, interaction and presence flow both in both directions and how we might want to design this to enable or restrict such flows, address audiences as well as users, or simply to lend them some tent-like excitement. Perhaps we are all interacting under canvas after all?


Banjos and Discrete Technologies

A confession

I begin this post with a confession. I play the banjo. There, it’s out in the open and I feel better already. Actually, I play the tenor banjo in Irish style, although this is a distinction that probably only banjo players care about (but boy will they care). You’ll often find me on a Sunday afternoon in the Vat and Fiddle, Bell or Hop Pole playing along at one of Nottingham’s traditional Irish sessions.

You would be forgiven for wondering what this has to do with human-computer interaction, but it turns out that even very traditional practices can shed new light on our interactions with computers. My colleagues Peter Tolmie, Yousif Ahmed  and myself undertook an ethnographic study of Irish music sessions which, fortunately for me, involved spending some considerable time hanging around in traditional sessions and observing what goes on as well as interviewing musicians.

Irish music sessions

At first glance an Irish music session is a traditional practice that seems far removed from the world of computers. A group of musicians sits around in a pub, playing traditional tunes on a variety of instruments – fiddles, flutes, whistles, mandolins, the bodhran (the traditional Irish drum) and guitars to name a few. Even banjos are tolerated. Well, subject to recital of the canonical list of banjo jokes anyway.

Irish Session at the Bell Inn Nottingham
Irish Session at the Bell Inn Nottingham

The structure of the music allows the musicians to improvise as a group. They typically play sets of several tunes that are strung together, with each tune being repeated several times. A typical set might consist of three different tunes, each repeated three teams, and where each tune consists of two parts that are themselves repeated. If this seems a little complicated, the important thing to remember is that several tunes are sequenced together and that there is a fair bit of repetition.  This structure enables improvisation, both through choosing which tune to segue into next as well as through embellishing a tune each time it is repeated. Repetition also supports playing by ear as musicians can try to recall a tune that they haven’t played for a while or even pick up a new tune from scratch if they are especially skilled.

How Irish musicians have taken to the Internet

Our study showed how Irish musicians have taken to the Internet, establishing a dedicated Irish social media site called The Session as far back as 2002. Since then, a worldwide community of musicians has been transcribing the Irish repertoire, building a database of thousands of tunes along with sheet music, notes on recordings, names (which are widely contested), variations, playing tips and suggestions for good sets. The picture shows an excerpt from the page for the tune Banish Misfortune giving alternative names, comments and also the music written our in ABC notation which has been specially developed for traditional music (the conventional ‘dots’ are also available). The Session also gives the locations and timings of sessions in many cities around the world in case an Irish musician is visiting a distant town and fancies a play. The Session is incredibly useful because it enables musicians to learn new tunes. It has since been supplemented by a variety of other services from YouTube, to the BBC’s Virtual Session  to the specialist learning sites such as jigsandreels.

Page for the tune Banish Misfortune from
Page for the tune Banish Misfortune from

Session etiquette

There is a very interesting tension at play here – one that speaks directly to the design of new technologies. On the one hand, Irish musicians appear to be enthusiastically adopting digital media to establish a common repertoire of tunes, while on the other the actual performance of these tunes in a live session is governed by a strong etiquette that emphasizes the importance of playing by ear. While there are of course huge local variations in etiquette, many musicians we spoke to were very aware of the idea of this tradition of playing by ear. There is a general nervousness about getting out sheet music in a session, especially for beginners, an idea that is reinforced by various published guides to session etiquette.

Our studies revealed the subtle ways in which musicians manage this tension, walking the line between extensive preparation and rehearsal away from a session, and the spontaneity of playing by ear within it. Many, for example, carry notebooks of tunes, pre-arranged into convenient sets. One notable strategy that we saw – and that I have seen several times since in sessions as far afield as Nottingham and Seattle – is to prepare a small piece of paper, perhaps in a pocketbook, that has the names of tunes grouped into sets alongside just the first few bars of music of each. These bespoke notations convey the essential information needed in a minimal form; an Irish musician needs to know which tune is coming next and just the first few bars of that tune so that they can segue into it, after which they are up and running from memory.

A discrete cribb sheet showing sets of tunes with the first few bards in ABC
A discrete cribb sheet showing sets of tunes with the first few bards in ABC

Discrete technologies

Such pieces of paper are an example of a discrete technology. They condense the essential information required to support the practice of playing by ear into a form that fits with session etiquette. A page of a small notebook, which after all could be used for writing down the names of new tunes, contact details and so forth, is a long way removed from a piece of sheet music on a music stand.

This idea of discrete technologies – ones that provide useful services in a way that respects the social etiquette of given situation – extends beyond pieces of paper to the digital world. We are all aware of the challenges of managing mobile phone use in various social settings. Indeed, these same technologies are present in Irish sessions that take place in pubs, busy social settings where people – including the musicians – have come together to chat, may even drink a pint or two, and enjoy ‘the craic’. In this context it may be quite acceptable to get out a phone, check a text or look up something on the Internet.  It can even be be acceptable to break of playing in the middle of a set to check one’s phone or receive a call – quite a contrast to a formal classical music concert. We also observed people using digital recorders to capture tunes as they are played.

It’s all in the ambiguity

So this notion of discrete technologies is a subtle one. Ironically, using a modern digital technology such as a mobile phone or digital recorder to support a traditional practice might possibly be more discrete that using a traditional technology such as a piece of paper. Both phones and pieces of paper can be used for many purposes, from taking notes to reading sheet music, and it is perhaps this apparent use that really matters. It all depends on how you are seen to use a given technology rather that on the form of the technology itself.

Perhaps the underlying issue is whether the apparent use of a technology is sufficiently ambiguous that it can plausibly fit some socially acceptable use, or whether in contrast, it overtly flaunts the local etiquette to the point where it can no longer be ignored. This idea directly builds on Bill Gaver’s discussions of the role of ambiguity in interface design, and especially on Paul Aoki’s  and Alison Woodruff’s subsequent application of this to the challenge of saving face in mobile phone calls.

The challenge for interface designers is to invent displays and services that can be used discretely with respect to a particular social setting; that don’t overtly flaunt social etiquette and that are open to ambiguous interpretation. Our study showed how traditional Irish musicians had been particularly creative in this respect, inventing their own discrete notation to help them bridge between their learning of tunes offline and their ability to recall them when playing live. Interface designers might learn a great deal from such inventiveness!

To read and hear more

For those who would like to read more, our ethnographic study of Irish Music sessions was first reported in a paper at CSCW 2012 while an extended version has recently been published in the book Ethnomethodology at Play.

Finally, it seems appropriate to end with one of my favourite banjo tunes – the rollicking jig Banish Misfortune that we saw earlier on. Feel free to play along, or perhaps I’ll see you in a session somewhere soon.