Bridging human and system perception

My last post opened up the topic of how humans interact with sensing technologies, particularly systems that sense their bodily responses to playful experiences. Today, I’d like to develop this theme some more, but this time focusing on a quite different kind of sensing technology, while also broadening the discussion to bring in a new perspective, that of the ‘designer’ whose task it is to bridge between the technology and its users.

First some broad background. Paul Dourish has made a compelling case that our experience of computers is becoming evermore embodied, that is more physical and material.  Interacting with new kinds sensing system would seem to be an integral part of this – from systems like the Kinect that track our movements and gestures, to those that track how we move physical objects across surfaces in tangible interfaces.

A tricky challenge with invisible sensing systems is knowing precisely what they are doing, or even how to interact with them in the first place. PARC’s Victoria Bellotti has framed this problem in terms of five key questions for the designers of invisible sensing systems.

  • How do I address one (or more) of many possible devices?
  • How do I know the system is ready and attending to my actions?
  • How do I effect a meaningful action?
  • How do I know the system is doing (has done) the right thing?
  • How do I avoid mistakes?

Over the past year I have been working on an unusual project to design and deploy a new sensing system called Aestheticodes with colleagues from the Horizon institute.  This is somewhat like the increasingly familiar QRcodes that festoon the world around us in that you point a camera (such as the one on your mobile phone) at a visual pattern (that contains a computer-readable code hidden within it) in order to trigger an interaction (such as the display of some digital media). However, it is based on a distinctive and very interesting that I first encountered  in a CHI 2009 paper by Enrico Costanza.

The basic idea is to embed a digital code into the topology of the image, by which I mean representing it through the hierarchical structure of a drawn image – specifically the number and depth of nested regions within regions. You will find the rules set out in the original paper, but as a quick example, the following image show a pattern with the simple code embedded within it four times. The code is represented by the numbers of solid blobs that are contained in a set of white regions which must be joined together into a single connected shape (the numbers are then simply written in ascending order). This simple code is clearly repeated four times within the image.

aestheticodes-1 aesthetocides-2

This idea excited me for two main reasons. First, the rules for making visual codes are expressed through a simple set of drawing rules that a human can quickly learn. This turns the generation of visual codes into a creative act that builds on people’s natural drawing skills. Second is the consequence that two very different looking images can represent the same code, as it is the topological structure of the image that matters, not the actual shapes that are drawn. For example, the second image here also contains numbers of blobs within connected white regions, but is drawn using different shapes and in a different style.

On the other hand, two very similar images can in fact represent quite different codes sue to subtle variations that the viewer doesn’t pick up on, for example a few extra small blobs here and there. This makes the overall approach very open to playfulness and creativity as we recently found out through a collaboration with Tony Quinn and Emily-Clare Thorne, two ceramic designers from Central St Martins college. We reported the full details at CHI this year, but in a nutshell, it took us about a day to train a group of ceramic designers to be able to draw these kinds of patterns, after which we commissioned them to each produce a set set of three designs to appear on plates, placemats and menus at the Busabi Eathai restaurant.

You can see their final designs online, but I include a couple of photos taken at an exhibition that was staged at Busaba during London Design Week last year.

2012-09-20 18.52.49

What  fascinated to me was how our designers were able to hide specific visual codes within rich and complex patterns, and especially the various strategies they employed for achieving this. One was to add embellishments that a human would see as part of the pattern, but are in fact not connected to the parts that contain the code. Another was to introduce large solid regions that look significant to a human, but actually are no different from a regular line as far as the topology of the drawing is concerned. A third  was to use different colours, that again are ignored by the ‘system’ that thresholds an image to black and white before processing it.

2012-09-20 18.52.55

On the other hand, designers also needed to be aware of  how the technology processes images in order to be able to produce patterns that would work on ceramics under testing real world conditions (especially tricky lighting as you might notice from the nasty specular reflections int he photos). Not only did they have to appreciate the basic drawing rules, but also had to understand how, for example,  lines and small gaps could become problematic when images were digitised and became pixelated.

In short – and here is where we can tie it back to Victoria Bellotti’s questions – in order to make rich decorative patterns that contained reliable codes our designers had to span two worlds. On the one hand, they had to understand how humans view patterns, carefully separating figure from ground and employing basic principles from Gestalt Psychology such as closure and similarity to get them to read a coherent pattern from possibly disconnected parts. On the other they had to be able to reason about how the technology processes the patterns – which is quite different – thresholding them and then searching for nested topological structures. By understanding both worlds, the designers  could bridge between them. Importantly could also exploit the fact that they don’t always overlap. There are some things that people see as significant but that the system does not (various embellishments) and others that are significant to the system, but not to people.

It is these differences between human and system perception of the patterns – that they only partially overlap – that provides the creative wiggle room for hiding codes within beautiful patterns. I’d emphasise two final points here in what has become quite a long post.

  • It is the designers who, at least in part, are dealing with Victoria’s questions, essentially mediating between the system and the end-users (although there are still challenges for these too as we discuss in our paper).
  • This is perhaps only possible because the operation of the sensing system, in this case an image processing system, is revealed to the designers through a relatively simple set of drawing rules that fit well with their natural skills and training.

So my final question is whether other – or perhaps even all – invisible sensing technologies  should be based on transparent rules that can be revealed to ‘designers’ so that they can both make them work reliably, but also mess around with them creatively? Conversely, might technologies that operate as if by magic not be amenable to such creative use? Or adding a further layer of subtlety, must human designers appreciate how the system works in order to craft the magic for everyday users?

It’s an intriguing question – and one that we are currently exploring further as we extend the approach to enable us to embed recognisable codes into fabrics – especially lace.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s