DESIGNING *for humans

Taxonomy of Physical Interactions

In a previous post I discussed the (disappearing) distinction between designing interfaces that primarily support physical interaction, and those that support cognitive interaction.  But this is not a black and white differentiation, so I thought it would be useful to describe a continuum of physical interaction in user interfaces.

Note that I am focusing on willfully or intentionally controlled interfaces.  There are examples, arguably, of involuntary interfaces - for instance, an autopilot function that activates if a pilot becomes disabled - but I'll table those interesting cases for a later discussion.

Continuum of Physical Interfaces

A continuum of physical interaction would range at one end from non-movement to that which requires significant, complex body movement.  At the first extreme we could include theoretical telepathic control, where no visible physical interaction is evident.  As we move away from that end of the spectrum we have existing and emerging neuroergonomic interfaces that rely on measurement of electrical potentials, cerebral blood flow, and MRI imagery other as triggers for action. 

The next step towards physical interaction are devices that measure or track relatively small motor movements and translate them into interface actions.  For example, Emotiv System's forthcoming Epoc device can translate facial muscle movement into expressions for online avatars. Eye tracking systems, already commonplace in supporting the physically disabled, track eye movements in place of mouse/keyboard controllers.

From here we move to relatively simple, ubiquitous traditional physical interaction controllers - buttons, keyboards, knobs, switches, levers - the stuff of mechanical and electro-mechanical devices that designers have been working with for years.  These controllers are typically binary (on/off), or at least incremental (having multiple, discrete states).  Most existing touch screen interfaces, such as bank ATMs, would fall under this category.

We then go from discrete, to continuous controllers, enabling multiple actions and greater flexibility.  The computer mouse was a breakthrough for human-computer interaction in this context as it supports various types of interaction and interfacing from a single control device.  In fact, while keys and buttons are typically designed with a specific function in mind, the mouse provided the opportunity for new user interfaces to be created for defining its functions.  Gestural interfaces, from multi-touch screen to the Wii are also examples of this flexible, "open" physical interaction category.  These are the "new "interaction devices that are opening up new possibilities for interaction designers.

We might imagine a Minority Report based interface as the ultimate extreme at the far end of the physical interaction spectrum, but as pictured in the video above, it is only limited to gestural hand movements.  What about more complex bodily interactions combining other limbs, postural movement and line-of-sight?  This is still largely unexplored territory might be best understood by observing how we use our bodies in the most dynamic and complex ways.  Musicians, athletes and dancers may be a more valuable source for developing future physical interaction ideas than science fiction.

A Metric for Physical Interface Complexity

Note that the continuum I described above, while not by any means arbitrary, was not based on a well-defined metric that quantified greater or less physical complexity.  If we were to do so, degrees of freedom would be an appropriate place to start.  A degree of freedom can be defined as any independent direction in which movement is possible.  A human finger has four degrees of freedom, made up of the extension/flexion of the three joints, as well as side-to-side movement.  Combining the individual degrees of freedom of the four fingers, thumb and wrist gives the hand 26 degrees of freedom.

Hypothetically, we could apply this to the entire human body to specify the maximum level of complexity for any single physical interaction, or sequence of interactions.  The total degrees of freedom for a fully functioning human is 1380.  In theory, we could go back to any physical interaction and quantify the (minimum) amount of movement required to come up with relative complexity measures.  But it actually gets more challenging as complexity is more than just the sum of the degrees of freedom, and would depend on the particular combination of movements, etc.  In other words, it's an interesting idea, but requires a lot more thought to pursue practically.

Mapping Physical Interaction Inputs to Outputs

Another important consideration is the relationship between physical inputs and the associated outputs in a user interface system.  Current discussion of gestural interfaces is primarily focused on using physical interaction to control virtual objects - a way to make the digital world more tangible.  But physical interaction interfaces can also be used to control physical systems, and not just in the literal sense.

Intuitive Surgical's da Vinci surgical systemsrepresent the leading edge of commercialized physical interaction devices.  As depicted in the video, the systems "translate and filter" a surgeon's precision hand motions into physical motions of surgical robot manipulators.  This requires a two-way physical interaction where the user not only provides physical output, but receives haptic input such as resistance to force.  So it's actually a physical-to virtual-to physical loop.

A Starting Point for Defining Physical Interactions in User Interfaces

While this is just a preliminary discussion, there are threads towards developing a taxonomy of physical interaction types:

• There is the complexity of the physical movement, characterized by the number and type of degrees of freedom involved.

• The output of the physical interaction, resulting in either virtual actions, physical actions or a combination.
• The directionality of the interaction: either one way from user to system, or bi-directional between user and interface.

Physical Interaction & The Future of User Interface Design

We are on the cusp of a significant trend in product design and design methodology, where the longstanding divide between physical and cognitive modes of interaction will disappear.  What do I mean by this? 

Consider, in general terms the distinction between primarily physical human-artifact interactions, and those that are primarily cognitive in nature. 

Physical artifacts are those whose primary design intent is the mechanical transportation or transformation of matter. Think automobiles, appliances, tools, basically the main focus of industrial design in the last century. Such products are ideally designed around human anthropometric and ergonomic principles to maximize efficiency, effectiveness and safety. Often these artifacts require significant experience to develop proficiency or “muscle memory”, and those who achieve a high level of skill may be considered craftsmen, or even athletes.

Compare this to cognitive artifacts that essentially store and transmit information – books, radios, telephones, wristwatches, personal computers and so on.  While these artifacts all intrinsically require physical interaction, it is typically trivial in its nature - it requires relatively limited manual dexterity to turn a page, click a mouse, press a button. The majority of design focus is on the user interface and its visual display of information – the “heavy lifting” with these artifacts takes place in the head of the user, so design for the mind, not the body, takes precedence.

 

So we have largely been designing two, co-existing, but separate types of products: those that utilize the complex mechanics of the body to transform matter, and those that accommodate capabilities of the mind for processing information.  Products that overlap these two worlds, that is, which take advantage of complex physical interactions to drive information processing, are few and far between.  The abacus the telegraph, and perhaps texting, comes to mind.  Musical instruments play an interesting role in this context where technical sophisticated body control is applied to the creation of sound, but even this is for artistic or entertainment purposes, rather than concerning the analysis of information in the scientific or business sense.

But now, we are starting to see the emergence of products that hint at using the mechanics of the body to interact with complex information.   For example Jeff Han’s multi-touch, interactive data wall, and virtual reality simulations that allow scientists to “feel” the forces between molecules.  But even these interfaces are only scratching the surface of what is possible from a physical interaction point-of-view.  Getting to this point has taken a long time because it is challenging to track and quantify the multiple degrees of freedom of movement of the body.  But it is also the result of our divided design processes, where physical interactions and cognitive interactions have traditionally been designed by different people with different expertise, at different times.

Moreover, such divided design processes are themselves the result of 20^th century psychology that treats humans as information processing machines.  As a consequence we think of human activity as comprised of discrete, sequential steps of thinking and then acting – I see something, then I reach for it.  In other words, a built-in division between cognitive interaction and physical interaction.  As a result it’s easy to see why we divide the world the way we do.

But there are alternative perspectives on human behavior, in particular the ecological psychologyy of J.J. Gibson.   Gibson coined the term “affordances” which is (mis) used and abused by interaction designers today.  But affordances, the relationships between people (or other organisms) and artifacts, are just a part of a larger “perception-action” framework.  In this view, perceiving or sensing information is a physical behavior itself, not just a means to drive a subsequent physical action.  Likewise, physical behavior drives perception – the two are connected, not divided, resulting in a perception-action loop. 

The implications of this for product design are subtle, but important.  Taking into account the physical interactions someone takes to acquire information is a useful for understanding and determining what and how to display information.  For example, rather than designing products where all information is accessed from a single point, information may be distributed across locations, where the location itself is informative above and beyond the content.  We see this emerging with augmented reality applications, where the user’s particular activity, such as walking to a particular location in a city, provides location-specific information. The format and content may be driven by variables such as the person’s posture and gait (e.g. in a hurry or browsing), direction of approach, and of course their physical characteristics such as eye height and reach.

To get to the point where we can design systems to take advantage of complex physical  interactions will require an taking a new look at how the fields of kinesthetics,  anthropometrics and optical flow , relate to interface design.  These fields will be as important to designers as information architecture and form giving are today.

As a starting point, I've been exploring the role of ergonomics in contemporary interface design, to better relate the divided fields of physical and cognitive product design.

 

 

 

Review - Wrench in the System

Harold Hambrose's Wrench in the System is a highly readable and cogent perspective on the lack of effective design in business software, in the spirit of Alan Cooper's The Inmates are Running the Asylum.

Hambrose, founder of user interface consulting firm Electronic Ink, essentially lived through and helped lead the growth of user-centered design over the past 20 years, and much of the book is directly drawn from his professional experiences {Disclaimer: I formerly worked for Hambrose as Director of Human Factors at Electronic Ink].

Wrench in the System combines personal narrative, case studies and historical perspective to point out that the creation of business software does not parallel the successful design processes in other industries such as architecture and product design:

"In the race to deploy new information technology, businesses and software developers often have stumbled because they have ignored traditional methods of product development and have seriously miscalculated the risks of developing software without pausing to find out more about how this new technology will be used.  The information technology industry, still in its adolescence, has been in too much of a rush to stop and tie its shoelaces." 

The book is targeted at business decision makers - CIOs and CEOs, who have the power to demand usable software, but lack the awareness of why it matters.  The focus is more specifically on enterprise software - large scale, technology driven, million dollar software systems that are notoriously challenging to use and difficult to change.  Hambrose frequently picks on SAP, both explicitly and implicitly - the book even includes a mini-fable called "The Emperor's New Enterprise System".

The majority of Wrench is spent defining the problem with examples from healthcare, financial services, energy, etc.  A reader who is already well versed in user-centered design will likely find these examples familiar, and somewhat repetitive.  But where Hambrose excels is in articulating the value and impact of design to business, for example:

"It's not a lack of technical know-how or business expertise or even a shortage of time or money that accounts for software's failure to provide the information we need.  The source of most of the trouble is a fundamental conflict between human beings and their machines, a conflict that causes profound misunderstanding"

The book itself, with clear typography and substantive paper stock, exemplifies user-centered design, as well as the still evident value of paper over electronic books.  It should benefit those sleepy business decision makers who clearly still need it, but it's language and stories will also benefit user-centered design practicioners who can apply its lessons to their own internal and external design sales challenges.