Physical Computing theme on Designing Interactive Artifacts F-2013 Proximity-based interaction

Physical Computing theme on Designing Interactive Artifacts F-2013 Proximity-based interaction Thomas Pederson <tped@itu.dk> Associate Professor Pervasive Interaction Technology Lab http://itu.dk/pit/ Interaction Design research group http://www.itu.dk/research/ixd/ 1

Agenda Reality-Based Interaction (Jacob et al., 2008) Proximity-based interaction Proxemics for Ubicomp (Greenberg et al., 2011) Situative Space Model (Pederson et al., 2012) virtual world physical world exercise proximity-based interaction using ultrasonic sensor 2

last week: mixed-reality perspective on physical computing Reality-Based Interaction physical world virtual world physical computing 4

last week: mixed-reality perspective on physical computing Reality-Based Interaction Augmented/Mixed Reality Ubiquitous/Pervasive Computing Graspable/Tangible User Interfaces Wearable Computing Context(ual) Aware Systems virtual world physical world 5

Reality-Based Interaction New interfaces are breaking the box Augmented/Mixed Reality Ubiquitous/Pervasive Computing Graspable/Tangible User Interfaces Wearable Computing command Context(ual) line Aware Systems WIMP virtual world physical world 6

Reality-Based Interaction Reality-Based Interaction (Jacob et al., 2008) a concept that ties together many new post-wimp interaction styles post-wimp interfaces "containing at least one interaction technique not dependent on classical 2D widgets such as menus and icons" (van Dam) the RBI framework helps us analyse these new interactive systems 7

Reality-Based Interaction Reality-Based Interaction (Jacob et al., 2008) 8

Reality-Based Interaction Reality-Based Interaction (Jacob et al., 2008) common sense knowledge about the physical world gravity, friction, velocity, the persistence of objects, relative scale RBI examples TUI constraints enforced by the size and shape of physical tokens illusion of mass in iphone menus ("rubber-band effect") 9

Reality-Based Interaction Reality-Based Interaction (Jacob et al., 2008) the understanding of our own body and it's state relative position of the limbs (proprioception), motion range, senses muscle coordination skills acquired early in life (limbs, head, eyes) RBI examples two-handed interaction full-body interaction 10

Reality-Based Interaction Reality-Based Interaction (Jacob et al., 2008) the understanding of spatial relationships among objects, near and far facilitate our sense of orientation and spatial understanding example clues: horizon, atmospheric color, fog, lighting, and shadow RBI examples Augmented/Mixed Reality systems make use of these visual clues Context-Aware systems base their behaviour on the physical context 11

Reality-Based Interaction Reality-Based Interaction (Jacob et al., 2008) People possess skills for social interaction and adapt to presence of others e.g. verbal and non-verbal communication RBI examples TUIs support co-located collaboration VR massively multiplayer online role-playing games (MMORPG) exploit social awareness and skills 12

Reality-Based Interaction implications for design basing a user interface on pre-existing real-world knowledge can... speed up learning decrease mental effort improve performance... but will not necessarily lead to a useful system: computer systems are powerful tools because they can help us do things beyond what can be done in the real world 13

Reality-Based Interaction RBI design tradeoffs suggested approach give up reality only explicitly and only in return for other desired qualities 14

Proxemics for Ubicomp (Greenberg et al., 2011) "When you walk up to your computer, does the screen saver stop and the working windows reveal themselves? Does it even know if you are there? How hard would it be to change this? Is it not ironic that, in this regard, a motion-sensing light switch is smarter than any of the switches in the computer...?" (Buxton, 1997) 15

Proxemics for Ubicomp devices are too blind Devices are blind to the presence of other computational devices non-computational aspects of the environment - people - non-digital objects - the room's fixed and semi-fixed features 16

Proxemics for Ubicomp devices are too blind design proposal "Just as people expect increasing engagement and intimacy as they approach others, so should they naturally expect increasing connectivity and interaction possibilities as they bring their devices in close proximity to one another and to other things in the ecology." 17

Proxemics for Ubicomp five dimensions of proxemics 18

five dimensions Proxemics for Ubicomp of proxemics distance is continuous, but can be interpreted in discrete steps 19

five dimensions Proxemics for Ubicomp of proxemics distance is continuous, but can be interpreted in discrete steps, e.g. "personal reaction bubbles" (Hall, 1966) 20

five dimensions Proxemics for Ubicomp of proxemics distance is continuous, but can be interpreted in discrete steps, e.g. "personal reaction bubbles" (Hall, 1966) "interaction zones" (Vogel & Balakrishnan, 2004) 21

five dimensions Proxemics for Ubicomp of proxemics continuous pitch, yaw, roll angles used in - gesture-based interfaces - tangible user interfaces discrete used in for instance attentive UIs - looking at the screen - looking nearby the screen - not looking at the screen 22

five dimensions Proxemics for Ubicomp of proxemics captures the distance and orientation of an entity over time speed of motion moving and turning toward versus away from another entity used in many kinds of post-wimp UIs full-body UIs (Nintendo Wii, Microsoft Kinect) gesture-based UIs tangible UIs 23

five dimensions Proxemics for Ubicomp of proxemics uniquely describes the entity different levels of detail exact identity and attributes (e.g. Thomas Pederson, the prof at ITU) only object type (a prof at ITU) a person 24

five dimensions Proxemics for Ubicomp of proxemics the physical context in which the entities reside e.g. a specific room and its characteristics 25

example system: Proxemics for Ubicomp the ViconFace The Proximity Toolkit and ViconFace: The Video http://www.youtube.com/watch?v=bht0qgtsddm 26

example system: Proxemics for Ubicomp the proxemic media player Proxemic Interaction: Designing for a Proximity and Orientation-Aware Environment http://www.youtube.com/watch? v=ohm9tevone8 27

exercise: apply RBI to the proximitybased media player Proxemics for Ubicomp how much of the interaction is something we know from the real world, how much is new? what RBI design tradeoffs have the designers made? 28

Situative Space Model (Pederson et al., 2012) 29

Situative Space Model used for modeling interaction situations in the emerging egocentric interaction paradigm: users are transformed into agents that are active in a mixed-reality environment also the classical concepts of (system) input and output become transformed into (human agent) action and perception, respectively - object manipulation and perception are processes that can involve any modality: tactile, visual, aural, etc. - virtual objects are manipulated and observed through "mediators" 30

Situative Space Model intended to capture what an agent can perceive and not perceive, reach and not reach, at any moment in time uses a number of spaces and sets for this purpose 31

Situative Space Model Spaces World Space encompasses all physical and virtual objects that are part of a specific model 32

Situative Space Model Spaces World Space encompasses all physical and virtual objects that are part of a specific model Perception Space the part of the space around the agent that can be perceived by the agent - varies continuously with agent movements - different senses have differently shaped Perception Spaces 33

Situative Space Model Sets Recognizable Set set of objects in Perception Space currently within their recognition distances 35

Situative Space Model Sets Recognizable Set set of objects in Perception Space currently within their recognition distances Examinable Set set of objects in Perception Space currently within their examination distances Selected Set set of objects in Action Space currently being physically or virtually handled 37

Situative Space Model An illustration: having breakfast Thomas Pederson DDDA- F2013 Proximity- based interaction 39

Situative Space Model An illustration: having breakfast Thomas Pederson DDDA- F2013 Proximity- based interaction 40

The Egocentric Interaction Paradigm An application The easyadl home Prototyping wearable activity support for persons suffering from mild dementia 41

Situative Space Model Uses Activity recognition Providing data for a multimodal interaction manager 42

Situative Space Model The challenge of determining the spaces and sets in realtime 43

time to play with arduino and proximity! activities lecture 1: why and how computing is getting physical a historical perspective and some example areas of research - exercise: arduino basics lecture: 2 proximity-based interaction - exercise: proximity-based interaction lecture 3: materiality - exercise: physical displays virtual world physical world mandatory hand-in: reflective report on physical computing 44