Multimedia Interaction and Intelligent User Interfaces - Principles, Methods and Applications

Preface

Contents

Retrieving Human Actions Using Spatio-Temporal Features and Relevance Feedback

Introduction

Action Retrieval Scheme

Action Retrieval Framework

Spatio-Temporal Interest Point Detection

Feature Description

Codebook Formation and Action Video Representation

Similarity Matching Scheme

Action Retrieval on the KTH Dataset

Dataset Processing

Performance Evaluation

Summary for Experiments on the KTH Dataset

Realistic Action Retrieval in Movies

Challenges of This Task

Implementation

Result Demonstration

Discussion

Application

Conclusions

References

Computationally Efficient Clustering of Audio-Visual Meeting Data

Introduction

Background

Challenges in Meeting Analysis

Background on Speaker Diarization

Background on Audio-Visual Synchrony

Human Body Motions in Conversations

Approach

The Augmented MultiParty Interaction (AMI) Corpus

Audio Speaker Diarization

Traditional Offline Speaker Diarization

Feature Extraction

Speech/Nonspeech Detection

Speaker Segmentation and Clustering

Online Speaker Diarization

Unsupervised Bootstrapping of Speaker Models

Speaker Recognition

A Note on Model Order Selection

Summary of the Diarization Performance

Extracting Computationally Efficient Video Features

Estimating Personal Activity Levels in the Compressed Domain

Finding Personal Head and Hand Activity Levels

Estimating Speakers Using Video Only

Associating Speaker Clusters with Video Channels

Audio-Visual Clustering Results

Using Raw Visual Activity

Using Estimates of Speaking Activity from Video

Discussion

References

Cognitive-Aware Modality Allocation in Intelligent Multimodal Information Presentation

Introduction

Modality and Human Information Processing

Modality and Sensory Processing

Modality and Perception

Visual Attention

Auditory Attention

Cross-Modal Attention

Modality and Working Memory

Working Memory Theory

Dual Coding Theory

Relating the Two Theories

Experiment on Modality Effects in High-Load HCI

Presentation Material

Task and Procedure

Measurements

Hypotheses

Results on Performance, Cognitive Load and Stress

Performance

Cognitive Load and Stress

Discussion

Text vs. Image

Visual Aid vs. Auditory Aid

Verbal Aid vs. Nonverbal Aid

Additional Aid vs. No Aid

Low Load vs. High Load

A Modality Suitability Prediction Model

Conclusions

References

Natural Human-Computer Interaction

Introduction

From Ergonomics to Human-Computer Interaction

Multimodal Interfaces

Natural Human-Computer Interaction

Natural Interaction Systems

Human-Centered Design

Intuitive Interaction

Natural Language and Tangible User Interfaces

Sensing Human Behavior

Sensed Spaces and Sensors Categories

Optical Sensors and Computer Vision Technologies

Image Analysis Techniques

Tracking Techniques

Observing Human Activity

People Detection

People Tracking

Gaze Estimation

State of the Art

Interactive Tabletop

Tangible User Interface

Smart Room

Smart Room with Tangible Natural Interaction

TANGerINE Smart Room: a Case Study

TANGerINE Smart Cube

Manipulation State Awareness

Gesture Detection Algorithm

Bluetooth-Based Proximity Awareness

Computer Vision Applied to the TANGerINE Platform

Observing Human Activity in TANGerINE Smart Room

References

Gesture Control for Consumer Electronics

Introduction

Sensing Technologies

Haptics

Handhold Sensors

Vision

Ultrasound

Infrared Proximity Sensing

Vision-Based Gesture Recognition

Body Part Detection

Gesture Tracking

Gesture Recognition

Gesture Control: Products and Applications

GestureTek

Toshiba

Mgestyk

Fraunhofer

TVs or Displays

Gaming

Mobile Phones

Automobiles

Conclusions

References

Empirical Study of a Complete System for Real-Time Face Pose Estimation

Introduction

Problem Definition

Problem Statement

Pose Estimation Algorithm

3D Mesh

Texture Extraction

Automatic Initialization

Face and Feature Detection

Mesh Initialization

Tracking

Overview of Method

2D Feature Tracking

Adaptation Step

Matching Criterion

Detection on Mesh Texture

Reconstruction Error of Mesh Texture

Detection of Failed Tracking

Results

Stability Analysis for Static Images

Accuracy on Videos

Semi-Automatic Annotation

Performance of Different Search Strategies: Angular Error

Performance of Different Search Strategies: MSE

Performance of Different Search Strategies: Computation Time

Influence of Texture Representation

Influence of Training Size

Benefits of the Proposed System

Analysis of Typical Results

Examples of Tracking Failure

Conclusions

References

Evolution-based Virtual Content Insertion with Visually Virtual Interactions in Videos

Introduction

System Overview

Essential Ideas

System Overview

Video Content Analysis

Frame Profiling

Motion Estimation

Region Segmentation

ROI Estimation

Aural Saliency Analysis

Virtual Content Analysis

Virtual Content Characterization

Behavior Modeling

The Cell Phase

The Microbe Phase

The Creature Phase

Virtual Content Insertion

Animation Generation

Layer Composition

Experimental Results

Summary

References

Physical Activity Recognition with Mobile Phones: Challenges, Methods, and Applications

Introduction

Background of Physical Activity Recognition

Practical Challenges on Mobile Devices

Accelerometer Based Physical Activity Recognition Methods

Data Format

Accelerometer Sensor Calibration

Signal Projection

Data Collection

Feature Extraction and Selection

Classification Algorithms

Smoothing Algorithms

System Design and Implementation

Applications and Use Cases

Physical Activity Diary

Mobile Healthcare and Wellness

Human-Centric Sensing and Sharing in Mobile Social Networks

User Interfaces

Conclusion and Future Work

References

Gestures in an Intelligent User Interface

Two Sides of the Same Coin

Related Work

A Human's Perspective

A System's Perspective

Experiment 1: Intuitive Gesturing

Method

Setup

Results

Condition Qx

Condition Xp

Sample Summary

Commands: Pointing

Commands: Selecting

Commands: Deselecting

Commands: Resizing

Conclusion

Experiment 2: Gesturing in the Interface

Method

Out-of-Range and Tracking

Select and Deselect

Rotate

Resizing

Restore and Remove

Results

Questionnaire

Observations

Conclusion

Conclusion and Discussion

References

Video Summary Quality Evaluation Based on 4C Assessment and User Interaction

Introduction

Related Work

Uniform Framework for Video Summary Quality Evaluation

Summary Unit Sequence Generation

Frame Alignment-Based Summary Unit Matching

Similarity-Based Automatic 4C Assessment

Coverage Assessment

Conciseness Assessment

Coherence Assessment

Context Assessment

User Interaction Based Individual Evaluation

User Interaction Based Requirement Gathering

Transformation of 4C Assessment Scores

Incremental User Interaction

Experiments

Validation of 4C Assessment Algorithm

Validation of Incremental User Interaction

Validation of Evaluation Result Transformation

Conclusions

References

Multimedia Experience on Web-Connected CE Devices

Introduction

Digital Photography Ecosystem

AutoPhotobook System

Design-Driven Photo Selection and Pagination

Blurry Image Removal

Duplicate Photo Detection

Theme-Based Pagination and Layout

Artistic Background Resizing and Assignment

STArt Design for Automatic Resizable Artwork

Transformation Algorithm

Dynamic Photo Layout Region on the Page

Theme Grammar for Photobook

Automatic Layout

Prior Related Work

The AutoPhotobook Layout Engine

Results Illustrating Text Support

User Interface Design

Powering CE 2.0 with AutoPhotobook

Conclusion

References

Index