Topic

The XEROX Star, X, the MacIntosh and Microsoft Windows together have defined how people will interact with computers for two decades. The screen, keyboard, mouse, windows architecture has been the basis for an explosion of interactive development. However, with the advent of mobile computing, new screen technology, and new interactive devices that architecture is no longer adequate.  Interactive systems architecture is caught in a local maximum that industry is not likely to escape unaided.

The system architecture defines what is easy to develop. It also defines what kinds of services can be implemented pervasively across many applications. Cut, copy and paste redefined how applications interoperate. Many more such pervasive features are possible but they rely upon innovative new architectures to support them.

Topics

Radical variation in display sizes are becoming the norm rather than a laboratory rarity. Applications must move between tiny cell phone screens to rooms with ten or more large displays. Computing power is portable and steadily increasing but adaptation to display capacity is severely limited. New display technologies will be wireless, dynamically assembled by users and widely varying among the many parts of a user’s life. The architecture must reflect this.

For the last 20 years new interactive techniques and devices have been invented, evaluated and demonstrated. These include, touch with multiple fingers, pens, gestures, new menu techniques and other approaches. Few of these interactive modalities have been deployed because the interactive system architecture cannot support them. If an input cannot be made to look like a keyboard or mouse the new technique cannot be used without rewriting all of the applications. New more flexible system models for interaction are required.

The architecture for interactive applications designed for one user, on one device. This does not reflect modern reality with many people individually owning and using many computing devices to work with diverse groups of people from many parts of their lives. Users interact in a many person, many device world but the underlying interactive system does not reflect this very human behavior.

Virtually every interactive application eventually divulges all of its information in the form of pictures. This makes information visible to people. However, this information is invisible or inaccessible to other programs. People can see information and can want to use it in other ways but such use must always be manual with individual labor because there is no architecture for sharing visual information between programs in effective ways. This architectural deficiency is particularly glaring when one attempts to assist users and leverage their abilities using machine learning algorithms.

Direct manipulation is the dominant interactive paradigm today. However, direct manipulation cannot handle the massive amount of information available to today’s user. Intelligent, learning-based assistance must amplify human effort. However, new interactive architectures are required to facilitate such assistance to reach into interactive applications to seek information or to perform actions on behalf of users.