The Devil's Advocate - Mac UI Ain't All That: The Future & History of the User Interface (with Snazzy New UI Videos)
by - August 15th, 2006

The current state-of-the-art User Interface (UI) we've been enjoying has remained largely stagnant since the 1980s. The greatest innovation that has been recently released is based on video card layering/buffering techniques like Apple's Expose. But, there is a large change coming. Rev 2 of the UI will be based on multiple gestures and more directly involve human interaction. Apple is clearly working in the area as some of the company's patent filings demonstrate. Nevertheless, these videos might make Mac (and Windows) users experience a huge case of UI envy, as a lot of UI development (in XGL in particular) makes the current Mac UI seem creaky and old fashioned.

I've been collecting these links to cool new UIs for a while (yea, I know, what a geek). Despite its geekiness, I thought some of you might enjoy them as well. It seems there is a good chance that multi-gesture/multi-input (MGMI) UIs will be in our future at least on some level (apparently they will be accompanied by no dearth of techno music).

Basically the rest of the article has a bunch of links, and brief descriptions to those links, to cool videos of various UI that include: Wobbly Window UIs (WWUIs), Multi-touch UIs, Mixed Physical UIs, Brain UIs, Art tools, and even a little bit of history on the UI. Enjoy!

Wobbly Window UIs

The "wobbly" UIs, largely represented by XGL, all seem to use layering techniques. All this layering of UI elements lets you see the desktop UI from more vantages. Wobbly Window UIs seemed to have started at SUN with Project Looking Glass (although they may have originated elsewhere, I'm unsure). It seems that Project Looking Glass "inspired" Apple's Exposé. Others have gone on to build on both SUN's and Apple's progress with the current state-of-the-wobbly-art with XGL. The 'gist' of how these UIs seem to work is they put a bitmap into a buffered backing plane and allow you to quickly move the resulting 2D bitmap layers through 3D space. Modern 3D video cards then allow you to move these relatively simple bit-mapped textured planes through 3D space rather easily.

WWUI Links:

Multi-touch UIs

Multi-touch UIs are much like the UIs on a touch pad or a tablet UI. You get to touch things directly on the display surface. However, this technology registers more than one touch, and it is capable of interpreting gestures as well. In reality, some of these UIs do not even require touching the surface, but just interpret your gestures near the surface. Multi-touMulti-touch UIs, along with mixed physical UIs, seems to be the direction where UI development is headed.

Multi-touch UI links:

Mixed Physical UIs

Mixed Physical UIs build upon multi-touch UIs, but they introduce physical objects into a multi-touch input plane. Some of the more advanced Mixed Physical UIs identify the types of physical objects that are placed onto the input surface, and then let you manipulate the computer environment when you manipulate the physical objects.

Mixed Physical UI links:

Brain UIs

Brain UIs employ techniques of reading brain impulses in order to control a computer.

Brain UI links:

Art tools

Art tools use UI techniques, already discussed above, to further artistic ends.

Art tools links:


Sadly, a great many people in the computer field have a pathetic sense (or rather ignorance) of history. They are pompous and narcissistic enough to ignore the great contributions of past geniuses. My hope is that more will watch some of these historical videos and learn from these innovators.

Perhaps, if more learned from the great computer science innovators of the past (who managed to make systems using computers with 192KB of memory that rival the responsiveness and features of modern systems having orders of magnitude more processing power), then later generations of computer scientists might adopt their elegant design philosophies, rather than the current philosophies responsible for the ill-thought out and implemented bloat software we "enjoy" today. It might be time to add a "History of Computers" class to the computer science curriculum so as to give new practitioners this much needed sense of history.

For example, it seems rare to find someone in the computer field who knows about the contributions of Ivan Sutherland and Douglas Engelbart. Ivan Sutherland came up with one of the earliest graphical interfaces (and threw in objected oriented programming techniques for good measure); he did this all in one year in the development of Sketchpad circa 1963. Amazing.

For a good historical overview of user interfaces and more, check out this video with Alan Key; it's fantastic.

Douglas Engelbart's demonstration of the online system, NLS, is THE demo that started them all. Most think that Apple/Microsoft stole their UI ideas from Xerox, but in reality, Douglas Engelbart and his team at Stanford made most of the conventions responsible for current computing (as well as current UIs). This is really the mother of all computer demos; Steve Jobs and others have basically copied this format, but no demo has ever equalled the thermonuclear impact of the NLS demo. Keep in mind, it was made in 1968! Douglas Engelbart is known as having invented the mouse, which is true, but that's a bit of a backhanded compliment as he did so much more. In reality, he and his team came up with our current paradigm of computer interactions, abstractions and development. It is beyond bizarre that he is not commonly credited with more.

Although a lot has come from Mr. Engelbart's work, I think he is actually disappointed with our current state of the art. I don't think he believed we should make UIs that are "simple" (or dumbed down), but instead, he was trying to develop UIs that empower us to embrace and engage in complexity; blame Jobs and team for our going astray with the one button mouse et al. Anyway, this is a long video and is somewhat tedious to those without a computer science background or sense of history as the technologies are not as whizbang by today's standards (and no techno track to keep you distracted). For its time, however, this was incredible; after this demo, many at the presentation left not believing the technology was real. Keep in mind this was achieved on a 192 Kilobyte (not Megabyte, not Gigabyte, but Kilobyte!) time share computer (that supported 20 users) with 0.5 MIPS of processing power.

This video is really the precursor to all modern demos and systems. In so many ways, it's frightening. Just setting up the demo was a technological marvel; it was projected on a huge display (using a very rare projector for the time) that they hacked to work with NLS; they even conducted a video teleconference with custom microwave based modem communications that they "just whipped up" to enable the demo.

How Douglas Engelbart, Ivan Sutherland, Doug Ross, and others have not collectively received a Nobel Prize for bringing forth modern computing is a mystery.

Xerox Smalltalk environment (one of the Xerox UIs from which everyone "borrowed").


I get the feeling we are about to enter a computer user interface renaissance. My guess is that somewhere in the next 2 years (and maybe as soon as with Mac OS X 10.5), we'll see the revitalization of the entire computer field because of UI advances. And with any UI renaissance, we will also see increases in new hardware sales supporting the new paradigms, which in turn will lead to further gains in personal productivity levels. If you find some other cool UI videos, share by adding links in the comments below.