Video of New iPad Interactions

During Apple's 90 minute unveiling of the iPad, a lot of new multi-touch interactions were shown off. But they went by fast. Now you can quickly just catch the UI in action on Apple's new native iPad and iWork applications in these videos.

iPad Interesting Moments

The idea of this collection is to capture the interesting moments during the various iPad interactions (slow down time) to understand the nuances in their design. Explore the image galleries for these interaction snapshots.

iPad Photoshop Template

The iPad Photoshop Template was constructed using vectors, so it’s fully editable and scalable. You’ll notice there are a few new UI elements as compared to the iPhone interface.

iPad Stencil for Omnigraffle

This is the first version of an OmniGraffle template for designing iPad apps. It contains backgrounds, title bars, buttons, selectors, and other iPhone UI elements. The text is fully editable on lists, title bars, buttons, and scroll wheels.

iPad icon Photoshop template

A Photoshop template based off the exact overlays, outlines, and masks the iPhone and iPad OS use to mask icons. It’s made up entirely of shape layers and layer effects and should be completely pixel-accurate.

iPad User Experience Guidelines

Apple's Human Interface Guidelines for the iPad outline how to create user interfaces optimized for the iPad device. According to Apple, the best iPad applications: downplay application UI so that the focus is on content; present content in beautiful, often realistic ways; and take full advantage of device capabilities to enable enhanced interaction.

More..

More design-centric articles about the iPad

• The average US Internet user watches 182 videos online in a month. (source).
• The duration of the average online video is 3.7 minutes. (source). This is up from 2 minutes 46 seconds in 2008 (source)
• The average video on YouTube (which serves 1 billion videos per day) will get 500 views over time. (source)
• 25% of those views will come in the first four days (source)
• By and large, only the first 30 to 60 seconds of an online video will be watched. (source)
• 45% of views come from direct navigation (search) and the other 55% of the time, people stumble across a video and "discover" it. (source)

"The more true to life your application looks and behaves, the easier it is for people to understand how it works and the more they enjoy using it." -Human Interface Guidelines for the iPad

Yet it's no secret that physical metaphors can easily be overdone in application user interfaces. Just see Microsoft Bob for an example. So I wonder if this guideline from Apple is a deliberate recognition of (or push for) the digitalization of many of our common physical objects.

As Jeff Dachis put it "everything that can be digital will be because it would be better, faster, and cheaper." We've certainly seen a lot of iPhone apps that essentially turn hardware products into software products. Similarly, it seems likely the iPad will begin to replace people's planners, address books, books, atlases, and more in the home. After all, the digital versions of these tools are better, faster, and cheaper.

Apple's application designs may be trying to make that obvious to us through their presentation as we're not likely to let go of physical objects easily. As William Gibson once noted: "one of the things our grandchildren will find quaintest about us is that we distinguish the digital from the real."

• When: Wednesday, February 17, 2010
• Socializing: 7:00-7:30 pm
• Program: 7:30-9:30 pm
• Where: Yahoo! Inc.
  701 First Avenue
• Meeting room: Building C, Classroom 5
  Sunnyvale, CA 94089

Official description

A Web application can't exist without users' input. We need the data, but do we need Web forms to get it? Traditional Web forms—made up of text fields, radio buttons, check boxes, and a Submit button—have been the mainstay of Web application design. In ecommerce, social networking applications, and productivity tools, Web forms continue to define crucial Web interactions. Thankfully, new approaches for user input are now giving designers more to work with and create better user experiences.

Luke will explore several novel ways in which Web applications can collect user input, through both mobile devices and desktop software, without forcing users to complete lengthy, sequential forms. He’ll provide detailed research-based solutions that articulate not only the how, but the why, as well. You'll walk away with practical solutions you can immediately put to use.

Hope to see some of you there!

The iPad user experience guidelines are © 2010 Apple Inc.

• Support All Orientations
• Enhance Interactivity (Don’t Just Add Features)
• Flatten Your Information Hierarchy
• Reduce Full-Screen Transitions
• Enable Collaboration and Connectedness
• Add Physicality and Heightened Realism
• Delight People with Stunning Graphics
• De-emphasize User Interface Controls
• Minimize Modality
• Rethink Your Lists
• Consider Multifinger Gestures
• Consider Popovers for Some Modal Tasks
• Restrict Complexity in Modal Tasks
• Downplay File-Handling Operations
• Ask People to Save Only When Necessary
• Start Instantly
• Always Be Prepared to Stop

More Info...

Details on these guidelines and further information on developing for the iPad can be found in Apple's Human Interface Guidelines for the iPad

• Nokia's Ovi Store is seeing more than 1 million downloads a day on handsets around the world - or roughly 30 million per month. (source)
• Apple's App Store now has over 140,000 apps and users have downloaded apps over 3 billion times to date. (source)
• Apple's App Store serves more than ten times the amount of apps per month as the Ovi Store on just 75 million devices total. Nokia in the fourth quarter of 2009 alone shipped 126.9 million devices! (source)
• AT&T’s had 85.1 million wireless subscribers at the end of 2009. Verizon ended the year with 87.5 million wireless subscribers. (source)
• During the fourth quarter of 2009, the number of wireless eReaders connected to AT&T’s wireless network increased by more than 1 million. (source)
• Forrester predicts that 10 million e-readers will be sold in the coming year. The firm estimated that 3 million eReaders were sold in 2009. (source)
• In 2009, the United Kingdom averaged 265 million text messages and 1.6 million picture messages per day. 2009’s text message total was 96.8 billion. (source)
• Some 740 billion text messages were sent in the first half of 2009 in the United States. That breaks down that astonishing figure to 4.1 billion texts per day. (source)

Well what's wrong with personal computing today? Complex operating system interfaces, technical input devices, security issues, and frequent crashes -to name just a few things. But how do the iPad's technical limitations potentially address these issues?

Running Multiple Apps at Once

The iPad follows the iPhone's application model. So you can't run two different programs at the same time and switch between them. It's one app at a time. Why would Apple do this? The iPad clearly has enough screen real estate to support multiple apps at once.

Ever have a computer screen that looks like the image above? Multiple apps open. Each with multiple windows. Working in this environment means a lot of time spent managing windows, switching between applications and generally clicking all over the screen but getting no work done. The iPad has none of that.

In fact, the iPad goes even further by abstracting away the file system entirely. "Rather than saving their documents into a file system, apps on iPad save all their documents within their own installation directory. Delete the app and you'll clean out all of its related files."

In the future, file management can be even easier once all your documents are in the cloud. No need to worry about backing up, not having the file you need with you, or duplicating files across multiple synched devices. The iPad has taken a first step toward this model as iPad apps can share documents wirelessly.

Not Supporting Flash

The iPad boasts having the best Web browsing experience of any device. Yet it does not support Flash which powers video, interactive charts, games, and more on many Web sites. Why would Apple not support Flash?

Turns out, Flash is the single leading source of application crashes on Mac OS X. It's a component Apple can't fix because they don't own it. It belongs to Adobe. "Apple controls the entire source code to the iPhone OS. If there’s a bug, they can fix it. If something is slow, they can optimize or re-write it." Not the case with Flash.

Update: Supposedly Steve Jobs confirmed this reason at an Apple Town Hall: "Apple does not support Flash because it is so buggy, he says. Whenever a Mac crashes more often than not it’s because of Flash. The world is moving to HTML5." HTML5's audio, video, and canvas elements provide open, standards-based ways to provide most of the functionality Flash adds online. Many Web publishers are actively supporting these standards.

The App Store Review Process

To get an application on the iPad, developers have to submit it to Apple for approval. Only then will it get published and be made available to users. Why would Apple prevent access to all applications? Why not just allow any application into the App Store?

At the start of 2009, there were 2.6 million malicious code threats online (viruses, trojans, etc.). 921,143 were added in the fourth quarter alone. In December 2009, a malicious phishing app on Google's Android platform was able to harvest bank login details from people who downloaded the app. The incident raised questions about whether a tighter vetting process is needed for the Android Marketplace.

Having a review process not only keeps malicious apps out, it also keeps buggy apps out. Again limiting the number of crashes people experience on the iPad and thereby preserving a great user experience.

Of course, you can't mention Apple's App Store review process without encountering the common assertion that, in technology (and especially the Web) "completely open" is better than "controlled".

Now, I'm as big a fan of the open Web as anyone. But that doesn't mean I don't appreciate or understand the value of closed systems. Closed systems enable companies to do seamless integration without putting the burden on their customers. An open system (like Android on smart phones) enables you to do what you like. It's like Linux on the phone -with all that entails.

Apple, instead, makes their living by tightly controlling the experience of their customers. It's why everyone praises their designs. From top to bottom, hardware to software -you get an integrated experience. Without this control, Apple could not be what it is today.

No USB Ports

The iPad only supports USB connections through a camera connection kit. Why not simply allow any device to plug into the iPad through several ports? Anyone who has had to manage disc drivers, device software installs, and upgrades for peripherals knows the pain they can cause. To add insult to injury, they make your computing life look like a rat's nest of wires (image by SinaX).

No Onboard Camera

I don't know why there is no camera on the iPad. It could be a cost issue but Apple sells the iPod Nano with a built-in video camera for $150. Guess we'll have to wait for v2 of the iPad.

A Simpler Computer

So while people are likely to continue assailing the iPad for these technical limitations, perhaps there's good reason behind Apple's decisions. Together they create a computing experience with less UI management, less crashes, less malicious apps, less device management, and thereby less complexity.

Thanks to Fraser Speirs' great retort to the negative reviews of the iPad that got me thinking about this topic.

Interactions in Apple's iWork Applications for iPad

Creating Bundles (0:21) In Keynote for the iPad, you can move multiple slides at once by collecting them into bundles. Start by selecting a single slide to move (hold finger down on slide). While holding this slide, tap other slides you want to move with your free hand. These additional slides get "bundled" together and you move the bundle to the desired position.

A similar bundle or “pile” treatment is used in Mail. As each message is selected, it gets added to a pile. You can then mass delete the pile of messages.

Multi-touch resize (0:41) Resizing an image is a simple matter of manipulating a drag handle with your finger (handles appear when you tap on an image). However, you can quickly make two images the same size by tapping on a second image while resizing the first.

Page Navigator (1:45) In Pages on the iPad, you can press and hold your finger in the scrollbar area to access a page navigator feature. As you slide your finger down the screen, you see a small preview of the page you will be taken to when you release your finger.

Contextual Keyboards (2:35) Another feature we’ve seen in use on the iPhone is the contextual keyboard. You can see how this is logically extended for the Numbers iPad application, providing a numeric keypad for number columns with access to formulas via a floating control panel.

Interactions in Apple's Native iPad Applications

Floating Control Panels (0:00) In portrait orientation email messages consume the entire screen, leaving no space for the list of messages. However, clicking an Inbox button at the top of the screen reveals a scrollable message list in a floating control panel. Note the similarity of this control panel to the standard iPhone navigation UI.

Optimized View by Orientation (0:10) When rotating the device from portrait to landscape in the Mail application, a split pane appears and the list of messages in the inbox is displayed on the left.

Spread/Pinch to expand/close piles of photos in iPhoto (0:38) In iPhoto, piles of photos can be expanded or closed by using a pinching motion. This gesture could be applied to virtually any document type.

Information Popovers (1:10) iTunes albums appear in a grid format. Tapping on an album cover invokes a visually rich popover containing a list of the album's songs. Tap any song to play.

Drag Over Lists for More Detail (1:35) In iCal's month view, dragging your finger over each event reveals a floating panel with additional detail.

More gestures in the SDK?

Of course, there’s plenty of user interface innovation yet to come from third party developers. In one example during the Apple keynote, Gameloft showed off a a game that used a three finger gesture to unlock a door. It will be interesting to see if developers have the flexibility to define a lot of new gestures using different combinations of fingers. After all, the large iPad screen provides scope for multifinger gestures, including gestures made by more than one person.

Much thanks to Craig Villamor for helping find and write-up these interactions!

The Market

The iPad is not a laptop nor is it a smart phone. It is a couch device, a bedroom device (don't read that the wrong way), and a kitchen device (swivel it to cook from a recipe you find online). In all these places, a laptop always felt wrong. The iPad is optimized for media consumption: surfing the Web, reading blogs/news/books, watching TV shows, playing casual games, listening to music, managing personal productivity (calendar, contacts) and looking at photos. Expecting it to provide the creation capabilities of a laptop is the wrong frame of reference.

Instead think of it a digital version of your leisure time activities –reading, chatting, light gaming, surfing, etc. The majority of these are consumption oriented –not creation oriented.

I suspect Apple tried to drill this point home by having all their keynote presenters sit in a big comfy chair while using the iPad. Steve Jobs himself spent over five minutes of the keynote just browsing the Web with it.

Now some might note that the modern Web is as much about creation as it is about consumption and that people are no longer just consumers of media –they are participants in the conversation. While this is increasingly true, the vast majority of time spent online is still focused on consumption. And a lot of creation remains lightweight (commenting, rating, status updates) –which the iPad easily supports.

So what about the inclusion of productivity tools like iWork (word pressing, presentations, spreadsheets) in the iPad release? I suspect a primary driver for that may be the college student market. The iPad can hold all their textbooks and communication tools but also supports getting things done with a set of office tools.

Tablet-Specific Applications

Apple redesigned all their core applications for the iPad. I really like the transition these applications have made from desktop to iphone to ipad. It’s interesting to see how the form factor influences the interface design and how these commonplace apps seem to get more refined each time they are redesigned.

Multi-Touch Interactions

The iPad features a 9.7 inch (diagonal) very responsive and precise capacitive multi-touch screen. Yet the device’s interface appears to only utilize multi-touch gestures for pinching and expanding UI elements and the virtual keyboard. None of the potential multi-finger gestures Apple has spent a lot of time exploring were demoed.

In fact, Steve Jobs stressed many times during the keynote that over 75 million people (iPhone and iPod touch owners) already knew how to use the iPad. So it seems the company opted for UI consistency and an easy on-ramp for existing customers over new gestures and interaction models.

Floating Touch-based Control Panels

As suggested in several Apple patents, many of the iPad’s user interface controls are accessed as “floating” multi-touch panels that sit on top of content and applications. Each of these contextual panels seem to have a visible trigger present on screen that hides and shows the appropriate controls.

Other options for accessing these controls were proximity sensors and multi-finger gestures. Proximity sensors could bring up controls when a finger gets close to the screen. For example, displaying play, rewind, and pause controls when a finger nears the screen during video playback mode. Multi-finger gestures could display controls when two or three fingers touched the screen at once. I’m assuming Apple experimented with both of these approaches and discovered that a high number of false positives prevented them from being viable solutions?

Media Interactions

The New York Times showed off a newspaper reader application that mimicked the physical paper’s format but utilized multi-touch gestures to zoom into and out of content and playback video inline. This is an early example of the rich media experiences a large screen multi-touch device can enable, but there’s much more that can be done.

The iPad Case

While it’s unlikely to get a lot of attention, the iPad case looks like a model of multi-purpose design. It not only protects the screen but swivels so the tablet can stand on its own and folds under the device to prop it up for easier typing.

iTunes, App Store, and iBooks is Becoming a Mess

The creation of a third store for media, iBooks, creates some confusion between the App Store, the iTunes Store, and iBooks. Where does one go for what? Are audio books in the iTunes store or in iBooks? Does the Amazon Kindle application go in the App Store or in iBooks? Will my iBooks show up in iTunes or the iPod on the iPhone and the iPad?

Additional Observations

• Apple will be charging $9.99 for their spreadsheet application, Numbers, on the iPad. Microsoft charges $229.95 for their spreadsheet application, Excel, on Windows 7.
• A maximum storage size of 64GB seems low for a collection of music, movies, photos, and books unless there is a cloud-based storage solution coming from Apple soon.
• Not supporting Flash leaves many holes on Web pages where videos and interactive elements currently exist. That could create a sub-par Web surfing experience. But hopefully this is something a software upgrade could fix? But there are both technical and political issues at play here.
• Unlike the iPhone, there’s enough screen space on the iPad to run multiple applications –why hasn’t Apple supported this? Simplicity or a technical hurdle?
• It’s unclear how the iPad will interact with the iPhone. Instead of a triangle of Mac, iPhone, and iPad –“now a V with the Mac at the vertex.” (thanks Neil)
• The lack of an integrated camera prevents video chat and more. But you need save some features for the next version of the product right?
• GPS capabilities are only available on the 3G version of the device.

To illustrate this point, let's look at a relatively simple question: what's your phone number? We can provide some basic help text next to the input field to let people know how they should answer the question: “Use xxx-xxx- xxxx as the format for your answer.” We can structure the input fields and their affordances to accept only a specific format: three text boxes of the appropriate length separated by the appropriate punctuation. Or we can provide a flexible input field that allows people to answer the question the way they wish.

In this example, there are basically five different ways a valid US phone number could be specified. A simple script can check to see if one of these has been used and submit the information in whatever format the back-end system requires. The chore of adhering to a specific format is taken off the person providing an answer and instead given to a small bit of code.

Without any prescriptive help text or input field affordances to dictate a specific format, people won't assume a specific format is required. They'll answer the question how they see fit. Great -except when an input fields looks flexible but isn't.

Consider this phone number field on the location-based service, Gowalla. No help text or input field affordances are present, so it seems any valid answer will do.

Unfortunately, that's not the case. Entering a valid phone number using parenthesis around the area code generates an unexpected error. Though this input field looks flexible, it fails to live up to its promise. If you're going flexible, go all the way.

For more on Form Design...

Check out Luke's book about Web form usability, visual design, and interaction design considerations: Web Form Design: Filling in the Blanks.

• Young Americans aged 8 to 18 spend more than 7.5 hours a day using a smart phone, computer, television or other electronic device, compared with less than 6.5 hours 5 years ago. (source)
• They also spend an hour and a half texting, and the half-hour they talk on their cellphones. (source)
• Because so many of them are multitasking — say, surfing the Internet while listening to music — they pack on average nearly 11 hours of media content into that 7.5 hours. (source)
• In 2004 45% of teens had a cell phone. Since that time, mobile phone use has climbed steadily among teens ages 12 to 17 – to 63% in fall of 2006 to 71% in early 2008. (source)
• 2004 survey of teens, 18% of teens age 12 owned a cell phone. In 2009, 58% of 12 year-olds own a cell phone. We also have found that cell phone ownership increases dramatically with age: 83% of teens age 17 now own a cell phone, up from 64% in 2004. (source)
• In comparison, 77% of all adults (and 88% of parents) had a cell phone or other mobile device at a similar point in 2008. Cell phone ownership among adults has since risen to 85% in April 2009. (source)
• In 2006, 51% of all teens had ever sent a text message, while 58% had done so by 2008.  In 2009, 66% of teens use text messaging. (source)
• On average, young people spend about 2 hours a day consuming media on a mobile device. They spend almost another hour on “old” content like television or music delivered through newer pathways like the Web. (source)
• Youths now spend more time listening to or watching media on their cellphones, or playing games, than talking on them. (source)
• More than 7 in 10 youths have a TV in their bedroom, and about a third have a computer with Internet access in their bedroom. (source)

• Apple has a 91% share of the $1,000-plus computer market. In the first quarter of 2010, Apple sold an all-time best 3.36 million Mac units representing a 33% annual unit increase. (source, update)
• Apple has sold over 240 million iPods and over 20 million iPod Touches. The company has a 73.8% marketshare of digital music players. (source)
• Apple Sold over 38 million iPhones accounting for 8% of handset industry revenues but 32% of industry operating profits (source)

Check out the full set of illustrations and logos on Flickr

Graphics & Visualizations

• Processing.js: a port of the Processing Visualization Language to Javascript. It allows you to draw shapes and manipulate images on the HTML5 Canvas element. The code is light-weight, simple to learn and makes an ideal tool for visualizing data, creating user-interfaces and developing web-based games.
• Raphaël a small JavaScript library that simplifies working with vector graphics on the web. If you want to create your own specific chart or image crop and rotate widget, for example, you can achieve it simply and easily with this library.
• Bluff: a port of the Gruff graphing library for Ruby to JavaScript. It allows you to easily create graphical charts on the HTML5 Canvas element.
• WebGL: a cross-platform Web standard for a low-level 3D graphics exposed through the HTML5 Canvas element. WebGL brings plugin-free 3D to the web, implemented right into the browser.

Browser Scripting

• Jetpack: a Mozilla Labs project that enables anyone who knows HTML, CSS, and JavaScript to create powerful Firefox add-ons.
• Cappuccino : an open source framework that makes it easy to build desktop-caliber applications that run in a web browser. With Cappuccino, you don't need to know HTML. You'll never write a line of CSS. You don't ever have interact with DOM. You need to learn one technology, Objective-J (essentially JavaScript 2), and one set of APIs.
• ie-7.js: Javascript code that makes Internet Explorer 6 run like a standards compliant browser.

Mapping

• Natural Earth: a data set (of shapefiles) that people can use to make maps of regions and countries.
• Mapnik: high quality open source map rendering. Mapnik takes XML based location information and allows you to style it.
• TileCache: serves up the images you need to render maps.
• Protovis: a Javascript library for rendering data marks: markers, size markers, colors, sparklines, charts, and more. Protovis defines marks through dynamic properties that encode data, allowing inheritance, scales and layouts to simplify construction.
• You can design with Mapnik –to draw maps. Serve maps with TileCache. Make better markers with Prototvis. This combination gives you control over the design of geographic experiences online.

In a nutshell, a plethora of networked, touch-based devices is set to expand home, mobile, TV, and even automobile Internet application usage. Think Internet everywhere.

CES 2010 had no shortage of new netbooks, e-book readers, tablets, and even smartbooks (hybrids of smart phones and netbooks, and perhaps e-book readers and tablets). The common thread among these networked devices is the ability to get online and thereby read and write Internet content or run services from "the cloud". For designers, this means digital content will need to adapt to lots of new form factors (screen sizes, display capabilities). So it's time to brush up on scalable design and liquid layouts.

Not only is digital content going to be available on many new devices. It's expanding to new environments as well. That means people can access and use Internet applications in their home, TV, mobile and car. Not only are companies releasing networked devices for these environments, they are supporting platforms for them as well. Once there's a platform -you can build on it. That means developers can create software for it or hardware to integrate with it, and content authors can publish to it. Many of these platforms are adopting Apple's app store model to distribute and sell applications including Ford's Sync platform for cars and Yahoo!'s TV widgets platform for TVs. In these new environments for Internet applications, not only will designers need to consider user context (on the couch, in the car) but specific application frameworks and capabilities as well. What Ford Sync's platform supports Apple's iPhone platform may not.

Last but not least, Natural User Interfaces (NUI) are the dominant mode of interacting with this new set of networked consumer device platforms. NUIs rely heavily on touch and gesture based interactions that reduce the distance between user and content. So the interface on NUI-based devices is physical and often invisible. As we are still in the early days of natural user interfaces, designers should expect significant improvements and many new UI experiments. For example, Motorola debuted its Backflip phone (with a touch surface in the back of a phone, allowing you to control the screen without obstructing your view of it) and Microsoft said it would launch Project Natal, (which has camera-based sensors that can detect your full body movements) in the fall of 2010. So if you haven't gotten your feet wet with natural user interfaces, now's the time to get some lessons learned.