Eric Florenzano’s Blog

What do apps look like in VR?

Jul 27, 2014

Let's assume that VR or augmented reality is the future. It's a leap to make that assumption, because who knows how the marketplace will react to VR, but let's make the assumption anyway because it's interesting. What do apps look like in this future VR world?

Maybe it works like apps do on the desktop, where there's an operating system interface, and apps have window boundaries that they have to fit within. This is the easier choice, since apps and websites already understand this concept and are able to be mapped into VR without a complete rewrite. Users are already understand how to interact with their computer in this way, so it's easy for people familiar with desktop computing to get going (and a lot of people are.) In fact, there is a cool demo of this working in Linux already.

This would be a bummer, if this is the way things go. While apps are easily mapped into this desktop environment, they're written without VR in mind. They don't take advantage of the third dimension: depth. They don't take into account your position or your head's tilt. (There's an argument to be made that they rarely even take advantage of two dimensions. Most apps and websites tend to primarily scroll in one direction.) In this scenario, the win is that you get more screen real estate to work with than you have with a monitor, and you get more spacial awareness than you have today using something like OSX's Mission Control. To me, this would be like Microsoft's first attempts at tablets, where they shipped it with a stylus and treated that as a mouse pointer. That is, it worked, and nothing needed to be rewritten, but it wasn't compelling enough for users.

Maybe this operating system interface will be written though, because whoever controls it will make a lot of money. A lot. Also we know by now that worse is better. But no, I think we're going to see it play out more like how games work on the desktop, and how both games and apps do on smartphones and consoles. Remember, VR is being driven primarily by gaming right now, so games will play a large role in setting expectations for how apps will work in VR. So let's add this second assumption to our list: apps in VR will take over the entire field of view, like games do.

Every computing platform that we use today has an understood interaction model. We have buttons you can tap, surfaces you can scroll with your mouse or finger, menu/tab/action bars, launchers, app windows, etc. Which of these things make the transition to VR, and what new interaction models need to be (re?)invented? For example, several Oculus demos are experimenting with letting users "click" on buttons by having the user "look" at the button with a pointer for a certain amount of time. This allows users to interact with the app without even using their hands. Even if this exact technique doesn't take hold (it's somewhat annoying to have to wait for a timeout in order to "click" on something), it's the experimentation that's happening that's exciting.

Notice that I put "click" in quotes. Because there's no clicking in that technique. Nor is there any tapping. We even have to invent new words to talk about this stuff!

Now in our hypothetical future we've got apps in VR and they take up the screen. Facebook bought Oculus, so let's talk about a Facebook property: Instagram. What does an Instagram app look like in VR? Is it a timeline in a narrow window, as it is today? Maybe clicking on a photo zooms in on it. Or maybe the timeline shows photos going down vertically, and all the comments are lined up horizontally without needing to tap into a detail view. Maybe photos aren't displayed linearly from top to bottom, but more like a huge Pinterest board projected inside a planetarium-like dome.

So the question of "What do apps look like in VR?" is not hypothetical, it's a real question that we're in the process of finding out the answer to. Maybe VR doesn't take off, or maybe worse really is better and the 3D desktop interaction model wins. But I don't think so. I think someone or some group is going to figure out how apps are supposed to look and work on VR, and that's going to be a big deal.