Reality bites

The infamous smartphone patent wars lasted from 2008 to 2014 and involved damages awards of more than $100 million, injunctions against product features with a dominant market share and companies with billion-plus annual patent licensing revenues. If augmented and virtual reality becomes the next computing platform, as is widely predicted, the conditions for another patent war look likely. While several major companies have been anticipating this and have positioned themselves accordingly, many others are still playing catch-up with their patent portfolios and may be forced to make significant acquisitions or pay substantial licence fees.

Smartphone patent wars

The smartphone patent wars refer generally to the patent litigation occurring between operating companies between 2008 to the present over smartphone market share. The Wikipedia entry (https://en.wikipedia.org/wiki/Smartphone_patent_wars) has a fairly accurate timeline for the operating company litigation. It begins roughly in 2009 with Nokia suing Apple, and Apple counterclaiming against Nokia. The terminal throes of the wars are bookmarked by the Apple v Samsung patent litigation.

Colourful graphics abound that depict the smartphone patent wars.

What do not appear in the graphics or the Wikipedia entry are the motivations and strategic moves of the various companies involved. Additionally, smartphones spawned a large number of non-practising entity (NPE) lawsuits on various aspects of the smartphone environment. These lawsuits were pure money plays (as opposed to plays for market share) – for example, Smartflash received a $532.9 million jury verdict against Apple, while the University of Wisconsin recently obtained a $234 million jury verdict against Apple, both related to smartphones.

However, the key takeaways were the motives and strategic moves of the traditional operating companies. The auction of 6,000 Nortel patents (directed more to enabling 4G/LTE networks than to the phones themselves) resulted in a winning bid of $4.5 billion from a consortium made up of Apple, BlackBerry, EMC, Ericsson, Microsoft and Sony, known as Rockstar. Over the years the Rockstar consortium has recouped part of this purchase price through a patent assertion and licensing programme, suing Google, Samsung, Huawei, ZTE, LG, HTC, Pantech and ASUSTeK in the Eastern District of Texas. In 2014 Rockstar was wound down, with 2,000 patents distributed to members and the remaining 4,000 sold to RPX for $900 million.

After losing at the Nortel auction, Google purchased Motorola Mobility – and its 17,500 issued and 7,500 pending patents – for $12.5 billion as a defensive measure. In 2014 Google again spun off Motorola to Lenovo for $2.91 billion, but retained many of its patents and presumably a licence for those patents included in the sale. Interestingly, the Motorola patent portfolio included some augmented and virtual reality patents, so Google is expected to continue to reap a future return on its investment.

Microsoft has largely been regarded as a loser in the smartphone patent wars. Its mobile operating system was never widely adopted, at least compared to Android and iOS. Microsoft avoided the smartphone handset business, except for its purchase of Nokia’s failing handset business.

However, this perception overlooks Microsoft’s strategic genius and consolation prize. Microsoft has developed a robust patent portfolio over the years related to smartphone functionalities and operating system features. While this did not translate into market share, it is still one of the most successful patent licensing programmes in history. Almost every Android phone company holds a portfolio licence and it has been estimated that this nets Microsoft $6 billion in annual revenue (see www.forbes.com/sites/ewanspence/2015/11/01/microsoft-android-patent-income/). If accurate, this translates into almost one-third of Microsoft’s net income. Not a bad consolation prize.

Augmented reality

The smartphone patent wars were fuelled by a high-stakes competition among established, deep-pocketed global brands around a highly disruptive new computing and communications platform.

Those same perfect storm conditions appear to be in play again with augmented reality.

Digi-Capital has boldly predicted that the augmented reality market will be worth $120 billion by 2020, with virtual reality (augmented reality’s fully immersive and current media darling cousin) to follow at a respectable $30 billion.

Those are staggering numbers, but they get a lot easier to swallow when the likes of Facebook’s Mark Zuckerberg is quoted as saying: “Every 10 or 15 years a completely new computing platform comes along, so in the 1990s we had these desktop computers, and now we have these much more intuitive devices with phones and tablets. And I think VR and augmented reality (AR) are going to be the next leap beyond this… I believe that VR/AR content will be the most shared form of content on the web within the next five years.” Zuckerberg goes on to explain: “I think it’s pretty easy to imagine that in the future we will have something that we can either wear – and it’ll look like normal glasses (so it won’t look weird like some of the stuff that exists today). And you’ll be able to have context with what’s going on around you in the world and communicate with people and not have to disrupt your conversations by looking down… We are focused on augmented reality.”

Digi-Capital managing director Tim Merel would appear to agree: “Augmented reality (ex Magic Leap, Microsoft’s Hololens) is like wearing a transparent mobile phone on your face. We think AR could fundamentally disrupt mobile, and the next Apple might come out of AR – depending on if and when Apple enters the AR market.”

Founder of Facebook-owned Oculus Palmer Luckey did not mince words about the long-term disruptive effect this new communications platform is expected to have. “More ubiquitous than the smartphone… eventually, augmented reality and virtual reality are going to converge and combine into the same sets of hardware… if you can carry it all the time…there’s no reason it can’t supplant smartphones.”

In addition to this talk, there have been some bold strategic moves and significant investment in the augmented reality space, much of this from some of the same players which participated in the smartphone patent wars.

Some notable examples include:

• Microsoft’s announcement of its Hololens augmented reality platform and head-mounted display;
• Google’s doubling down on augmented reality (post Google Glass) with its $500-plus million investment in Magic Leap, which recently announced an additional funding round of $1 billion, rumoured to be led by Alibaba; and
• Apple’s purchase of augmented reality-enabling software company Metaio.

Figure 1. Augmented and virtual reality revenue forecast ($ billions)

Augmented reality IP landscape

So if augmented reality is an essential part of the next-generation computing and communications platform, which companies have been investing heavily in intellectual property related to it?

What is augmented reality?

Terminology is important. Augmented reality generally refers to an artificial image on a real background, while virtual reality generally refers to an artificial image imposed on an artificial or virtual background. These constructs are useful for basic understanding, but the technology and use cases are not limited to two layers – that is, multiple layers of real and virtual content are combinable. For this reason, some refer to ‘artificial reality’, ‘mixed reality’, ‘cinematic reality’ or ‘hybrid reality’.

Because terminology is not used consistently, we looked primarily at augmented reality patents. The results are thus skewed in that this study misses important patents related to mixed or artificial reality that can be used in either augmented or virtual applications. For example, the few artificial reality patents that Facebook possesses relate to virtual reality – unsurprising, given its ownership of Oculus Rift. Further, a large number of mobile devices, such as head-mounted devices (HMDs), are potentially useful in either augmented or virtual reality applications. A useful review of HMD patents can be found at http://patentvue.com/2015/05/26/patents-shows-widespread-augmented-reality-innovation/. Of course, augmented and virtual reality are not mutually exclusive, and as noted by Luckey, eventually the hardware will converge and combine.

Early movers: issued US patents filed before 2010

For a high-level view, we looked at US patents that mention augmented reality in the specification. A reasonable starting point is to look to issued US patents filed before 2010. The most interesting and valuable patents are normally the earliest pioneering ones.

Looking solely at issued US patents which mention augmented reality in the specification with filing dates before 2010 yields a few surprises. Figure 2 organises the data by assignee.

Canon and Siemens clearly dominate this category. However, on closer inspection, most of Canon’s patents are photography related, while most of Siemens patents relate to industrial and manufacturing applications – important, to be sure, but not the type of consumer markets that drove the smartphone patent wars.

Notable by their absence are any substantial early priority date patent positions by any of the big ecosystem, hardware and enabling technology players, which we see currently positioning themselves to win in this lucrative space.

Figure 2. Issued US augmented reality patents with priority dates before 2010

Figure 3. Issued US augmented reality patents as of November 2015

Recent activity: total issued US patents as of November 2015

Substantial investments in augmented reality have been publicly announced by Google, Nokia, Microsoft, Qualcomm, Nvidia, AMD, Intel, Facebook, Apple, Samsung, Sony and others.

Have the patents issued been commensurate with this massive investment? Not really; but it appears that Google and Microsoft have taken action to buttress their augmented reality patent portfolios.

So far, the big winners appear to be Google and Microsoft because of the inclusion of augmented reality patents acquired by acquisition.

For example, Google acquired a substantial number of augmented reality patents from Microoptical and Motorola Mobility. Microsoft has been a big patent applicant in its own name, but in addition has been busy building its technology and patent portfolio. Presciently, Microsoft acquired some of the Osterhout patents in 2014, although most are still pending.

Presumably, Microsoft’s transaction with Nokia includes a freedom to operate, if not outright ownership of, a large portfolio of augmented reality patents (Nokia has 61 issued, with many early patents).

Sony not only has early augmented reality patents, but also has continued its efforts in this area. Demonstrating once again that Sony can execute on a long-term strategy, its Project Morpheus is just the beginning of its foray into the artificial reality space.

Figure 4. Issued US augmented reality patents with augmented reality claims

Issued US patents that include augmented reality in claims

It is hard to draw definitive conclusions as to the scope of patent claims simply by looking for keywords in claims. Nevertheless, Microsoft has 204 issued US patents that mention augmented reality in the specification, but only 33 that actually include the phrase in a claim. Canon has only three patents that claim augmented reality, so the strength of its portfolio bears scrutiny. Similarly, of Nokia’s 61 patents which mention augmented reality in the specification, only six actually include the phrase in a claim. Apple barely features, with only two patents that claim augmented reality. However, it recently acquired Metaio, which owns eight patents claiming augmented reality.

Issued US augmented reality patents and pending applications

Microsoft has about 832 pending applications in addition to its 204 issued US patents. As mentioned, the transactions with Osterhout and Nokia have given the software giant an even greater treasure chest of patents. The transactions are not sufficiently transparent to be precise, but Osterhout has around 155 pending US applications and Nokia 250, meaning that Microsoft has access to, if not ownership of, about 300 issued US patents and 1,200 pending US applications which mention augmented reality.

AMD, Qualcomm, Nvidia and Intel have all announced products in artificial reality. However, AMD and Nvidia have virtually no patents directed to augmented reality. Qualcomm has clearly dedicated resources to this field and has 105 issued US patents and about 280 pending applications. Intel has 92 issued US patents and over 450 pending applications directed to augmented reality. Certainly Intel and Qualcomm are well positioned to take advantage of the next major computing platform.

What has Apple been doing? It appears that the tech behemoth is playing catch-up, with only eight issued US patents mentioning augmented reality, two issued US patents claiming augmented reality and 26 pending US augmented reality patent applications. Acquiring Metaio was a good move – while it has only 18 issued US patents directed to augmented reality, it has 65 pending US applications. Still, if Apple is to be a dominant player in the next major computing platform, it has a lot of work to do.

Facebook has no issued US patents directed specifically to augmented reality and only five pending US applications. However, it has 11 issued US patents directed to virtual reality and 32 pending virtual reality US applications. While this makes sense given its purchase of Oculus and its focus on virtual reality, it calls into question Facebook’s ability to expand into augmented reality.

At first glance, despite Google’s large investment in Glass, it appears to be a laggard in the patent space. However, Google has acquired patents from Microoptical, Motorola Mobility and Xerox, and counting these had 212 issued and 438 pending US patents directed towards augmented reality.

Further, it has invested $542 million in Magic Leap and its ‘cinematic reality’, resulting in 162 patent applications related to augmented reality in 2015. About half of these claim priority from a single November 2013 patent application filing by Brian T Schowengerdt. While Magic Leap has only one issued US patent assigned to it in augmented reality, its early patent rights are difficult to ascertain. It also draws heavily on the work of Chunyu Gao, Hong Hua and Yuxiang Lin from the University of Arizona (eg, US Patent 20120162549, filed December 24 2010). Additionally, some of its technology appears based on the work of Eric Seibel and Brian Schowengerdt from the University of Washington. It is difficult to know whether Magic Leap had any patent rights before the 2015 filing blitzkrieg without knowing its licensing arrangements with the University of Arizona, the University of Washington, Gao and Schowengerdt. However, it is clear that the company has a catch-up strategy and is investing heavily in patents.

Samsung has clearly noticed its deficiencies and is doing its best to remedy these. With 46 issued US augmented reality patents and close to 277 pending applications, Samsung’s IP strategy is closing the gap on its business strategy.

Figure 5. Issued US augmented reality patents and applications as of November 2015

Up-and-comers

Blippar, Meta, Augmate, HTC, Vuzix and Epson are just a few of the exciting new players that are driving innovation and application in this dynamic space. However, as is so often the case, their IP positions often do not reflect their market impact and potential.

Additionally, other established social media platforms that are curating location-based user-generated image and video content (eg, Snapchat’s Our Stories) will be expanding and enhancing their use of augmented reality to generate much-needed revenue – alas, Snapchat has no augmented reality patents.

Patents – why bother?

Since the Supreme Court’s decision in eBay, Inc v MercExchange, LLC, injunctions have been rare for all patent owners and almost non-existent for an NPE-owned patent. However, there are indications that obtaining an injunction against a competitor is the new normal. In Apple v Samsung the Federal Circuit determined that the trial court should have granted Apple an injunction, finding that it need only show that the patented features affected a consumer’s buying decision. Where a product has multiple patented features – as are typically found in augmented reality enabling devices – injunctive relief may become common for operating companies.

In the past, Federal Circuit law was such that a finding of wilful infringement – and the concomitant increase in damages – was rare. This contributed to the rise of “efficient infringing” (see “The Patent Troll Smokescreen”, The New York Times, October 23 2015). The thought process was – never pay for a licence. An injunction is not possible and the worst-case damages scenario is to pay a reasonable royalty for a licence.

The Supreme Court has now agreed to review the law on wilful infringement, which presumably means that the law is about to change (Halo Electronics, Inc v Pulse Electronics). If the Supreme Court relaxes the standard for a finding of wilful infringement as expected, enhanced damages will become more frequent. While the patent statute caps an increase in damages to three times, increased damages will decrease the incentives to operate as an efficient infringer.

In Alice Corp v CLS Bank International the Supreme Court made several sweeping announcements concerning patent subject-matter eligibility. District courts have run with the Alice decision and extended patent ineligibility to many domains. Curiously, the US Patent and Trademark Office (USPTO) is raising ‘subject matter’ rejections at about the same rate as pre-Alice (see, http://patentvue.com/2015/09/09/software-patents-are-resilient-in-the-wake-of-alice-corp-vs-cls-bank/) outside the business methods classifications (Class 705). This raises the possibility that a judicial reinterpretation of Alice will invigorate a large number of existing patents.

Augmented and virtual reality subject matter are not immune to Alice subject-matter inquiries. However, patents in this area are distanced from business method patents and should be allowed and issued at normal rates. There is no reason to suspect that augmented and virtual reality patents will have any unusual subject-matter problems. Particularly for such patents with claims written after Alice, there is reason to believe that they will generally be enforceable.

After prompting by the Supreme Court, the Federal Circuit has clarified the law on joint and divided infringement in Akamai v Limelight (Fed Cir 2015). A company now may be liable for direct infringement of a method claim even if its customer has performed some of the steps of the patented method. The test is fairly nuanced – the company must condition “participation in an activity or receipt of a benefit” upon the performance of a method step. While the law is evolving on the application of Akamai, clearly more method claims will be litigated with an increased chance of finding direct infringement. Such method claims were common in the smartphone patent wars and will undoubtedly be common in augmented and virtual reality patent litigation.

The net result is that the universe of potential infringers is greater thanks to Akamai. Further, increased damages and injunctive relief make the efficient infringer strategy a much riskier proposition. For example, there is no guarantee that Microsoft will license its extensive patent portfolio to a competitor at any price. Similarly, there is no legal reason to think that patent wars will not develop around augmented reality on the scale of the smartphone patent wars.

Key players

Microsoft has clearly positioned itself with a valuable portfolio of internally developed patents, as well as acquiring certain rights in Nokia patents and the entire Osterhout patent portfolio. Microsoft’s large number of pending applications indicates that this dominant position will only get stronger. Given that this article focuses on augmented reality, this may even underestimate the strength of the Microsoft patent portfolio because of it penchant for using the terms ‘mixed’ and ‘hybrid’ reality.

Google’s patent portfolio – including its acquired patents – appears to be commensurate with its technology development and further emphasis could position the search giant well.

Sony has made substantial investments and is well positioned to develop intellectual property around its technology. Canon is worth monitoring, while Siemens could dominate business and industrial application of augmented reality with its patent portfolio.

Qualcomm is still one of the best companies at bundling patents with technology and its efforts are noticeable. However, it recently elected to sell its Vuforia assets, which are most connected with augmented reality, to PTC.

Intel has made substantial investments in technology in this area and the intellectual property protecting it to bolster its position. For example, in June 2015, it acquired Recon Jet, an augmented HMD for athletes.

HTC has partnered with Valve and is bringing to market a virtual reality HMD, the Vive. Additionally, HTC participates in Valve’s SteamVR project. However, with only four issued and pending augmented reality patents, HTC has some work to do in strengthening its patent portfolio.

What is missing is the strength expected from Apple, Facebook, Nvidia and AMD. Despite large investments, these market leaders simply do not have the patent portfolios.

As happened in the smartphone patent wars, many Chinese and Korean firms are being left behind. Samsung is working hard to develop a patent portfolio relevant to the next major computing platform, particularly in view of its recent product announcements (eg, GearVR), but its patent efforts are still embryonic. However, movement can be seen – for example, Tencent has invested in Epic Games and Altspace VR and recently announced its own HMD.

Among small independent companies, only Information Decision Technologies and Solomoar appear to have noteworthy patent portfolios.