Who owns the future of autonomous vehicles?

While the transformational impact of autonomous vehicles (AVs) over the next 10 years has been much discussed, there has been little specific focus on how to achieve this goal effectively and the key role to be played by intellectual property. In the race to mass produce safe and affordable AVs, light detection and ranging (LiDAR) has emerged as a key enabling technology. LiDAR allows a detailed 360-degree view of the surrounding environment and plays a critical role in the plans of many potential AV players. However, LiDAR faces stiff technical, performance and cost challenges. In this environment companies with a strong IP position will emerge as big winners.

IP assets provide a detailed window into the innovation strategies of various companies and form the basis for emerging clusters that will allow traditional players to access patents and technology through partnerships. This article addresses these issues through patent-driven strategic insight, which highlights diverse LiDAR players and their IP positioning, as well as young companies which may possess seminal patents but small portfolios.

Figure 1. 3D map LeddarTech

LiDAR – the basics and the challenges

With its reliance on invisible light waves, LiDAR provides active sensing that illuminates the surrounding environment in order to generate detailed three-dimensional (3D) maps (see Figure 1). This allows LiDAR to provide high-quality sensing under all lighting conditions (eg, at night), in contrast with traditional camera-based systems. LiDAR also yields higher image resolution than radar and allows the environment to be mapped and the position to be determined simultaneously by comparing the image to previously scanned environments.

Figure 2. Waymo Minivan

Figure 3. Ford with Velodyne HDL-32

LiDAR’s strength is also its vulnerability. Since it is light based, LiDAR does not perform well in rain, fog or dust. In addition, it provides no velocity information (eg, the speed of an object ahead moving towards your path). For this reason, complementary sensing systems are often used that combine LiDAR, radar and image processing. Typically, a system selects the optimal signal source (eg, radar when velocity information is required about detected objects, or image data to confirm whether a stop light is red or green). For this reason, the term ‘sensor fusion’ is commonly used to describe AV sensing systems that rely on complementary technology. Cost represents a major commercialisation hurdle, with current LiDAR systems running as high as between $50,000 and $60,000 per vehicle. The consensus is that broad commercial market acceptance will require a 90% reduction in product cost over the next few years. Various solutions to this are being pursued, as highlighted in the patent analysis that follows. There are also a host of technical and performance challenges that must be addressed, including the design of smaller LiDAR units, lower power consumption and greater system reliability given the harsh vehicle environment in which LiDAR operates. LiDAR also generates huge amounts of information and therefore requires specialised high-performance processors to handle this data, along with software to create the resultant 3D maps for path planning and object avoidance.

Active companies

A look at several high-profile AV concept cars and their LiDAR units highlights the diverse solutions being pursued and the emerging constellation of LiDAR firms and traditional car manufacturers.

Figure 4. Quanergy - Mercedes

Figure 2 shows a recent Waymo self-driving vehicle with a LiDAR unit on the roof that relies on spinning technology. After using Velodyne’s LiDAR units for its early AV efforts, Waymo has now made significant investments to develop proprietary LiDAR technology in-house. Its efforts are highlighted and covered by a significant number of patents. In contrast, Ford is partnering with LiDAR pioneer Velodyne for its autonomous vehicle fleet, which is targeted towards ride-sharing services. Figure 3 shows a Ford Fusion with four Velodyne HDL-32E spinning LiDAR units. Another approach is highlighted in Figure 4, which shows a Mercedes vehicle with compact solid-state LiDAR sensor units from California start-up Quanergy fully integrated into the vehicle.

Figure 5. Partnerships web

A snapshot of the quickly evolving LiDAR ecosystem and the strategic partnerships among players is illustrated in Figure 5. For the most part, automotive companies continue to rely on their traditional suppliers for advanced parts and technology, and these suppliers have established various links with select LiDAR vendors. Thus, Delphi has links with Innoviz, LeddarTech and Quanergy, while Bosch has established links to Tetravue. Velodyne is working with Ford and the Chinese internet giant Baidu and Luminar is a strategic partner for Toyota. The large German automotive supplier ZF has partnered with Hella and Ibeo; Audi in turn has partnered with Valeo. Several non-traditional players have also emerged, as illustrated by Samsung which has announced plans to develop a software platform for autonomous cars. In support of this goal, Samsung invested in Quanergy and Tetravue. Google/Waymo has emerged as another non-traditional player, with a focus on developing LiDAR for its projected fleet of AVs. Other firms are accelerating the creation of advanced LiDAR technology by spinning off internal groups along with IP assets, such as Continental Advanced LiDAR Solution LLC. A more detailed listing of the developing partnerships and investments is set out in Table 1.

Key players

A high-level view of the IP landscape defining LiDAR is provided in Figure 6, which illustrates the patent holdings of the key players. Google/Waymo has the largest portfolio, closely followed by Continental Advanced LiDAR Solutions. The remaining players – AEye, Bosch, Hella, Ibeo, Velodyne, Luminar, Trilumina, Quanergy, Leddartech and Tetravue – have significantly smaller IP portfolios. Figure 6 also illustrates whether the patents are focused on hardware or system level claims (blue) or pertain to signal processing and conditioning (red).

Figure 6. LiDAR holdings by assignee

To ensure accurate patent numbers, the IP portfolios of all listed firms were manually reviewed by technical experts; ‘relevant’ patents were defined as those including claims germane to LiDAR for AV sensing. Patents teaching other applications of LiDAR were considered non-relevant and excluded (eg, LiDAR for aerial mapping or handheld devices).

AV pioneer: Google/Waymo

Waymo, formerly known as Google’s self-driving car project, is widely seen as one of the pioneers in the self-driving car race. Its patenting activity is high in all aspects of autonomous driving and includes a strong and diverse portfolio covering LiDAR technology.

Waymo’s self-driving activity began as early as 2010 and entailed an investment of more than $1 billion. Its early effort relied on Velodyne’s sensor hardware for self-driving cars. In an effort to get a tight grip on the software and hardware interplay, Waymo ended this partnership in 2012 and shifted development of its LiDAR technology in-house. Waymo’s patent filings show a surge in LiDAR patent filings in 2013 – these cover a wide range of inventions crucial to the technology. This sustained effort and underlying innovation strategy are illustrated in Figures 7 and 8. The impact of this internal R&D was dramatic: optimising the placement and alignment of the optical components reportedly allowed Waymo to reduce the cost of its units by 90%.

While many filings pertain to making incremental improvements to mechanically spinning units mounted on top of vehicles, they also contain more fundamental hardware and processing patents that should remain commercially relevant if the technology shifts to a solid-state approach in the future. Waymo currently has no IP holdings depicting solid-state LiDAR; its commitment to spinning LiDAR systems is evident by its continued filing focus in this space.

Figure 7. Waymo’s IP portfolio and innovation strategy

LiDAR veteran: Velodyne

Velodyne LiDAR is one of the most experienced companies in the game. A spin-off from Velodyne Acoustics, Velodyne LiDAR traces its roots back to 2005, when founder David Hall entered the DARPA Grand Challenge with a prototype of a new generation of laser-ranging devices for autonomous driving. This prototype would ultimately be commercialised as the pioneering HDL-64E spinning LiDAR unit, which was the go-to LiDAR hardware for many early efforts in the self-driving car area, including Google. Today, Velodyne claims to work with the vast majority of autonomous driving programs out there; it has received significant investment from Ford and Baidu.

Over the years the overall sensor unit size, weight and cost have continued to fall. However, the underlying principle of the entire product line-up has stayed the same: a linear array of lasers mechanically sweeps across the environment by rotating at several hundred revolutions per minute around the housing axis. In April 2017, this changed with the announcement of a new type of LiDAR sensor based on solid-state technology. Produced in mass volumes, this has the potential to reduce costs by orders of magnitude compared to the mechanically spinning units.

Despite it being one of the most recognised names in the LiDAR space, Velodyne’s patent filings through 2015 are sparse. Early pioneering work in the spinning LiDAR space is protected only by three patents, although a recent surge of patent filings since the end of 2016 hints at a change in this innovation strategy. Most of these recently published applications focus on the control of the laser diode pulses and their repetition patterns but are still traditionally spinning. However, Velodyne’s most recent application, published in December 2017, uses a beam-scanning movable mirror to scan the field of view and no longer requires the entire housing to spin.

Figure 8. Analysis of Waymo’s filing strategy

Early solid-state player: Quanergy

At the beginning of 2016, Quanergy created a lot of buzz in the industry and beyond after announcing its solid-state LiDAR system for driverless cars, marketed as the Quanergy S3 and estimated to cost $250 or less – vastly less expensive than existing systems on the market. With no moving parts, solid-state LiDAR promises to seize the lead in some of the most key areas pertaining to automotive sensing: reliability, weight, size, power efficiency and cost. This potential seems to be recognised by all corners of the industry. Over various funding rounds Daimler, Delphi, Samsung, Sensata and others have invested over $135 million so far, driving the valuation of Quanergy up to nearly $2 billion.

Quanergy is betting big on the move to solid-state. This is largely in line with its patent filings. After Quanergy was founded in October 2012, its earliest patent filings focused on a more robust version of traditional, spinning LiDAR units. Filings then shifted to inventions pertaining to optical beam forming and steering using solid-state technology, which now make up the bulk of its portfolio. As of December 2017, two of its solid-state applications were granted, with four still pending. Quanergy’s other publicly available pending applications pertain to 3D mapping using a plurality of LiDAR units around the vehicle, as well as signal processing (for an overview of its IP portfolio and innovation strategy, see Figure 9).

A partnership announcement in 2017 involves one of the largest original equipment manufacturers (OEMs) in the world: Koito Manufacturing Co and Quanergy are collaborating on seamlessly embedding Quanergy’s solid-state LiDAR sensors into Koito’s vehicle headlight units in the hope that not only cost but also more pleasing aesthetics will drive adoption and commercialisation of self-driving technology. A similar and more recent collaboration with Fisker was announced in January 2018. Under this agreement Quanergy will integrate five of its S3 LiDAR sensors into the all-new Fisker Emotion electric car.

Figure 9. Quanergy’s IP portfolio and innovation strategy

Challenger: Luminar

The California start-up Luminar was founded in 2012 but operated in stealth mode until earlier in 2017. Since announcing itself to the world, it has been making headlines in technology publications and mainstream media alike. Claims of unprecedented resolution and range in a LiDAR sensing unit combined with a major cost reduction have attracted both investors and industry partners. Last year, the Toyota Research Institute unveiled its newest autonomous test vehicle, powered by Luminar’s sensing technology.

For its new sensor, Luminar has built its own proprietary version of LiDAR from the ground up, keeping the underlying technology under wraps until recently. Since May 2017 a number of pending patent applications and granted patents have been published, providing valuable insight into the technology behind the buzz. Luminar explicitly avoids solid-state technology and uses moving mirrors to steer the beam. Its six granted patents and three pending patent applications focus on the laser source used and on increasing the scanning speed of the entire system. Key to the increased resolution and range is the choice of wavelength: while current automotive LiDAR units rely on the near infrared, Luminar utilises much higher power pulses at longer wavelengths to boost performance. While CEO and co-founder Austin Russell holds patents on augmented reality and 3D imaging, he is not listed as an inventor on any of these LiDAR-related patents and applications.

OEM: Continental Advanced LiDAR Solutions

Continental, one of the world’s largest OEMs which is headquartered in Germany, has been involved in the self-driving car effort from its infancy: in 2007, Continental was part of the winning team in the DARPA Urban Challenge, together with engineers from General Motors (GM), Caterpillar and Carnegie Mellon, while in 2012, Continental was the first automotive supplier to earn approval to test self-driving vehicles on Nevada’s public roads. Fast forward to 2017 when Continental joined the self-driving platform developed by BMW and Intel.

While the company has been involved in automotive sensing for some time, it became an important LiDAR player overnight when it acquired a significant portfolio with the purchase of Advanced Scientific Concepts’ 3D Flash LiDAR business in March 2016. The acquired patents – now assigned to Continental Advanced LiDAR Solutions LLC – complement existing holdings related to radar and camera-based sensing for driver assistance systems and automated driving. Equipped with early priority dates, Continental has been cited by the US Patent and Trademark Office as the reason it has rejected some more recent work by other players in the space.

Most recent LiDAR acquisition: Strobe Inc

Strobe Inc, a three-year-old California-based start-up, is another player developing a low-cost LiDAR solution for the automotive market. It was recently acquired by GM for an undisclosed sum as part of its expanding push into AV, to give GM the expertise to integrate LiDAR units into its perception systems.

Strobe is a spin-off from OEWaves, an established research and commercialisation company which specialises in photonic technologies. OEWaves transferred patents pertaining to tunable lasers and optical resonators dating back to 2007 to Strobe. In addition to acquiring key technology and expertise, the acquisition also provided GM with promising patents covering LiDAR.

The LiDAR path that Strobe is pursuing is based on sending out frequency-modulated laser chirps, which promises high spatial resolution and dynamic range. This allows a vehicle to detect shaded objects adjacent to bright sunshine, a challenge for many existing LiDAR systems. While many of its granted patents broadly focus on laser technology, a recent patent application going back to 2015 discloses a compact LiDAR system which integrates several functions on a single chip.

Silicon Valley versus Detroit

At a high level the LiDAR ecosystem is quite complex, with thousands of patents in core classes and sub-classes. However, a deep analysis of this IP space reveals that only a small sub-set of these patents is actually relevant. The underlying IP portfolios are further differentiated by technology (eg, spinning, mechanically scanning and solid state), filing date and focus (hardware versus processing). This lens allows insight into the IP assets, evolving focus and IP creation process of various players.

Silicon Valley firms such as Google, shaped by the experience of the smartphone wars, tend to be highly IP-centric. Their creation of IP assets is systematic and broad, covering a wide range of product features that will enhance performance and reduce cost. Thought is also given to the creation of IP assets that might possibly become standard-essential patents.

In contrast, automotive firms have surprisingly few LiDAR patents in their portfolios, despite large investments in AV companies or internal AV projects. The traditional automotive model remains in force with car makers relying on their close suppliers for LiDAR and other key innovative technology. Faced with this mandate, the automotive suppliers (eg, Magna, Delphi, Bosch and ZF) have augmented their limited IP holdings germane to LiDAR by partnering with (or investing in) holders of LiDAR technology. In many instances, these partnerships are with younger firms that have small IP portfolios.

Insight from intellectual property

The analysis of the LiDAR patent landscape provides significant insight into the technological and financial forces at work in this area. Viewing the timings of inventions, the intensity of technological efforts, the concentration in specific approaches, the frequency of breakthrough technologies identified in patent applications and grants, the continuity of investment (and results) by specific firms, among other facts, can provide enormous insight. Such detail is obscured by a high-level perspective of an IP space. These insights can help guide company strategists, board members, investors, R&D management and engineers striving to become leaders in LiDAR.

However, LiDAR is but one element (albeit a vital one) in the AV space – it was selected for analysis because it is fundamental as a starting point. However, similar attention should be given to other key elements of AV advances, including telecommunications connectivity, software, vehicle controls and others.