Patent analytics through reverse image search engines: tools and application

In our previous report we briefly introduced patent searching and outlined its importance to businesses. We also introduced the concept of reverse image searching. The following report is a continuation of these themes and provides a closer look at reverse image search tools and their application to the domain of patent searching and analytics.

Prior art searching – methodology
As a preliminary step, searchers form a general understanding of an invention using images and descriptive text. When identifying prior art, analysts establish the relevancy of the patents through image-based contextual analysis. This approach generally applies to inventions relating to mechanical and allied domains. Identifying an image that is visually similar to an invention improves search relevance and reduces the effort and time required to identify relevant results. However, manual analysis is still required to establish the relevance of these results. For trademark and design patent searches, a combination of keywords and classes is used to shortlist related documents based on their images.

Reverse image search tools
Users simply upload a picture and the search engine recovers similar images – everything a user could wish to know through the use of a single picture. Patent analysts would agree that trademark and design searches can be made considerably simpler using reverse image search tools.

Search engines such as Google, Bing and Yandex achieve this by evaluating the submitted picture and creating a calculated model of it using advanced procedures. This is then matched with other photos or images in the Google, Bing or Yandex databases before returning matches and similar results. When available, Google also utilises an image's metadata (eg, its description).

Brief history of reverse image search tools
Google Images was introduced in 2001 and has since helped millions of users to find appropriate image results using keywords. However, this service is unknown to approximately 70% of users, which could be the result of a lack of publicity. Moreover, as relatively few people find them necessary, these tools are continually overlooked.

Some of the best reverse image search engines are as follows:

• Google Images – a widely used website to search images. Introduced 15 years ago, Google Images has the largest image database compared to other websites. In June 2011 Google Images introduced its reverse image search feature, which uses algorithms based on attributes like shape, size, colour and resolution to find similar images. Google Images is free to use, there is no limit on file size or type, and because it has the largest number of indexed images, the possibility of finding a match is extremely high.
• TinEye – a product of Idee Inc, a Toronto-based company. TinEye is claimed to have been the first website to use image identification technology. It was the first image search engine on the Internet to use image identification technology rather than keywords, metadata or watermarks. Its tagline is: “Search by image and find where that image appears online.” TinEye supports JPEG, PNG and GIF images, and the upload size is limited to 20 megabytes. TinEye does not recognise people or objects in pictures – it recognises the picture as a whole. While the paid version of TinEye is recommended for photographers and digital media professionals, the free version works perfectly well. TinEye Lab also features a multi-colour engine that extracts colour from creative commons images (ie, from Flickr), which makes images searchable by colour. Idée, Inc was founded by Leila Boujnane and Paul Bloore in 1999. Idée launched the service on May 6 2008 and went into open beta in August that year. As of October 2015, TinEye's search results claim to have over 13 billion images indexed for comparison.
• Bing Image Match – a reverse photo search tool that was added to Microsoft’s search engine, Bing, in March 2014.
• Yandex – Russia’s largest search engine. It contains a reverse photo search tool and users can filter search results by file size (large, medium and small). Yandex developed this search engine to track duplicate images. It is a hassle-free reverse image search engine that does not require signing up, is free to use and works smoothly.
• Pinterest’s visual search tool – announced in November 2015, this tool allows users to search for visually similar images or ‘pins’. Unlike other reverse image search tools, users can zoom in on an image, drag the zoom tool over a specific part and search for it. The instant search results are visually similar to only the part selected by the user.

Since 2001 many companies have ventured into reverse image search tools. Examples of these tools are listed below – however, some of them have been discontinued due to a lack of users:

• Image Identify by Wolfram;
• Image Raider;
• Baidu;
• Karma Decay;
• Tiltomo (a Flickr-based tool);
• Byo Image Search (also maintained by Ideeinc.com);
• Live.com;
• Xcavator;
• Incogna;
• Berify; and
• Terragalleria.

With myriad reverse image search engines available globally, inventors, professional searchers and patent examiners have a tough choice when deciding which tool to use. Arguably, for patent searchers, Google Images is the best tool for finding visually similar images, while TinEye and Pinterest’s visual search tool are recommended for digital media professionals and photographers.

Reverse image search tools function in various ways: some are simple enough to allow users to drag and drop images for searching, while others request users to provide a URL of the image. Moreover, some allow users to search for tags and keywords which further define the image. However, the use of these tools has yet to receive global acceptance in patent searching, which may be due to the lack of awareness or lack of appropriate search algorithms used in these tools. Our next report will examine how to use a few of these tools.