The relevant information in many vision applications is encoded into the color of the scenery. This information in normal color cameras is extracted based on the three standard color channels; red, green and blue (RGB), respectively. This color reproduction technique is an approximation and it is often insufficient to reliably solve a given machine vision problem. Hyperspectral imaging overcomes this limitation by providing a greater number of spectral bands, while maintaining an adequate spatial resolution. The required narrow-band spectral filters can be implemented at the sensor level owing to the recent advances in sensor design. Hyperspectral cameras equipped with these sensors show a significantly decreased level of complexity. This enables compact, reliable, and easy-to-use hyperspectral cameras that can benefit virtually any application in which accurate color is key to success.
Color is one of the key parameters in many vision applications and it is often used as a basis for classification, alternate background and foreground discrimination, or object identification. Typically, color cameras are equipped with three broadband color channels, red, green, and blue (RGB). These channels are implemented in the form of a regular, mosaic-like filter pattern; the so-called Bayer-pattern. With only these three standard filters, the resulting color information is merely approximate and often insufficient to reliably identify subtle color gradients. However, robust color discrimination is often the key to success (e.g., the discrimination between tissue, nerves and blood vessels during non-invasive surgery) and therefore the performance of conventional color sensors hampers vision-based solutions in many applications.
In contrast, hyperspectral imaging, or imaging spectroscopy, is a combination of digital imaging and narrow-band spectroscopy. This technique allows the light intensity to be captured for each pixel on the detector for a greater number of spectral bands (typically some tens to hundreds). Consequently, each pixel in the image contains a full color spectrum (in contrast to restricting three values for red, green and blue), which can characterize the scenery with greater color, detail, and accuracy. This feature enables object classification pipelines based on spectral properties, via statistical matching or neural networks; thereby enhancing entirely new approaches in the vision industry.
The recent progress in sensor design and processing speed means that a wide field of applications now can benefit from hyperspectral imaging. The applications include industrial parts inspection, specimen classification in medicine and biophysics, and, airborne remote sensing and military target detection. This article explains the two most common operating principles of hyperspectral cameras, and highlights the main applications in which they are typically used.
Principles of Hyperspectral Cameras
The spectral decomposition of light is traditionally achieved with a narrow slit in combination with dispersive optical elements. This technique allows high spectral accuracy, but results in an elaborate and complex optical setup inside the camera. This can lead to a large camera footprint, unreliable performance, and increased costs.
In recent years, the advances in sensor design have enabled the implementation of precisely tuned narrow-band spectral filters at the pixel-level. While conventional color sensors have filter patterns with only three distinct broadband color channels, hyperspectral sensors have filter patterns that sample the full spectral range with a greater number of evenly distributed narrow band filters. Depending on the application, this range can span from the ultra-violet to the near-infrared spectrum, and might be significantly beyond the perception of the human eye. Hyperspectral cameras can be divided into two main categories, based on the characteristics of the filter pattern and the resulting operating modes: snapshot mosaic and pushbroom scanning. Both these modes impose different requirements on the application setup.
Pushbroom Scanning
The relative motion between the camera and the imaged object is often a natural requirement in many vision applications. Typical examples include parts inspection on conveyor belts, remote sensing via aircraft or satellites, or autonomous agriculture via unmanned ground vehicles. These kinds of applications are best addressed with pushbroom-scanning hyperspectral cameras, in which contiguous pixel rows of the image sensor are coated with spectrally adjacent narrowband filters. The relative motion between the camera and the scenery causes the object to effectively drift over the image sensor. By synchronizing the sensor line read-out to the relative motion speed, the scenery is imaged line by line; or, due to the row-wise filter coating of the sensor, one spectral band after the other. The full spectrum is obtained after the object has passed completely over the sensor. Therefore, on a two-dimensional area sensor, the pixel rows image one spatial dimension, while the columns capture the spectral dimension. The second spatial dimension is obtained from the relative motion of the camera and the scenery; termed the pushbroom scan. This working principle is illustrated in Figure 1.
Typically, the number of spectral bands in pushbroom-scanning hyperspectral cameras is on the order of one hundred. This feature yields very detailed spectral information, which in turn enables more reliable identification and classification results. At the same time, the maximum spatial resolution is very high, and is determined by the raw resolution of the sensor along one dimension (typically 2048 − 4096 pixels), and the scanning speed along the other dimension. It is important to note that the high spectral and spatial resolutions come at the expense of the scanning requirements, and can potentially lead to more complicated application setups. The scanning requirements, however, are often intrinsic to the application, and therefore this is not considered to be a general disadvantage.
Snapshot Mosaic
Snapshot mosaic hyperspectral cameras are very similar to standard color cameras. The filter coating is arranged as a mosaic of repetitive tiles, but, contrary to the 2 × 2 Bayer pattern, typically these tiles consist of 4 × 4 or 5 × 5 pixels. The individual pixels in each of these tiles are coated with narrowly-defined bandpass filters (compare with Figure 2) Therefore, the number of spectral bands is significantly increased compared to the traditional red, green, and blue color channels. It is important to note that this gain in spectral information is accompanied with a decrease in spatial resolution, which results from the large size of the individual tiles in the filter mosaic.
Figure 1: Outline of the CMOSIS CMV2000 sensor with a narrow band filter pattern aligned with the sensor rows. Each spectral band covers an area given by the full width (2048 pixels) multiplied by 8 rows. The spectral range from 600nm to 1000nm is sampled with 100 spectral bands. The modified sensor is provided by IMEC and is implemented by a number of camera manufacturers. Image credits: XIMEA https://www. ximea.com/support/attachments/4675/XIMEA_imec_HSI_technology-Part-V1.1.pdf
Figure 2: Outline of the CMOSIS CMV2000 sensor with repetitive 4×4 pixels tiles. In each tile, the full spectral range between 465nm and 630nm is sampled with 16 spectral bands. The modified sensor is provided by IMEC and is implemented by a number of camera manufacturers. Image credits: XIMEA https://www.ximea.com/ support/attachments/4675/XIMEA_imec_HSI_technology-Part-V1.1.pdf
Typically, the resulting raw resolutions are of the order of 500 × 250 pixels, but can be increased with sophisticated interpolation algorithms.
The complete spatial and spectral information can be obtained in one snapshot, as implied by the name, and for this reason snapshot mosaic hyperspectral cameras can be used for conventional video acquisition, or other applications where scanning is not applicable. Consequently, snapshot mosaic hyperspectral cameras are very versatile and can be easily integrated into virtually any application, as a substitute for conventional color cameras. These applications include quality inspection, food sorting, tissue analysis, endoscopy, and microscopy. The only drawback of this “ease-of-use” is the limited number of spectral bands of approximately 20 compared to over a 100 with pushbroom-scanning cameras. However, often this is still sufficient to address imaging problems that cannot be solved with normal color cameras.
Further considerations
Both the sensor designs explained above do not put any special constraints on employed lenses, other than a high transmission and low chromatic aberration over the spectral range of interest. Consequently, cameras with hyperspectral image sensors can be readily equipped with existing professional-grade machine vision lenses.
The output of hyperspectral cameras comes in the form of 3D data cubes, with two spatial and one spectral dimension, i.e., a full spectrum for each pixel. The concept of this type of data cube is depicted in Figure 3, where x and y represent the well-known spatial dimensions of the image and the vertically arranged λ1..n represent the n spectral bands. Note that a significant amount of image post processing is required to transform the raw information into data that can be further implemented for object identification or classification.
Conclusion
It is now possible to implement narrow-band spectral filters at the pixel-level with semiconductor thin-film processing. Hyperspectral cameras using this technology can be implemented as reliable, compact, and easy-to-use systems that can be integrated into many different applications. These applications can range from precision agriculture supported by unmanned vehicles, to robust discrimination between tissue, nerves, and blood vessels during non-invasive surgery. In addition, this technology can also significantly improve food sorting or quality inspection, by providing more detailed and accurate spectral data than conventional color sensors. Specifically, when combined with powerful computing approaches like neural networks, capable of analyzing and extracting the desired information from vast amounts of raw data, hyperspectral cameras will enhance virtually all applications in which the color of the object plays a crucial role.
