Named as one of CB Insights
100 Most Promising AI Startups 2023
In the era of smartphones and digital cameras, we often capture stunning images effortlessly without diving into the technicalities. However, behind the scenes (or behind the lens of any image sensor), there is a tiny rectangle of silicon that actually captures the image.
Compared to the human eye, image sensors have very different color and brightness responses, and they also add some undesirable distortion. Every single sensor therefore, requires a component called an Image Signal Processor (ISP) to correct and enhance the image. The performance of this can be more critical to the quality of the image than the choice of hardware of the actual sensor or lens.
Image signal processing plays a crucial role in transforming the raw sensor data into visually appealing and captivating images. Here, we’ll embark on a journey to demystify the ISP pipeline and explore the intricacies of each stage.
The ISP pipeline begins with the capture of raw data from the camera sensor. The sensor measures the intensity of light hitting each pixel and generates a grayscale image. The raw data contains various imperfections like noise, lens distortion, and color inaccuracies, which need to be addressed for optimal image quality.
Most camera sensors use a Bayer filter, which consists of alternating red, green, and blue color filters over each pixel. The demosaicing stage interpolates the missing color information for each pixel by utilizing neighboring pixels' values. This process results in a full-color image, where each pixel now contains red, green, and blue color components.
The captured image often suffers from color inaccuracies due to factors such as the sensor's spectral sensitivity and lighting conditions. Color correction techniques, such as white balance and color matrix transformations, are applied to adjust the image's color rendition. White balance ensures that white objects appear neutral under different lighting conditions, while the color matrix accurately transforms the color information to match the desired output.
White balance in image processing is vital to produce accurate and visually pleasing images by compensating for the color temperature of the light source and eliminating unwanted color casts. It ensures that the colors in the image appear as close to the original scene as possible, enhancing the overall quality and realism of the photograph.
Human perception of brightness is nonlinear, meaning that equal changes in pixel values appear differently based on their original intensity. Gamma correction is applied to adjust the brightness levels and improve the image's overall contrast. It maps the linear pixel values to a nonlinear gamma curve, enhancing the details in both dark and bright regions of the image.
Denoising is one of the critical functions of an ISP. Image noise is a phenomenon that is particularly noticeable in low light and is often seen as speckles or blurs in an image. Image sensors introduce various types of noise. Shot noise is the random nature of the quantum interactions of photons with the atoms of the sensor. Dark current noise is caused by thermally generated electronics. And quantisation noise is the result of inaccuracies in the process of ‘counting’ the tiny electrical charges that are generated by the incoming photons.
Noise reduction algorithms are employed to suppress these imperfections while preserving important image details. Techniques like spatial filtering, temporal noise reduction, and advanced denoising algorithms are applied to minimize noise artifacts.
Noise reduction is one area where the performance of traditional hardware based ISPs is limited, and Visionary.ai’s AI-based approach improves performance in low light conditions, in real-time.
It is possible that a small number of pixels in the sensor will not work correctly, most likely due to a fault in the manufacturing process. Through monitoring the output of a sensor over many frames it is possible to identify pixels that are “always-on” (i.e. are detecting light when there is none) or “dead pixels” (ie not detecting light when they should). Suspect pixel mapping can identify such cases and replace their outputs with information from surrounding pixels
The image signal processor is closely coupled with the sensor. It takes raw image data out, as we have seen, but it also provides feedback to control the exposure. When the ISP detects that an image is too dark, it can adjust the exposure time. Note that increasing exposure time comes with some downsides, as long exposure can lead to motion blur.
To enhance the image's sharpness and bring out fine details, sharpening algorithms are applied selectively. These algorithms work by enhancing high-frequency components of the image while preserving low-frequency information. Techniques like unsharp masking, edge enhancement, and deconvolution play a vital role in making images appear crisper and more appealing.
The center of images are often brighter than the corners due to an effect called lens shading. This is a byproduct of the optical design of the camera and lens. The ISP can be tuned to compensate for this effect.
Cameras often capture a wider dynamic range than can be displayed on typical screens. Tone mapping and dynamic range compression techniques are used to compress the image's dynamic range, ensuring that both shadow and highlight details are preserved while maintaining a visually pleasing appearance. This stage is particularly important when capturing high-dynamic-range (HDR) images.
The final stage of the ISP pipeline involves various image enhancement techniques to further improve the visual quality. These techniques can include contrast adjustments, saturation enhancements, local tone mapping, and artistic filters, allowing photographers to achieve their desired aesthetic preferences.
This final stage involves compressing the image data to reduce its size for storage or transmission. There are different amounts of compression applied, but in almost all cases, the raw input data from the image sensor contains a high bit-depth (often 12 to as high as 24 bit) which is often reduced to 8 bits in the output signal.
The ISP pipeline is a complex process that breathes life into the raw data captured by camera sensors. Each stage of the ISP pipeline contributes to the transformation of the image, addressing challenges like noise, color inaccuracies, and dynamic range limitations. By understanding the intricacies of the ISP pipeline, we gain a deeper appreciation for the technology behind the captivating images we capture every day.