Image Pre-processing Strengthens and Streamlines Image Processing Systems
Image pre-processing on the frame grabber is a central intermediate step in the image processing process from image acquisition with the camera to processing in the frame grabber and output of results in the host PC. It ensures that image data is optimized for the respective application, transferred securely and without loss of individual images to the subsequent processing to guarantee smooth operation. The aim of image pre-processing is to prepare the data in order to simplify further processing steps and to relieve the load from processing in the host PC.
This includes a better visualization of the images in order to emphasize certain properties or structures that are too weakly pronounced or that are important for further processing (delimiting areas, segmentation and feature extraction). This is usually based on the entire raw image. This also includes the automatic selection of image areas (Region of Interest, ROI) according to image content and the JPEG image compression - more about this later.
The processing steps used in pre-processing always depend on the existing image quality of the camera sensor. The steps are executed on the FPGA processor of the frame grabber, and also on the other vision hardware such as the camera itself, the vision sensor or embedded system. On these vision devices, VisualApplets, the easy-to-use graphical development environment can be used to implement extensive pre-processing as well as a complete image processing solution on the FPGA, thus realizing a real-time capable system.
Pre-processing in the Camera
For camera interfaces such as USB3 Vision and GigE Vision, which do not require a frame grabber, or for embedded systems, pre-processing takes place directly in the camera and sensor. It is always necessary when camera interfaces such as GigE Vision or the computer interface only transfer limited amounts of data or when using embedded systems that offer less computing power. By reducing the amount of data already in the camera, the duration for transmission and analysis can be shortened, thus increasing the throughput of the application.
Basler cameras already offer a basic set of pre-processing features such as debayering, color anti-aliasing, image sharpening, and noise reduction. Among other things, it greatly improves the brilliance, detail and sharpness of the image, while simultaneously reducing noise. If the pre-processing is extended by programmable frame grabbers, complex calculations such as filter operations and color space conversions can be carried out, thus relieving the CPU.
Frame Grabbers Extend the Pre-processing
A frame grabber is necessary for many camera interfaces such as CoaXPress, Camera Link and Camera Link HS and can be indispensable for real-time requirements or large data volumes. It enables a much wider range of pre-processing directly on the FPGA. If the image processing system is set up with standard frame grabbers, pre-processing such as high-quality debayering, look-up tables (LUT) and mirroring are usually included, depending on the application. With programmable frame grabbers the possibilities are even more versatile:
Dead pixel compensation
White balancing
Shading correction
Flat field correction
Remove blurred areas
Reduce noise in the image, by averaging or using different filters, e.g. smoothing filters
High dynamic range (HDR) to compensate for areas of the image that are too bright or too dark
Geometric rectification to normalize the shapes of structures, e.g. by affine transformations
Increase contrast, e.g. by using Look-Up-Tables (LUT) already during image acquisition as well as histogram flattening or stretching
Color space conversion
Filtering
Such a variety of pre-processing can also be realized on all VisualApplets compatible vision devices, to varying degrees depending on the computing power. This makes cameras and other vision hardware real-time capable, which opens up new fields of application. With VisualApplets, no special knowledge of circuitry and timing in the FPGA is required for implementation. It can therefore be realized by application or software engineers.
Download Visual Applets
Blob Analysis
The development possibilities with VisualApplets go far beyond image pre-processing, the software can be used to solve countless image processing tasks, such as blob analysis. It separates related pixel areas as separate objects from each other and from the background (segmentation) and assigns properties such as area size, contour length and bounding box coordinates to each object (classification). If this is already done during pre-processing, a large part of the image segmentation is shifted there even before the data is saved for the first time.
Reduce Data Volume via JPEG Compression
The bandwidths at which image data is transferred from the camera to the frame grabber have continuously increased in recent years due to faster and larger sensors and color processing, for example to 5 GBps for CXP-12 with four links. In order to process them, the amount of image data can be greatly reduced by JPEG compression. In laser welding in the automotive industry, for example, the data throughput could be reduced from 100 MBps (input bandwidth of frame grabbers) to about 5 MBps. On the other hand, the frame rate can be significantly increased by reducing the amount of data required to transmit a single image. This is particularly useful in applications where JPEG compression is performed directly in the camera, allowing the transmission path to be used more efficiently as a bottleneck.
Control of the Actuators and Triggering
Frame grabbers use signal processing to control camera triggering, lighting and actuators as well as the evaluation of sensors via the digital I/O interfaces. Complex control loops and combinations with pre-processing are also possible. For example, an exposure measurement can be carried out by means of image pre-processing in order to carry out a corrected control of the illumination or triggering of the camera.
Robust, Accelerated and Efficient Image Processing
All in all, image pre-processing ensures to accelerate computationally intensive algorithms and to improve application performance. Furthermore, it ensures to process image data with very high resolution and bandwidth efficiently by increasing the data throughput in order to enhance the accuracy and robustness of image analysis. The complete bandwidth of a high-speed camera can be safely transferred while relieving the CPU of the host PC and subsequent analysis software.
Using VisualApplets, image pre-processing on the FPGA can be realized with little effort by end users and without hardware programming knowledge. This enables them to flexibly adapt the functional range of cameras, frame grabbers and other vision devices in a short period of time when application requirements change without having to install a new camera.
Advantages at a Glance
Use simpler camera interfaces
Increase of the data throughput
Cost-efficient system design through low-cost interfaces (GigE)
Acceleration of computationally intensive algorithms
Increase application performance
Real-time image processing enables new applications
Use smaller processors with less computing power
Optimized neural networks
Saving IPCs
Reduced CPU load
Increase accuracy and robustness of image analysis
Bypass in-house developments
Application Examples to Efficiently Solve Image Processing Tasks with Best Possible Results
A common application is color reconstruction (debayering) and color space and bit width conversion, which allows the raw images from the camera to be transferred at the full speed of the camera interface, thus enabling maximum bandwidths. The frame grabber can thus provide the host PC with a processed image.
Applications such as high dynamic range (HDR), JPEG compression, automatic ROI selection or 3D laser triangulation effectively reduce data volumes already in the pre-processing stage.
Illumination and image distortion corrections increase the image quality and thus simplify the analysis in later processing steps.
Object segmentation and feature extraction (blob, center of gravity, angle, eccentricity, image moments, etc.) provide further processing with properties for classification.
In print inspection, it is essential to achieve highly precise position recognition and alignment of the images obtained in order to enable comparison with the master. Subpixel accurate geometric distortion corrections are used with the aim of keeping color artifacts low. The automatic position detection is aligned to edge or image features.
