Pixel Perfect

A real-time image processing system for biology

t 1Scientific visioning systems often rely upon pixel-perfect precision to produce meaningful data. Cutting-edge equipment used in the study of cell signaling is no exception; proper image alignment is critical for successful experiments. Biologists at Pacific Northwest National Laboratory (PNNL) put together a special multi-spectral confocal microscope that was capable of acquiring live images of cells and proteins using two simultaneous spectral channels. But there was a problem: the dual images resembled poorly registered Sunday comics and were of limited use. This article describes how the biologists worked with programmers to fix the difficulty and make the microscope a truly useful and unique instrument.
The problem
The study of cell signaling involves monitoring the process by which cells send signals to neighboring cells, as well as their own responses to signals that bring about various changes. These types of discoveries help to improve the understanding of live cell behavior - an important factor in comprehending organism well-being, especially as its cells undergo exposure to environmental toxins, disease, temperature changes and everyday work.

Knowing the presence and behavior of these signaling molecules is rather tricky. One approach uses fluorescence imaging techniques that involve the attachment of a fluorophore, a special dye, to specific proteins or molecules of interest. When such a sample is excited by a laser operating at the fluorophore's excitation wavelength, the fluorophore emits a specific wavelength of light. These emissions pass through an optical bandpass filter before reaching a CCD camera. In the resulting image, where each pixel represents a 0.16 µm square in the sample, the marked molecules appear as small, bright dots.

Multi-spectral confocal microscopes are set up to produce multiple images from a sample, where each image represents a certain slice of the visible spectrum. This makes it possible to infuse a sample with multiple fluorophores, one for each slice of the visible spectrum supported, and observe them either independently or together as a colored composition (Figure 1). Each fluorophore can be attached to a desired type of protein and ultimately react at its unique wavelength. It follows, then, that two different proteins in close proximity should cause an excitation to appear in the same locations on both spectral images. Furthermore, techniques such as frequency resonance energy transfer (FRET) can be performed on the spectral images to test for protein interactions.

Figure 1: A series of multi-spectral images acquired at different focal planes on the confocal microscope are reconstructed in this 3-D composite image. 
Figure 2: This dual-channel acquisition of biological structures from an octopus is severely affected by camera misalignment. 

This procedure may seem straightforward, but is readily thwarted by alignment issues between the two cameras (Figure 2). The path that must be taken by the fluorescent emissions from the sample to each CCD camera involves a combination of lenses, mirrors and filters. Subtle temperature changes within the laboratory (not to mention the occasional accidental bump) can significantly alter the alignment of the two cameras. In addition, imperfections in the image path may cause significant, nonlinear distortions. Although a beam-splitting device is employed (Optical Insights Dual-Cam) that features convenient knobs for roughly aligning the two cameras, this system alone does not provide the amount of precision that is ultimately required. The success of an experiment hinges entirely upon the assumption that the pixels produced by the pair of cameras are truly dead-on.

This image registration problem was the foremost challenge in a larger project to create a practical imaging system for the dual camera confocal microscope, as shown in
click the image to enlarge 

Figure 3: The control system coordinates image acquisition, processing and data archiving, as well as instrument control.
Figure 3. This system was intended to facilitate high-speed image capture and processing, involving pairs of 800x600 images taken at a rate of 15 frames per second — 14.4 million pixels per second.

The new design
Early in the project, the team of biologists and programmers had to make fundamental architectural decisions to determine what sort of platform and framework would best support real-time image processing, instrument control and data archiving. Although a very limited set of third-party software solutions existed for dual camera capture, questions abounded concerning how to do custom image processing in a potentially proprietary environment.

The biologists had grown weary of serious bugs and user interface issues, lack of configurability and high maintenance costs in their existing third-party solution. The decision was made to develop a new application in-house with a comprehensive interface for running the microscope. The resulting application replaced all the functionality of the old solution and opened many opportunities for advanced image processing — all while being more favorable to the biologists.

The application (Figure 4) consists of a user interface implemented in Microsoft Visual Basic 6.0, holding together a series of control subsystems that communicate with
click the image to enlarge 

Figure 4: Microscope control application 
interface hardware from National Instruments. The application has controls for setting the laser wavelength and amplitude, three-axis stage positioning, image capture settings, and incorporates the Sax Basic Engine v6, allowing the user to write Visual Basic-like scripts to automate all aspects of the application.

With assistance from signal processing researchers from Utah State University (USU), the team also mapped out the approach for doing the image processing, including the image registration. It was critical that the image processing subsystem would be able to keep up with the incoming images from the cameras, but it was also important to leave room for future image processing steps without impacting the overall throughput.

The team chose to pursue field programmable gate array (FPGA) technology for its speed, parallelism and ability to be integrated with software. By implementing the image processing system in hardware, processes could be pipelined, and both latency and total throughput could be guaranteed to remain at a constant value. It also was important to realize that, while the upfront development costs for the FPGA were significant (over $25,000), the final production system could be run on inexpensive parts costing less than $250 that could potentially be integrated directly into the cameras or in the physical data path.

The development environment chosen consists of the Celoxica DK suite (an environment that supports Handel-C, an ANSI C-like programming language, and its associated memory and data transfer libraries) and an Alpha Data ADM-XRC-II board with a six million gate Xilinx Virtex-II FPGA and six banks of ZBT-RAM. In particular, the presence of multiple banks of memory made this board a good choice for image processing applications.

All that remained to complete the package was a software component to tie the cameras into the FPGA image processing system and then deliver the results through the control application to the user. While speed requirements dictated that it be written in a fast language, it also had to be closely integrated with the user interface of the Visual Basic control application. Moreover, developer requirements warranted a separate development path from the rest of the project.

The solution was to design the imaging subsystem as an ActiveX control that could be positioned within a VB form and readily accessed. This control was written in C++ and communicates with the Coreco IFC imaging library, which retrieves incoming images from the pair of PC-DIG capture boards, controls the configuration and data transfer to and from the FPGA, and then makes the processed data available to the control application. The control has a number of methods for configuring the image subsystem, including setting parameters for image registration and setting display preferences for the live image display.
Getting the pixels in line
The stage was now set for aligning the images with each other. That functionality was destined to reside on the FPGA, along with the frame averager and ratiometric color mapper, as well as future filtering, convolution and blob analysis and characterization algorithms. The registration system was to be implemented through the use of a cubic warp with bilinear sampling. Initially the programmers intended to characterize and remove all nonlinear imperfections from both images and then automatically bring them into registration through translation and rotation. After much work, they grudgingly confirmed that this approach was not exact enough. Many portions of the images were displaced by four pixels or more.

As the programmers wrestled with trying to better characterize the imperfections, the biologists conveyed that the importance of image registration did not lie within the idea of making the images straight; rather, it was only important to ensure that pixels were aligned with each other. Therefore, it was now necessary to warp just one image so that it lined up with the other. However, this process called for a new kind of calibration procedure.

The established approach for discovering the polynomial coefficients used in the cubic warping procedure was to use a well-chosen calibration slide to locate coincident points within a pair of images from the two cameras. The observed offsets between the coincident points could then be fed into a least-squares solver. While the biologists had several existing slides that could possibly work, a simpler and much more effective calibration pattern already existed within the system and did not require the use of any slides at all.

The microscope's Nipkow disk is a wheel that is perforated with many holes, each containing a tiny prism. As this wheel is rapidly spun, the incoming laser light is deflected to each pixel location within the sample in a raster scan fashion. When the wheel is stopped and incandescent light is shined through, an excellent calibration pattern is projected to the cameras: a screenful of 500 or so high-contrast dots.

The programmers worked to characterize the dots, each of which consisted of approximately 15 pixels. The controls of the beam splitter were used to mechanically align the images as precisely as possible. Then, the center of each dot was located using flood-fill blob analysis, and the corresponding centers from each image were compared against one another in order to find the offset values. This significantly large set of offset values allowed the least-squares solver to generate excellent coefficients, and made it possible to disregard the few dots that were cut off along edges or were poorly represented in one of the images without affecting the quality of the outcome.

The remaining task was to create the real-time cubic image warper in the FPGA hardware. Although reconfigurable computing technology is exciting and has great potential, designers must still be very conscious of the number of gates consumed by a particular algorithm. Unfortunately, the direct evaluation of the cubic warp calls for the evaluation of two polynomials: 30 floating-point multiplications and 14 additions per pixel — a prohibitive proposition for the available resources. Thanks to the fact that the cubic warp can easily be evaluated in raster scan fashion, evaluating each pixel successively, a mathematical trick known as forward differencing reduces the number of required calculations to a very acceptable six additions per pixel, which can be performed in parallel to further speed up the computation rate.

High image quality was retained in the FPGA cubic warp implementation by using the fractional portion of the fixed-point numbers used in the forward differencing process to sub-sample pixels from the source image. The multiple banks of memory on the FPGA board made it possible to perform four reads per pixel, thus achieving fast bilinear interpolation and ensuring a smooth finish on the warped image, rather than the jagged aliasing resulting from simpler nearest-neighbor sampling.

The outcome of all of this work is a significant advancement in biological imaging. The precisely registered images (as seen in Figure 5) enable the reliable use of FRET




Figure 5: Although these unregistered (a) and registered (b) dual channel composites of octopus samples may contain subtle visual differences, the misalignment produces erroneous features when the unregistered images undergo a ratiometric calculation (c). Note the absence of the extra "ripples" (d), thanks to proper registration. 
techniques for the study of protein-protein colocalization and the real-time image processing performance allows computed images to be displayed to scientists immediately — a factor that is essential for unencumbered experimentation. These traits, coupled with the inherent expandability of the system, make what was once an impractical instrument an elegant and useful tool for performing FRET analyses.

While the work accomplished so far has opened doors to new types of experimentation, many opportunities still exist to transform the current software and hardware into a production system. The team seeks to finalize the FPGA image processing algorithms and migrate them to inexpensive FPGA chips located in the cameras or on the physical data path. It is likely that such self-contained image processing modules can be used on other camera imaging systems within other parts of the laboratory and beyond.
t 1The authors gratefully acknowledge William Chrisler for contributing images that appear in this manuscript. The research described in this paper was conducted under the Biomolecular Systems Initiative under the Laboratory Directed Research and Development Program at the Pacific Northwest National Laboratory (PNNL), a multi-program national laboratory operated by Battelle for the U.S. Department of Energy under Contract DE-AC05-76RLO1830. Part of this work was performed at the W. R. Wiley Environmental Molecular Sciences Laboratory, a national scientific user facility sponsored by the U. S. Department of Energy's Office of Biological and Environmental Research and located at Pacific Northwest National Laboratory.

Kenneth Perrine is a scientist in Computational & Information Sciences; Derek Hopkins and Brian LaMarche are scientists at the Environmental Molecular Sciences Laboratory; and Marianne Sowa, Ph.D., is a scientist in Chemical Structure & Dynamics, all at the Pacific Northwest National Laboratory. Scott Budge, Ph.D., is an Associate Professor in the Department of Electrical and Computer Engineering at Utah State University. The authors may be contacted at