Multi-particle three-dimensional coordinate estimation in real-time optical manipulation

We have previously shown how stereoscopic images can be obtained in our three-dimensional optical micromanipulation sys-tem [J. S. Dam et al. , Opt. Express 16 , 7244 (2008)]. Here, we present an extension and application of this principle to automatically gather the three-dimensional coordinates for all trapped particles with high tracking range and high reliability without requiring user calibration. Through deconvolving of the red, green, and blue colour planes to correct for bleeding between colour planes, we show that we can extend the system to also utilize green illumination, in addition to the blue and red. Applying the green colour as on-axis illumination yields redundant information for enhanced error correction, which is used to verify the gathered data, resulting in reliable coordinates as well as producing visually attractive images. [DOI: 10.2971/jeos.2009.09045]


INTRODUCTION
A method for gathering true three-dimensional coordinates of particles is of high interest when performing threedimensional (3D) trapping and manipulation of multiple particles simultaneously, whether by holographic trapping [1] or counter-propagating beam trapping [2]- [4].The method may also be implemented in other fields such as microfluidics [5].In a previous paper [6], we showed how red and blue off-axis illumination can be used to make anaglyph stereoscopic images, providing a 3D visualization of the trapped particles.We also showed how the anaglyph images can be analyzed manually on a frame-by-frame basis to deduce the 3D coordinates of trapped objects.Alternative methods for determining three-dimensional coordinates of trapped particles include holographic microscopy [7], or direct observation from the side with an auxiliary microscope [8].The images obtained from holographic microscopy are not visually appealing and hard for the human brain to understand, but it does give accurate 3D coordinates for the trapped particles.On the other hand direct observation from the side gives relatively easy to understand images for the human brain, but it is hard to align and set up for a computer to match corresponding particles from the two viewing angles.Moreover, direct observation from the side requires special square sample chambers and additional optical components.There is also a significant amount of work involved in aligning the two observation planes to observe the same scene.
Whereas red/blue stereoscopic images may look appealing when observed through red/blue stereoscopic glasses, they are not well suited to be combined with the laser safety goggles worn while performing the experiments.Also, ambiguities can occur when pairing red and blue particle shadows, so coordinates cannot be reliably calculated by a computer.To solve these issues and to enhance the precision and reliability of the particle coordinates gathered, we show how utilization of the green colour plane enhances both the visual appearance and allows for automated three-dimensional coordinate estimation.Furthermore, we demonstrate the accuracy and reliability of these coordinates, and demonstrate how they can be used in an experiment to simultaneously verify the trapped positions of multiple particles.The coordinate analysis is also highly valuable in experiments like computer automated assembly [9] of three-dimensional structures, verifying the structures are indeed trapped at the desired positions.

IMPLEMENTATION OF THE GREEN COLOUR
The experimental setup is identical to the one described in [6], where three-dimensionally manipulated microparticles can be illuminated with user-defined colours at different angles.This is done by projecting coloured light patterns to the back aperture of the lower objective, which is coming from an LEDbased DLP projector, Samsung SP-P310ME, as described in our previous work [6].To solve the issues described above and enhance performance, we add green colour illumination along the optical axis to supplement the dual (red and blue) off-axis illumination used previously.This on-axis illumination creates an output image that not only gives exact x-y coordinate information but also facilitates the pairing of matching blue and red shadows, which are expected at equal distances from the green shadow.When an optically trapped particle is simultaneously illuminated with well-balanced blue, green and red light at different angles, the shadows being cast are missing one or more of the three colours, as shown in Figure 1.For example, a particle well out of focus will cast a "green shadow" at its on-axis location.Since only the green colour is missing from this area the shadow will appear magenta.Likewise, the "blue shadow" will be yellow, and the "red shadow" will be cyan.The negative image is easier to comprehend as shown in Figure 1.

Lower microscope objective
Objective back aperture Object plane relayed to camera 1 By projecting red, green, and blue spots to the back aperture of the microscope objective, we get three shadows for an out of focus object.The displacement between the coloured shadows contains the information required to determine the distance from the plane being imaged on the camera.To ease the human comprehension it helps to present a negative image.

CORRECTION OF COLOUR PLANES
Using three colours for illumination is not without challenges.Even viewing a traditional anaglyph stereoscopic image will have issues with signal bleeding [10].Our stereoscopic imaging system uses commercial colour cameras that are designed to mimic the sensitivity of the human eye, so light with just a single wavelength may, in fact give a response in more than one colour plane.This was not an issue in the red/blue illumination patterns used previously [6] since they are sufficiently separated spectrally.They have no cross-signal between the red and blue colour planes, and so the signals are truly independent of each other.However, both red and blue illuminations give a detectable signal in the green colour plane, and green light also makes a difference in the red and blue colour planes.With a little exercise in mathematics, we can correct for this colour bleeding.First, we need to know the background signal, which for our camera is not identical for the colour planes, since the gains are set differently for the individual colour planes in the camera configuration software.The offset can be seen as the sum of two parts.The first part is the dark signal intrinsic on the CCD camera, and the second part is coming from the imperfect contrast of the projector.Even though the projector has a contrast ratio of 1:1000 it still results in a significant offset, because the contrast leak illuminates the full back aperture of the lower objective, thereby giving a weak narrow focus white image on top of the intended signal.For particles even slightly out of focus, this white background is completely homogeneous, and for particles in focus it overlays the coloured image parts.For these reasons, it is safe to treat the contrast bleed as just an extra offset, and thus include it when measuring the offsets.The colour plane bleed ratios are determined from the signals measured in the colour planes when each colour is turned on one after the other.
After having measured the offsets and colour plane bleed ratios, we have all the information necessary to correct for the colour bleeding.We construct a set of linear equations describing how the colour planes mix, and the computer can solve these equations to give the corrected image planes.The set of linear equations for the colour plane mixing may be written in matrix form as where (r, g, b) refers to the true value and (R, G, B) the measured values of the red, green, and blue colour plane intensities, respectively.Retrieving the true colours requires colour correction, which is done by solving for (r, g, b) in Eq. ( 1) to get .
Eq. ( 1) describes the measured colour planes (R, G, B, minus their respective offset levels) as superpositions of the light incident on them.The corrected colour planes in the equation are (r, g, b).The bleed factors (x 12 , x 21 , etc.) are determined by turning on a single colour and observing by what ratio the colour plane in question increases compared to the illuminating colour.For example, we see the blue signal increasing by ∼ 23% of what the green signal increases when turning on green illumination, which means that x 32 in Eq. ( 1) is 0.23.For these colour plane corrections it is critical to use a camera with a linear response profile (γ = 1), or at least correct for it, if is not linear.The measured values for the bleed factors are between −11% and 23%.It is somewhat surprising that a couple of the bleed factors are negative, but they all retain good linearity, irrespective of how much the other colours are giving in signal.We attribute the negative bleed factors to electronic effects within the camera.The offsets and bleed factors are uniform throughout the camera image.Solving the set of linear equations for (r, g, b) from the measured (R, G, B) intensities, gives us the bleed-corrected colour planes.The same equation is applied to all the individual pixels to find the corrected pixel intensities.It is emphasized that the matrix needs only be constructed (and inverted) once for a given camera.Since all the non-diagonal terms are small, only small amounts of noise are added in the correction process.Also mathematically, only one solution exists to the above equation.By utilizing efficient matrix/image calculations in LabVIEW, the correction and subsequent analysis can be performed in real time.
Even if the camera behaviour is not linear, the colour correction could still be done, but it would require careful construction of a look-up table where the camera signals are mapped as function of incident amount of light of the three colours.In that case the inversion process is done by doing a reverse lookup in the constructed table.For good quality cameras this approach should also be able to give unique solutions, although it cannot be mathematically guarantied.
Due to the relatively narrow line widths of the light from the LED based projector, we have much less bleeding between colour planes than would be the case when using a halogen bulb based projector.Performing the same correction with a bulb-based projector would result in a significant decrease of signal-to-noise ratio.Furthermore, the fundamental assumption behind the mathematical colour separation is that we have only three different wavelengths in the illumination, which is a much better approximation for the LED based Samsung projector, as is evident from Figure 2. The bleed between colour planes is ideally determined by the spectral width of the individual colours, and the spectral response of the camera.From Figure 2 it is apparent that the bleeding between colour planes is much less for the LED based projector.Unfortunately, the 3-coloured illumination creates images that can be confusing and hard to interpret for the human mind.
Therefore, we have the option to extract the green colour plane only and use it to render an image in a preferred false colour, making the image presented to the user look exactly like a regular high depth sharpness illumination image.The 3D coordinate analysis routine is uninfluenced by this and is working on the full colour image in the background.
The x, y coordinates for the trapped particles are taken directly from the "centre-of-mass" of the green shadow.The zcoordinate is calculated from the set of nearby red/blue shadows, as described in our previous paper [6], however with the important addition that we can now verify that the midpoint of the red/blue shadows match up with the location of the green shadow.This displacement between the green shadow and the midpoint of the red/blue shadows gives a measure of the uncertainty of the coordinate measurement, and can be used as a discriminator to match the correct blue/red pair with the green shadow.

EXPERIMENTAL RESULTS
To demonstrate the 3D coordinate analysis we have created a real-time video, where we trap and manipulate particles in 3D while simultaneously tracking their individual coordinates (video online).The coordinates are displayed simultaneously with the video in real-time using overlay graphics.A frame from this video as well as the results of image analysis is shown in Figure 3.It should be noticed that the blue shadows consistently appear too far up in the analyzed image in Figure 3.This is caused by the focal plane of the objective being slightly displaced for blue light, thus imaging a few microns above the green and red image planes.The error depends on the choice of objective lenses and can be measured and corrected for.In the present case the only effect is an overestimate of the uncertainty of the measured z-coordinate, and an offset to the measured coordinates.
In practice, the colour correction increases the range over which 6 µm particles can be accurately tracked to over a 180 µm vertical distance.This tracking range is illustrated in Figure 4, which shows a plot of the measured particle position as a function of the real position for a particle resting on a surface that is being raised with a computer-controlled motor.Larger particles can be tracked much further, whereas smaller particles have a reduced tracking range.The range of accurate coordinate tracking is larger than the range of stable trapping.The experimental data presented here were obtained using 50× objectives with an NA of 0.55.The authors expect the system to perform well and give significantly more accurate results in higher NA systems, due to the higher angles of incidence of the coloured light and higher imaging resolution available at high NA.There might be a small trade-off in the tracking range, though, when increasing the angle between the coloured illuminations.
While the tri-colour illumination is similar, in principle, to the dual colour illumination in [6], the redundant information obtained using the on-axis green illumination is a necessity for trustable automatic coordinate gathering.When used with proper colour correction, tri-colour illumination achieves co- ordinate reliability over the whole working range, which is not quite the case if we disable the colour correction feature.
The exact same experiment depicted in Figure 4, when car- the reported motor position.A more reasonable measure may be expected from the deviation between consecutive errors, which is a significantly smaller number and remains low until the limit for stable particle tracking is approached.The reliability of the reported coordinates can be increased further by averaging a few measurements.
ried out without the colour correction, gives unreliable result as shown in Figure 5.It is clear that the colour plane correction is invaluable when it comes to giving reliable coordinate measurements over a large working area.

CONCLUSION
We have shown that by replacing a regular white light source with an inexpensive LED-based projector, we can extend the functionality of our 3D optical micromanipulation workstation to simultaneously provide accurate 3D coordinates for trapped and laser-manipulated objects in real-time.From a user perspective, this coordinate gathering illumination can be completely hidden by only displaying the on-axis illumination colour plane overlaid with the measurement results.
Adding the green colour to the illumination solves some important issues when compared to red/blue stereoscopic imaging.We can now present a completely regular looking im- age to the user, thereby retaining a simple point-and-click user interface for the trapping and manipulation of particles.Furthermore, we can rely on a coordinate analysis routine to match the correct set of shadows, because of the available redundant information.The use of three colour illumination is successful due to the spectral characteristics of the LED based projector and a real-time computerized mathematical separation of colour planes.
In future applications the accurate 3D coordinates, available in real time, open for possible live-feedback 3D optical manipulation, where the trapping beams are dynamically reconfigured to obtain highly reliable volume positioning of a plurality of particles simultaneously.The reliability of the coordinate system can be improved in future work by implementing some form of particle tracking algorithm [11] and use the historical knowledge of particle positions to enhance the sensitivity.

FIG. 2
FIG. 2 Projector spectral emission and camera spectral response.Top: Normalized spectra of blue, green, and red light from an LED based Samsung DLP projector, and an Acer halogen bulb based DLP projector.Bottom: Camera spectral response curve for blue, green, and red light (copied from specifications for the JAI TMC-1405GE camera).Note that the respective spectral widths of the three colours are significantly narrower for the LED based Samsung projector.

FIG. 3
FIG. 3 Three-dimensional coordinate analysis of seven 6 µm particles trapped and rotated in the x-y plane while being simultaneously moved up and down along the zaxis.(a) Extracted frame from the full-colour video data showing particle coordinates (X, Y, Z, ∆Z) in overlay graphics.∆Z is an estimate of the uncertainty of the Z measurement based on the distance between midpoint of the red and blue shadows, and the centre of the green shadow (video size: 1.73MB, format: avi, see Fig3a.avi).(b) Image analysis of a single frame from the video.(c) Greyscale image generated by extracting the green colour plane from the raw image.

FIG. 4
FIG. 4 Measured axial position, Z (in air), as a function of motor position for a tracked 10 µm particle resting on a surface being moved by a motorized stage.This demonstrates a high tracking range, and verifies that the calculated coordinates can be trusted.The right half of the figure shows the difference between measured and motor position.The typical error is less than 1 µm, which already includes errors from

FIG. 5
FIG.5The tracking of 3D coordinates without colour correction, but under otherwise identical conditions to those used to generate Figure4.The range of reliable tracking is significantly reduced, and also a systematic error is introduced when shadows overlap (apparent near -20 µm in this example).The linear fit is based on measurement data from -43 µm to +38 µm.