Skip to main content

Semi-Automatic Image Collection

Subproject 4.1 Focus-Chasing Microscope

Tracing the path of a bead undergoing cilia-driven flow in lung cell cultures indicates what paths are available to bacteria being swept along in mucus during clearance and whether features such as tumble flow are present. These beads move in X, Y, and Z over a range that is much larger than the focal depth and camera field of view, which requires automatically following a bead that is rapidly going out of focus and off-screen. The details of motion for such a bead indicate how tightly coupled it is to the beat frequency of the cilia at various distances from the cilia. Capturing such information on cilia that beat at 15-30 Hz requires 120Hz video capture coupled with stage motion information.

Rheology experiments in mucus and other biologic materials require collecting movies of individual or groups of beads at different locations in 3D within the specimen. Studies of fibrin clot formation require collecting videos repeatedly over time and in various locations in X, Y or Z (at, for example, areas of different thrombin concentration, or to study clot heterogeneity). The calibration of magnetic fields near pole tips also requires 3D tracking of a number of beads in the presence of varying magnetic fields.

Experiments with the light microscope are always limited by the trade off between magnification, image resolution, field of view and depth of focus. The more resolution, the more limited the field of view and focal depth. The Focus Chasing Microscope will fundamentally address this limitation by allowing the user directly, or a user-defined algorithm, to follow an object of interest outside the field of view in X, Y or Z, while keeping track of the original field of reference for later analysis. This requires the following:

  • Aim 15: Online, Single-Bead Tracking – Track a bead moving in X, Y, and Z over the working range of the positioning stage while recording 120-fps video and position.
  • Aim 16: Online, Multi-Bead Tracking – Track a field of moving beads, time-multiplexing between bead clusters, to capture video at various locations and times during an evolving process.
  • Aim 17: Offline Accurate Tracking – Perform subpixel-accuracy tracking on single or multiple beads, combining the motion platform and video data with Z stacks to recover 3D motion.

Background and Significance: A measure of how in focus a target bead is corresponds to an absolute distance measurement in z from the focal plane. There have been many proposed methods for determining depth in an image from defocus. The following methods could determine focus and thus z position of the target bead: (i) convolution with a known point-spread function; (ii) active illumination of the target (such as the 3DFM laser-tracking system); (iii) and image analysis techniques (sum-modulus-difference (SMD)(Jarvis 1976) and frequency-domain analysis (Horn 1968; Schlag 1983; Nayar 1994; Watanabe 1998)).

The choice was narrowed down to image analysis techniques after rejecting comparison with pre-measured images of a bead at multiple Z offsets (computationally expensive, less robust both to inaccuracies in bead center and poor/different video quality), and active illumination (too complex for experiments with other media around bead, custom hardware, single bead, must be locked on prior to experiment).

Chern et al. (Ng Kuang Chern 2001) surveyed the available image-analysis techniques and compared their performance on a focus-determination task. They cited SMD and Tenengrad edge detection filters as typically performing the best as a measure of focus. SMD was selected because it was computationally quick, robust to aberrations in bead shape/size, and we could exploit our knowledge that beads are roughly radially symmetric to do just a 1D focus measure. It also enables us to normalize the focus measure so we can compare measures taken with different 1D slice lengths.

Preliminary Results: Computer-science graduate student Ryan Schubert began the development of a new application to enable 3D tracking of beads over large areas using 2D video cameras plus motion platforms. Ryan selected SMD as the focus-determination metric and has calibrated its response curve to depth differences in bead images. By coupling this to our existing Mad City Labs 3-axis motion platforms and network-based control system, his system has demonstrated the ability to move the stage to track a single bead as it moves slowly over large ranges in Z. This has also been coupled to the existing Video Spot Tracker technology to enable tracking of beads in X, Y, and Z.

Research Methods: (Aim 15): Our current method of tracking individual beads relies on a combination of Spot Tracking (project 3.2.5) for 2D tracking and an estimation of bead depth by evaluating a normalized SMD as a measure of focus. The SMD measure relates image information to an estimated bead depth. The bead is tracked at a location that is intentionally out of focus so that we can detect which direction (up or down) it moves off of the focal set-point; the orange line indicates the best-fit linear mapping between SMD value and depth. The maximal allowable bead velocity is determined by the trackable focus range (see figure below on the next page). To enable robust bead tracking over extended time periods, we will explore the use of local bead appearance information, going beyond the SMD measure (effectively switching between multiple simple approximations to the focus measure curve), to increase the trackable range and thus the maximally allowable bead velocities. This should be possible, because (at least in bright field microscopy images) beads have a distinct depth-dependent appearance.

SMD Focus technique and capture region
SMD Focus technique and capture region

(Aim 16): The automatic bead finding algorithms developed in Spot Tracking (project 3.2.5) will be used to count the number of beads present at each given location and perform preliminary tracking on the beads. The methods developed in Live Preliminary Analysis (project 2.2.4) will be used on these preliminary tracks to determine when sufficient data has been collected at each location so that new locations can be sought. The region of interest within the specimen will be sampled in a 3D grid, with local sweeps in Z being used at each location to find locations with sufficient beads in the same focal plane.

(Aim 17): While aim 15 requires relatively simple approaches to bead tracking to facilitate real-time implementations, offline bead tracking has no such strict timing requirements, which allows the use of more accurate (and computationally demanding) methods for bead depth estimation. We will employ (as in Aim 15) image appearance information, but use a higher-order approximation to the focus measure/z coordinate curve to allow for more accurate estimations of bead depth.

Risks and Remedies: If the SMD focus measure fails to be accurate enough for some bead specimens, we will use the frequency-domain methods, using available CUDA libraries to accelerate the calculations.


Validation: We will use the algorithms to track in XYZ a set of fixed beads (beads in a medium that does not let them move), using known state motion from the MCL stage. We will then compare the motion estimates (especially in Z) with those reported by the stage.