Core 2: Ideal Microscope Interfaces

green_lady — Andrei State drawing of the computer as a lens

We use interactive 3D computer graphics, haptics, and image analysis techniques to map the data and control from scientific instruments’ native coordinate systems and data formats into the 3D space of the specimen, where the scientist most directly perceives and acts.

Combine: Modern experiments often require more than one instrument to measure and control all relevant parameters. We are able to register and remap their multiple outputs into one 3D percept that displays the information from all instruments in their common context. We also map their manipulation capabilities into the same frame of reference, enabling direct viewing, touching, and manipulation of the specimen in the 3D space that is natural to the scientist. This decreases the intellectual effort required to mentally merge these separate streams, thereby increasing the intellectual energy available to the scientist to plan, evaluate, and analyze the experiment at hand.

Augment: Because the desired outcome of much of science is the development of a model that explains what is observed, we also seek to enhance the scientists’ ability to:

fit models to the data (we add image analysis techniques),
imagine the quantitative results predicted by their model (we add computer simulation), and
compare models to experiment data (we seek to display them together and to optimize model fit).

The Advanced Model Fitting and Analysis projects (section 3.2) seek to provide the analysis, model building, simulations, and techniques for combined display required. The present core seeks to integrate the results of that research seamlessly into the instrument (that is, experiment) interface itself.

Projects: The nanoManipulator interface to scanned-probe microscopes (SPMs) has proven itself over many years to be useful in DNA, virus, mucin, cell manipulation, fibrin and other experiments. This success and inherent limitations of SPMs led us to develop an interface to the 3D Force Microscope (3DFM-UI), a novel microscope that combines fluorescence, laser tracking, and magnetic-based force feedback into an imaging and manipulation tool. The success of the 3DFM-UI at enabling experiments to study mucus rheology, DNA extension, cilia mechanics, fibrin clots and other biological systems, and the need to perform orders of magnitude more such experiments, is pushing us to develop a new tool, the High-Throughput Magnetics manipulation environment. This tool will shift our focus from an emphasis on “scientist-in-the-loop” to “image-analysis-in-the-loop” feedback to guide experiments to successful conclusions. The increasing needs of our collaborators to view ever-larger regions of specimens at high resolution and to perform video-based tracking of single and multiple beads moving in 3D is also pushing us to develop Semi-Automatic Image Collection techniques.