Frequently asked questions (FAQ)

What is an image registration algorithm?

The goal of image registration is to match different sets of data into one coordinate system. Two images (a reference image I_ref and an image to register I) are matched in order, for example to compare or combine their information.

What is an optical flow algorithm? What does it do?

An optical flow algorithm is an image registration technique which aims at estimating a dense velocity field assuming an apparent grey level intensity conservation during displacement. In other words, each spatio-temporal variation of grey level intensity is attributed to "motion". However, with a problem stated like that, an infinite number of velocity fields may potentially be a solution: any voxel-matching from I_ref to I is a solution as long as the paired voxels have similar grey level intensities. An additional constraint is thus required. The method originally proposed by Horn and Schunck introduced an additional "physical constraint" by assuming that the motion field is smooth in the neighborhood of each estimation point.

The reader is referred to the question "How do optical flow algorithms work?" located below for more details about the optimized process implemented in RealTITracker.

What are the possible applications in the field of real-time image-based therapy guidance?

Recent developments in medical imaging systems, associated with fast data processing strategies, now allow acquiring functional and positional information in real-time during an interventional procedure. Dynamic MRI or Echography are thereby promising candidates to assess an on-line retroactive control of the therapy on mobile organs. For example, real-time processing of MR images, combined with a High Intensity Focused Ultrasound (HIFU) system with rapid electronic displacement of the focal point, can be used to achieve a regional temperature control. Similarly, the recent development of integrated MRI linear accelerator, which are designed for simultaneous irradiation and MR-imaging, shows great potential for on-line radiotherapy guidance.

Although these new techniques appear well suited for cancer therapy in vital organs such as kidney and liver, their physiological motion induced by breathing and/or cardiac activities requires a precise real-time motion management during the interventional procedure to ensure:

A complete dose coverage and a minimal damage to nearby organs at risk.
An on-line correction of motion related errors on images acquired on the fly.

To this end, several techniques are actively developped to assess either qualitative or quantitative motion information: While a qualitative assessment of physiological activities can be obtained by means of a respiratory pressure belt or cardiac ECG, a quantitative displacement information in the vicinity of the targeted region can be provided by navigator echos or ultrasonic echos.

More recently, fast MR-acquisition protocols allow acquiring images with a respectable spatial and temporal resolution and a good contrast with a clear depiction of both targeted and healthy regions. These images can be conveniently used to estimate organ displacements during the therapy using image registration algorithms. In particular, optical flow formulations derived from the original approach of Horn & Schunck, which are based on a spatial regularity constraint of the motion, were recently shown to be well adapted for the estimation in real-time of elastic organ deformations. RealTITracker thereby demonstrated a great potential for:

3D registration of intra-patient data for the planning of an intervention.
2D/3D real-time tumor/organ tracking based on real-time imaging.

What is the benefit of using optical flow algorithms compared to other competing image registration strategies?

The main advantages of optical flow approaches are focused around:

The reduced number of required control parameters: The optical flow algorithm requires fixating only a single input parameter, which reflects the elasticity of the tracked organs. Although this parameter may theoretically be adapted for each specific data-set, a fixed value can generally be found in practice for a specific medical application. The reader is referred to the section "What are the input parameters for the algorithm?" located below for a brief discussion about of the calibration of the single input parameter.
The reduced computation time required to estimate organ displacements: The cost function optimized by RealTITracker for the motion estimation process is differentiable (the reader is referred to the question "How do optical flow algorithms work?" located below for more details about the energy formulation). As a result, the metric optimization results in the resolution of a linear system which can be solved very fastly using Gauss-Seidel or Successive over-relaxation methods.

How do optical flow algorithms work?

Differential methods of estimating optical flow, based on partial and higher-order partial derivatives of the image signal and/or the sought flow field. These methods are called differential since they are based on local Taylor series approximations of the image signal.

Optical flow algorithms allow estimating a velocity field assuming an intensity conservation during displacement, mathematically expressed by the optical flow constraint equation (OFCE) as follows:

where u and v are displacement vector components, and I_x,y,t are the spatio-temporal partial derivatives of the pixel intensity of the image.

However, a direct estimation by minimizing the deviation from the OFCE is an under-determined problem and thus an additional constraint is required. The optical flow formulation of Horn&Schunck, initially proposed in the context of motion estimation in video sequence in 1981 by Horn and Schunck, is based on a L² spatial regularity constraint of the motion, which was recently shown to be well adapted for the estimation in real-time of elastic organ deformations: the algorithm proposed by Horn and Schunck introduced an additional "physical" constraint by assuming that the motion field is smooth in the neighborhood of estimation point. This approach seeks a motion field minimizing:

where α is a user defined weighting factors designed to link both intensity variation (left part of Eq. 2) and motion field regularity (right part of Eq. 2).

The complete description of the numerical resolution of the metric of Eq. 2 can be found here.

To ensure the convergence of the algorithm, the averaged variation of the estimated motion amplitude can be compared to a maximal allowed tolerance (of 10^-3 pixels for example).

RealTITracker also includes a modification proposed by Zachiu et al. which relaxes the condition of intensity conservation using a L¹ data fidelity constraint.

What is the accuracy of the estimation process?

Optical-flow based algorithms rely on the assumption of conservation of local intensity along the trajectory which can be violated during MR-acquisition. As an example, for magnetic resonance imaging, the following image artifacts may lead to local intensity variations, which in turn can be misinterpreted by optical-flow based algorithms as "motion":

Rapid MR-imaging or large spatial covering are in general associated with low Signal-To-Noise ratio (SNR) which limits the accuracy of the registration process.
Possible fold-over MR-artifacts as well as mixtures of static/dynamic parts of the entire field-of-view and/or complex motion paterns may lead to local grey level intensity variations. A carreful attention must thus be paid when parallel imaging sequences, such as SENSE, or radial acquisition schemes are employed.
During an interventional procedure, several relevant tissue properties are subject to change during imaging, which may result in local grey level intensity variations. For example, during an hyperthermia procedure, since the tissue is heated, several MR-relevant tissue properties are subject to change during the intervention.
A 3D image acquisition is optimally required. However, it is often not achievable in practice for real-time applications. Although it is well established that MR-imaging can provide motion estimates with a high spatial resolution, it is difficult in practice to acquire on-line 3D isotropic images because of the technical limitations, spatial and temporal resolution trade-offs, and low SNR associated with fast 3D acquisition sequences. 2D multi-slice acquistion schemes are consequently often preferred. Hopefully, in practice, it is possible for respiratory induced motion to choose the imaging plane direction parallel to the principal axis of the organ displacement and thus we evaluated the technique in two dimensions. However, although the motion trajectory of the kidney and the lower part of the liver can for example be approximated in first order by a linear shift, the true trajectory is a curve in 3D space. In particular, the upper part of the liver, which is subjected to an elastic deformation, is hard to contain in a static 2D imaging slice during the entire respiratory cycle. It is often not possible to ensure that the target area remains observable by a single static image slice during the entire motion cycle and much efforts are thus currently conducted toward 3D real-time motion estimation:
- A strategy introduced in the following paper, consists of combining 2D MR-images and 1D MR-navigators to obtain 3D target motion estimates: While the MR-navigator provides a measure of the target motion along one direction, MR images are used to track the target motion in the two remaining spatial dimensions using image-based motion estimation algorithms. The position of the imaging plane was also adjusted based on the MR-navigator to reduce through plane motion artifacts in the MR images. This approach has been successfully demonstrated during real time 3D beam steering MRgHIFU experiment.
- An alternative strategy consists of estimating the 3D motion from several volumetric scans obtained before the interventional procedure using retrospective gating strategies. However, the data acquisition and reconstruction tasks are time consuming, which would substantially extend the duration of the intervention.
If based on MR image registration algorithms, the accuracy of the beem steering may limited by the image distorsions intrinsic to the employed MR acquisition sequence. The distorsions mainly originate from the inhomogeneities of the MR scanner’s static magnetic field and from chemical shifts (ex. water-fat shift) that might occur at organ and body borders. The resulting theoretical shifts must be compared to the employed image resolution, in order to quantify weither this effects can be deemed to be neglijible or not. In order to eliminate the effects of the water-fat shift, the MR-sequence can be optimized to be fat supressing. This will not only remove fat shifts (and thus image artifacts) at the organ and image boundaries but it will also increase the contrast-to-noise ratio of the entire image, fact which is beneficial for the accuracy of the optical-flow based motion estimation.

How is the accuracy of the estimated optical-flow based motion assessed in the litterature?

The quantification of the precision and the accuracy of the estimated motion field for a specific application is quite a hard task, especially in-vivo since in this case the true motion is not known. Here are detailed several tests used in our previous studies to assess the performance of the RealTITracker toolbox during pre-clinical interventional procedures:

Compare estimated image-based displacements to gold standard diplacements provided by one or several external sensors: This technique is easy to implement and commonly used to assess the accuracy of the motion estimate on an ex-vivo target. For example, we employed this strategy in the following paper in order to evaluate an MRI-controlled HIFU system together with spatial temperature control: a mobile evaluation platform was installed on the patient MRI bed as follows:
Mechanism used to displace the target in a cone-shaped piece of meat. The HIFU transducer is placed in the center of the magnet, and the motor is at the end of the patient bed.
A plastic bag with fresh pig muscle was installed on the platform. Its translational displacement was remotely controlled by a mechanical transmission line. The motion period was varied by adjusting the voltage applied to the motor. To study rigid motion (e.g.), translation, the position of the transmission line is measured with a millimeter-striped bar. Two transceiver optical diodes detect the position of the striped bar with a precision of 0.5 mm. A microcontroller at 20 MHz (model PIC16F688; Microchip Technology Inc., USA) records the number of graduated displacements with a temporal accuracy of microseconds. As soon as a graduation change is detected, the timing and position of the target are transferred to the HIFU therapeutic computer by an RS232 connection in a few milliseconds. The real-time target position data were used to quantify the accuracy of the estimated displacements derived from the MR images.
Compare estimated target positions with manually defined and tracked landmarks: This strategy is much more tedious, especially when it concerns the motion estimation of dynamically acquired images with high framerate! However, a small number of 3D data bank and the corresponding tracked landmark can be found in the web. In the following paper, the motion estimation process was evaluated on the POPI-model which is a 4D thorax virtual phantom. It consists of 10 CT images of resolution 482×360×141 from a free breathing volunteer. The images were acquired at the Léon Bérard Cancer Center, Lyon, France. In each image 41 landmark points have been manually identified, and these points were used for validation.
Quantify the matching of the targeted organ/tumor edges using manually defined region of interests: In the study available here, the quality assessment of the registration was analyzed by computing the DICE similarity coefficient (noted DSC) between the position of the organ (ROI_t) in each registered frame (t) and its position in the reference frame (ROI_ref), as follows:

Each ROI was obtained by manual segmentation. A value of 1 for DSC indicates an ideal registration (perfect ROI matching).
Evaluate of the robustness against low SNR values: In the following paper, Rician noise of various amplitude was added to a noise-free image, in order to gradually reduce its SNR. A mean square error (MSE) between the current estimated motion vector field and the noise-free image were then computed in order to detect the lowest value of the SNR for which the estimated motion is still reliable.

What are the input parameters for the algorithm?

Compared to other image registration techniques, on of the benefit of optical flow approaches is focused around the minimal number of required control parameters. The optical flow algorithm requires fixating only a single parameter (referred to as α in Eq. 2), which reflects the elasticity of the tracked organs.

The reader is referred to the following paper for a complete analysis of the impact of the α value on the outcome of the optical flow metric: While an increased α value intrinsically improve the robustness against low SNR values, it also limits the estimation of elastic deformation. A compromise must consequently be found. The accuracy was assessed ex-vivo by comparing the estimated image-based displacements using gold standard diplacements provided by external sensors, and in-vivo using gold standard landmark points manually positioned tracked by a staff scientist.

For example, with the implementation employed in RealTITracker, it was shown any value in the range of 0.3 and 0.5 for α provided tracking performance within the the gold standard precision for 2D sagital images of the abdomen, Signal-To-Noise Ratio around 10-15, and in-plane pixel size of 2×2 mm².

Is it true that optical flow algorithms are restrained to the estimation of displacement of small amplitude?

Since the Taylor approximation of the Horn&Schunck formulation of Eq. (2) holds only for small displacements, it is often heard that Horn&Schunck based methods are restrained to the estimation of displacement of small amplitude. However, since 1981, many improvements have been carried to improve the numerical resolution of Eq. (2) and to stabilize the convergence of the employed algorithm. In particular, a multiresolution scheme can be performed, which iterates the registration algorithm from a downsampled image step by step to the original image resolution. In addition, the warping theory formalized here proposes to update partial derivative I_x,y,t within each resolution level (this procedure is referred to as "iterative refinement"). All these techniques together allows estimating motion of large amplitude.