#OpticalFlowPyrLK

The OpticalFlowPyrLK operator is used to calculate the changes of specific pixel points in consecutive multi-frame images, which is commonly used in scenarios such as target feature point tracking and improving BOX stability.

#Operator Effect

Input continuous frame image	Parameter	Output Image
	-

#Principle

The sparse optical flow algorithm, known as the Lucas-Kanade algorithm, is a motion estimation method commonly used in computer vision to estimate the direction and speed of motion of pixels in an image sequence. The Lucas-Kanade algorithm is based on two assumptions:

1. Brightness constancy assumption: the brightness of pixels of the same object remains constant over a short period of time.

Assuming that the object located at $(x,y)$ pixel position at time $t$ is located at $(x+u,y+v)$ position at time $t+Δt$, based on the assumption of brightness invariance we have.

$I(x,y,t)=I(x+u,y+v,t+\Delta t)$

A first-order Taylor expansion of the right-hand side of the equation yields:

$I(x+u,y+v,t+\Delta t)=I(x,y,t) + I'_xu + I'_yv+I'_t\Delta t$

It can be obtained from the above two equations:

$I'_xu + I'_yv+I'_t\Delta t=0$

The collation can be expressed as:

$\begin{aligned}\begin{bmatrix} I'_x , I'_y \end{bmatrix}\begin{bmatrix} u \\v \end{bmatrix}=-\Delta I_t\end{aligned}$

Among them, $I'_x$ and $I'_y$ are the partial derivatives of the image luminance in the $x$ direction and the $y$ direction at the $(x,y)$ pixel point, respectively, the gradient of the image in the $x$ and $y$ directions.

2. Neighborhood optical flow similarity assumes that the direction and size of pixel movement in a small image region is essentially the same.

With the help of this assumption, all pixels in the field of pixel point $(x,y)$ have the following equation.

$\begin{aligned}\begin{bmatrix} I'_{\text{x1}} , I'_{\text{y1}} \\I'_{\text{x2}} , I'_{\text{y2}} \end{bmatrix}\begin{bmatrix} u \\v \end{bmatrix}= \begin{bmatrix} -\Delta I_{\text{t1}} \\-\Delta I_{\text{t2}} \end{bmatrix}\end{aligned}$

The above equation, which is of the form $Ax=b$, leads to a least squares solution for the optical flow vector:

$x=(A^TA)^{\text{-1}}A^Tb$

One of the requirements is that $A^TA$ is invertible, and in order to fulfill this requirement, the Lucas-Kanade method selects corner points as feature points. In addition to being based on the luminance invariance assumption and the neighborhood optical flow similarity assumption, the Lucas-Kanade algorithm solves for the case of large image offsets with the help of image pyramids, where large offsets become small offsets on high-level low-resolution images, thus solving for the optical flow.

Therefore, the OpticalFlowPyrLK operator requires the pyramid layers of the previous and previous frames, and the feature points of the previous frame as input, where the feature points are usually chosen to be corner points. Successive frame calculations amplify the error, when it is also necessary to introduce a priori feature points of the current frame, which are used to correct the error.

#API Interface

For detailed interface information, please refer to hbVPOpticalFlowPyrLK.

#Usage