English

Optical Flow Sample

The optical flow processing sample uses the hbVPPyrDown, hbVPCornerHarris and hbVPOpticalFlowPyrLK interfaces, which involve LKOF and DSP hardware IP. It shows how to perform optical flow tracking on consecutive frame images for detailed implementation, please compare the practice with the sample source code.

Function overview

This sample realizes the function of feature point tracking on the image input of consecutive frames. The input image is of NV12 type, which is converted to gray type and then used hbVPPyrDown to generate 5 layers of pyramid layers. The first frame uses hbVPCornerHarris to generate harris corner points, and subsequent frames track the corner points generated in the first frame using hbVPOpticalFlowPyrLK. Every 5 frames hbVPCornerHarris is called again to generate a priori eigenpoints to calibrate the tracking results.

The sample code process can be roughly divided into the following four steps, the detailed implementation of the sample source code please combine.

Allocate memory for input image and input-output feature points.
Prepare parameters for optical flow calculations.
The computations for each frame are performed sequentially, including generation of pyramid layers, corner detection and optical flow tracking.
Release memory resources.

Operation guide

The optical flow processing sample execution script is located in the ucp_tutorial/vp/vp_samples/script/04_optical_flow directory and contains the following script:

04_optical_flow
├── README.md
└── run_optical_flow.sh

The command line in the run_optical_flow.sh script reads as follows:

bin=../aarch64/bin/optical_flow_sample
lib=../aarch64/lib

export LD_LIBRARY_PATH=${lib}:${LD_LIBRARY_PATH}
export PATH=${bin}:${PATH}
export HB_DSP_ENABLE_DIRECT_MODE=true
export HB_DSP_ENABLE_CONFIG_VDSP=true
export HB_DSP_LOG_LEVEL=3
export HB_DSP_VDSP_LOG_LEVEL=3

# Usage: ${bin} <image_file_list> <frame_num> <lkof_threshold> <harris_top_k>
${bin} ../../data/frame_images/images.list 10 5 5

The meanings of the command line parameters are as follows:

${bin}: Compiled executable file.
image_file_list: Input image list。
frame_num: The number of running frames can be adjusted to the number of calculated frames according to the actual needs.
lkof_threshold: Optical flow confidence threshold for filtering the results of optical flow calculations, user adjustable.
harris_top_k: Filtering top k number results of corner point calculations, user adjustable.

To use it, go to the vp_samples/script/04_optical_flow directory, and then directly execute run_optical_flow.sh. The execution method and log output are as follows:

#!/bin/sh

root@j6dvb:/userdata/app/horizon/vp_samples/script/04_optical_flow# sh run_optical_flow.sh
[I][9150][08-09][15:13:49:256][optical_flow.cpp:289][optical_flow_sample][SAMPLE] frame [0] tracking end.
...

Description of the results

At the end of the sample run, a visualization picture will be generated in the vp_samples/script/04_optical_flow directory, and the contents of the generated object for this sample is as follows:

04_optical_flow
├── lkof_1.jpg
├── lkof_2.jpg
├── lkof_3.jpg
├── lkof_4.jpg
├── lkof_5.jpg
├── lkof_6.jpg
├── lkof_7.jpg
├── lkof_8.jpg
├── lkof_9.jpg
├── README.md
└── run_optical_flow.sh

The first frame will show the results of the harris feature point detection and the subsequent frames will show the optical flow tracking trajectory. Among them, the blue points are the feature points of the previous frame image, the red points are the feature points of the current frame image, and the green color is the optical flow trajectory. The tracking effect of the first and nth frame is shown as belows:

First Frame	Frame n

Attention

The PyrDown and CornerHarris calculations for this samples are processed on a DSP, so please configure your DSP environment in advance.
If you need to replace a picture, please make sure that the replacement input picture is a continuous frame image, otherwise the result of the optical flow calculation may be inaccurate.

PTQ Conversion Tools

The hb_compile Tool

PTQ Conversion Steps

PTQ Conversion Samples

FAQ and Common Failure Resolutions

Appendix

Tutorial

Advanced Tutorial

API Reference

QAT

Model Export

Horizon Operator

Vision Process

Vision Algorithm

Sample

VP API Introduction

Data Structure

Function Interface

Model Inference Development

Model Inference API Introduction

Data Type and Data Structure

API Interface

Model Inference Tool Introduction

High Performance Library

High Performance Algorithm

HPL API Introduction

Data Structure

Functional Interface

UCP Common API Introduction

Data Structure

Functional Interface

UCP Performance Analysis Tools

Horizon Torch Samples

FRAMEWORK

TUTORIALS

EXAMPLES

API REFERENCE

Toolchain Operator Support Constraint List

Optical Flow Sample

The hb_compile Tool

QAT

Model Export

Horizon Operator

Vision Algorithm

Sample

VP API Introduction

Data Structure

Function Interface

Model Inference API Introduction

Data Type and Data Structure

API Interface

Model Inference Tool Introduction

High Performance Algorithm

HPL API Introduction

Data Structure

Functional Interface

Data Structure

Functional Interface

FRAMEWORK

TUTORIALS

EXAMPLES

API REFERENCE

#Optical Flow Sample

Optical Flow Sample