English

GaNet Lane Line Detection Model Training

This tutorial focuses on showing how to train a GaNet model from scratch on the lane line dataset CuLane using HAT, including floating-point, quantitative, and fixed-point models.

CuLane is one of the most commonly used datasets for lane line detection, and many advanced lane line detection studies are preferentially based on this dataset for good validation.

Prior to the model training, we need to prepare the dataset first. Here we download the official dataset and the corresponding labeled data CuLaneDataset. Note that the annotations_new.tar.gz file must be extracted at the end.

The structure of the data directory after unzipping is as follows:

tmp_data
  |-- driver_100_30frame
      |-- xxx.MP4
          |-- xxx.lines.txt
          |-- xxx.jpg
  |-- driver_161_90frame
  |-- driver_182_30frame
  |-- driver_193_90frame
  |-- driver_23_30frame
  |-- driver_37_30frame
  |-- list
      |-- test_split
          |-- test0_normal.txt
          |-- test1_crowd.txt
          |-- test2_hlight.txt
          |-- test3_shadow.txt
          |-- test4_noline.txt
          |-- test5_arrow.txt
          |-- test6_curve.txt
          |-- test7_cross.txt
          |-- test8_night.txt
      |-- test.txt
      |-- train_gt.txt
      |-- train.txt
      |-- val_gt.txt
      |-- val.txt

Where list/train.txt contains the path to the training data and list/test.txt contains the path to the test data.

Training Process

If you just want to simply train the GaNet model, then you can read this section first.

Similar to other tasks, HAT performs all training tasks and evaluation tasks in the form of tools + config.

After preparing the original dataset, take the following process to complete the whole training process.

Dataset Preparation

To improve the training speed, we packed the original dataset and converted it to the LMDB format.

Simply run the following script to complete the conversion:

python3 tools/datasets/culane_packer.py --src-data-dir ${data-dir} --split-name train --pack-type lmdb  --num-workers 10 --target-data-dir ${target-data-dir}
python3 tools/datasets/culane_packer.py --src-data-dir ${data-dir} --split-name test --pack-type lmdb  --num-workers 10 --target-data-dir ${target-data-dir}

The above two commands correspond to the transformation of the training dataset and the verification dataset, respectively. After packaging, the file structure in the ${target-data-dir} directory should look as follows:

${target-data-dir}
  |-- train_lmdb
  |-- test_lmdb

train_lmdb and test_lmdb are the packaged training dataset and verification dataset. You can then start training the model.

Model Training

Before the network starts the training, you can first calculate the number of network operations and parameters using the following command:

python3 tools/calops.py --config configs/lane_pred/ganet/ganet_mixvargenet_culane.py

The next step is to start the training. Training can also be done with the following script. Before training, you need to make sure that the dataset path specified in the configuration has already been changed to the path of the packaged dataset.

python3 tools/train.py --stage "float" --config configs/lane_pred/ganet/ganet_mixvargenet_culane.py
python3 tools/train.py --stage "calibration" --config configs/lane_pred/ganet/ganet_mixvargenet_culane.py

Since the HAT algorithm package uses the registration mechanism, it allows each training task to be started in the form of train.py plus a config file. The train.py is a uniform training script and independent of the task, and the tasks we need to train, the datasets we need to use, and the hyperparameter settings related to training are all in the specified config file.

The parameters after --stage in the above command can be "float", "calibration", which, respectively, indicates the training of the floating-point model and the quantitative model, and the conversion of the quantitative model to the fixed-point model, where the training of the quantitative model depends on the floating-point model produced by the previous floating-point training.

Export FixedPoint Model

Once you've completed your quantization training, you can start exporting your fixed-point model. You can export it with the following command:

python3 tools/export_hbir.py --config configs/lane_pred/ganet/ganet_mixvargenet_culane.py

Model Verification

After completing the training, we get the trained floating-point, quantitative, or fixed-point model. Similar to the training method, we can use the same method to complete metrics validation on the trained model and get the metrics of Float, Calibration, and Quantized, which are floating-point, quantitative, and fully fixed-point metrics, respectively.

python3 tools/predict.py --stage "float" --config configs/lane_pred/ganet/ganet_mixvargenet_culane.py

python3 tools/predict.py --stage "calibration" --config configs/lane_pred/ganet/ganet_mixvargenet_culane.py

Similar to the model training, we can use --stage followed by "float", "calibration", to validate the trained floating-point model, and quantitative model, respectively.

The following command can be used to verify the accuracy of a fixed-point model, but it should be noted that hbir must be exported first:

python3 tools/predict.py --stage "int_infer" --config configs/lane_pred/ganet/ganet_mixvargenet_culane.py

Model Inference

HAT provides the infer_hbir.py script to visualize the inference results for the fixed-point model:

python3 tools/infer_hbir.py --config configs/lane_pred/ganet/ganet_mixvargenet_culane.py --model-inputs img:${img-path} --save-path ${save_path}

Simulation Board Accuracy Verification

In addition to the above model validation, we provide an accuracy validation method identical to the on-board environment, which can be accomplished by:

python3 tools/validation_hbir.py --stage "align_bpu" --config configs/lane_pred/ganet/ganet_mixvargenet_culane.py

Fixed-point Model Checking and Compilation

As the quantitative training toolchain integrated in HAT is mainly prepared for Horizon's processors, it is a must to check and compile the quantitative models.

We provide an interface for model checking in HAT, which allows the user to define a quantitative model and then check whether it can work properly on the BPU first.

python3 tools/model_checker.py --config configs/lane_pred/ganet/ganet_mixvargenet_culane.py

After the model is trained, you can use the compile_perf_hbir script to compile the quantitative model into an HBM file that supports on-board running. The tool can also predict the performance on the BPU.

python3 tools/compile_perf_hbir.py --config configs/lane_pred/ganet/ganet_mixvargenet_culane.py

The above is the whole process from data preparation to the generation of quantitative and deployable models.

Training Details

In this note, we explain some things that need to be considered for model training, mainly including settings related to config.

Model Construction

The network structure of GaNet can be found in the Paper, which is not described in detail here.

We can easily define and modify the model by defining a dict type variable like model in the config file.

from hat.models.backbones.mixvargenet import MixVarGENetConfig
bn_kwargs = {}
radius = 2
hid_dim = 32
attn_ratio = 4

model = dict(
    type="GaNet",
    backbone=dict(
            type='MixVarGENet',
            net_config=[
                [MixVarGENetConfig(in_channels=32, out_channels=32, head_op='mixvarge_f2', stack_ops=[], stride=1, stack_factor=1, fusion_strides=[], extra_downsample_num=0)],
                [MixVarGENetConfig(in_channels=32, out_channels=32, head_op='mixvarge_f4', stack_ops=['mixvarge_f4', 'mixvarge_f4'], stride=2, stack_factor=1, fusion_strides=[], extra_downsample_num=0)],
                [MixVarGENetConfig(in_channels=32, out_channels=64, head_op='mixvarge_f4', stack_ops=['mixvarge_f4', 'mixvarge_f4'], stride=2, stack_factor=1, fusion_strides=[], extra_downsample_num=0)],
                [MixVarGENetConfig(in_channels=64, out_channels=96, head_op='mixvarge_f2_gb16', stack_ops=['mixvarge_f2_gb16', 'mixvarge_f2_gb16', 'mixvarge_f2_gb16', 'mixvarge_f2_gb16', 'mixvarge_f2_gb16', 'mixvarge_f2_gb16'], stride=2, stack_factor=1, fusion_strides=[], extra_downsample_num=0)],
                [MixVarGENetConfig(in_channels=96, out_channels=160, head_op='mixvarge_f2_gb16', stack_ops=['mixvarge_f2_gb16', 'mixvarge_f2_gb16'], stride=2, stack_factor=1, fusion_strides=[], extra_downsample_num=0)]
            ],
            disable_quanti_input =False,
            input_channels = 3,
            input_sequence_length = 1,
            num_classes = 1000,
            bn_kwargs = bn_kwargs,
            include_top = False,
            bias = True,
            output_list =[2, 3, 4],
    ),
    neck = dict(
        type="GaNetNeck",
        fpn_module = dict(
            type="FPN",
            in_strides=[8, 16, 32],
            in_channels=[64, 96, hid_dim],
            out_strides=[8, 16, 32],
            out_channels=[hid_dim, hid_dim, hid_dim],
        ),
        attn_in_channels=[160],
        attn_out_channels=[hid_dim],
        attn_ratios=[attn_ratio],
        pos_shape=(1, 10, 25)
    ),
    head=dict(
        type="GaNetHead",
        in_channel=hid_dim,
    ),
    targets=dict(
        type="GaNetTarget",
        hm_down_scale=8,
        radius=radius,
    ),
    post_process=dict(
        type="GaNetDecoder",
        root_thr=1,
        kpt_thr=0.4,
        cluster_thr=5,
        downscale=8,
    ),
    losses=dict(
        type="GaNetLoss",
        loss_kpts_cls=dict(
            type="LaneFastFocalLoss",
            loss_weight=1.0,
        ),
        loss_pts_offset_reg=dict(
            type="L1Loss",
            loss_weight=0.5,
        ),
        loss_int_offset_reg=dict(
            type="L1Loss",
            loss_weight=1.0,
        ),
    ),
)

In addition to backbone , the model also has neck , head , targets , post_process and losses modules.

In GaNet , backbone is mainly to extract the features of the image, neck is mainly for feature enhancement, and head is mainly to get the predicted fraction and offset of key points of lane lines from the features.

targets are training targets from gt and post_process is mainly the post-processing part, which is used in inference.

The losses part uses LaneFastFocalLoss and L1Loss from the paper as the training losses and loss_weight is the weight of the corresponding loss.

Data Enhancement

Like the definition of model, the data enhancement process is implemented by defining train_data_loader and val_data_loader in the config file, which correspond to the processing of the training sets and verification sets, respectively. Taking train_data_loader as an example, the data enhancement uses FixedCrop, RandomFlip, Resize, RandomSelectOne, RGBShift, HueSaturationValue, JPEGCompress, MeanBlur, MedianBlur, RandomBrightnessContrast, ShiftScaleRotate and RandomResizedCrop to increase the diversity of training data and enhance the generalization ability of the model.

train_data_loader = dict(
    type=torch.utils.data.DataLoader,
    dataset=dict(
        type="CuLaneDataset",
        data_path=train_data_path,
        to_rgb=True,
        transforms=[
            dict(
                type="FixedCrop",
                size=(0, 270, 1640, 320),
            ),
            dict(
                type="RandomFlip",
                px=0.5,
                py=0.0,
            ),
            dict(
                type="Resize",
                img_scale=(320, 800),
                multiscale_mode='value',
                keep_ratio=False,
            ),
            dict(
                type='RandomSelectOne',
                transforms=[
                    dict(
                        type="RGBShift",
                        r_shift_limit=(-10, 10),
                        g_shift_limit=(-10, 10),
                        b_shift_limit=(-10, 10),
                        p=1.0,
                    ),
                    dict(
                        type="HueSaturationValue",
                        hue_range=(-10, 10),
                        sat_range=(-15, 15),
                        val_range=(-10, 10),
                        p=1.0,
                    ),
                ],
                p=0.7,
            ),
            dict(
                type="JPEGCompress",
                p = 0.2,
                max_quality = 85,
                min_quality = 95,
            ),
            dict(
                type='RandomSelectOne',
                transforms=[
                    dict(
                        type="MeanBlur",
                        ksize=3,
                        p=1.0,
                    ),
                    dict(
                        type="MedianBlur",
                        ksize=3,
                        p=1.0,
                    ),
                ],
                p=0.2,
            ),
            dict(
                type="RandomBrightnessContrast",
                brightness_limit=(-0.2, 0.2),
                contrast_limit=(-0.0, 0.0),
                p=0.5,
            ),
            dict(
                type="ShiftScaleRotate",
                shift_limit=(-0.1, 0.1),
                scale_limit=(0.8, 1.2),
                rotate_limit=(-10, 10),
                interpolation=1,
                border_mode=0,
                p=0.6,
            ),

            dict(
                type='RandomResizedCrop',
                height=320,
                width=800,
                scale=(0.8, 1.2),
                ratio=(1.7, 2.7),
                p=0.6,
            ),
            dict(
                type="ToTensor",
                to_yuv=False,
            ),
        ],
    ),
    sampler=dict(type=torch.utils.data.DistributedSampler),
    batch_size=batch_size_per_gpu,
    pin_memory=True,
    shuffle=True,
    num_workers=data_num_workers,
    collate_fn=collate_2d,
)

Since the final model running on BPU uses a YUV444 image input, and the general training image input is in the RGB format, HAT provides BgrToYuv444 data enhancement to convert RGB to YUV444. To optimize the training process, HAT uses batch_processor, which allows some enhancement processes to be placed in batch_processor to optimize the training:

def loss_collector(outputs: dict):
    losses = []
    for _, loss in outputs.items():
        losses.append(loss)
    return losses

train_batch_processor = dict(
    type="BasicBatchProcessor",
    need_grad_update=True,
    loss_collector=loss_collector,
    batch_transforms=[
        dict(type="BgrToYuv444", rgb_input=True),
        dict(
            type="TorchVisionAdapter",
            interface="Normalize",
            mean=128.0,
            std=128.0,
        ),
    ],
)

In which loss_collector is a function that gets the loss of the current batch data.

The data transformation of the validation set is relatively simpler, as follows:

val_data_loader = dict(
    type=torch.utils.data.DataLoader,
    dataset=dict(
        type="CuLaneDataset",
        data_path=val_data_path,
        to_rgb=True,
        transforms=[
            dict(
                type="FixedCrop",
                size=(0, 270, 1640, 320),
            ),
            dict(
                type="Resize",
                img_scale=(320, 800),
                multiscale_mode='value',
                keep_ratio=False,
            ),
            dict(
                type="ToTensor",
                to_yuv=False,
            ),
        ],
    ),
    sampler=dict(type=torch.utils.data.DistributedSampler),
    batch_size=batch_size_per_gpu,
    pin_memory=True,
    shuffle=False,
    num_workers=data_num_workers,
    collate_fn=collate_2d,
)

val_batch_processor = dict(
    type="BasicBatchProcessor",
    need_grad_update=False,
    loss_collector=None,
    batch_transforms=[
        dict(type="BgrToYuv444", rgb_input=True),
        dict(
            type="TorchVisionAdapter",
            interface="Normalize",
            mean=128.0,
            std=128.0,
        ),
    ],
)

Training Strategy

Train the floating-point model on the CuLane dataset using the Cosine learning strategy with Warmup, as well as applying L2 norm to the weight parameters. The float_trainer, calibration_trainer, and int_trainer in the configs/lane_pred/ganet/ganet_mixvargenet_culane.py file correspond to the training strategies for floating-point, quantitative, and fixed-point models, respectively.

The following is an example of float_trainer training strategy:

import torch
base_lr = 0.01
num_epochs = 240

float_trainer = dict(
    type="distributed_data_parallel_trainer",
    model=model,
    data_loader=train_data_loader,
    optimizer=dict(
        type=torch.optim.Adam,
        params={"weight": dict(weight_decay=4e-5)},
        lr=base_lr,
    ),
    batch_processor=train_batch_processor,
    stop_by="epoch",
    num_epochs=num_epochs,
    device=None,
    sync_bn=True,
    callbacks=[
        stat_callback,
        loss_show_update,
        dict(
            type="CosLrUpdater",
            warmup_len=1,
            warmup_by="epoch",
            step_log_interval=10,
        ),
        val_callback,
        ckpt_callback,
    ],
    train_metrics=[
        dict(type="LossShow"),
    ],
    val_metrics=[
        dict(type="CulaneF1Score"),
    ],
)

Quantization Training

For key steps in quantitative training, such as preparing the floating-point model, operator substitution, inserting quantization and inverse quantization nodes, setting quantitative parameters, and operator fusion, please read the Quantized Awareness Training (QAT) section. Here we focus on how to define and use the quantization models in lane prediction of HAT.

If the model is ready, and some existing modules are quantized, HAT uses the following script in the training script to map the floating-point model to the fixed-point model uniformly.

model.fuse_model()
model.set_qconfig()
horizon.quantization.prepare_qat(model, inplace=True)

The overall strategy of quantitative training can directly follow the strategy of floating-point training, but the learning rate and training length need to be adjusted appropriately. Due to the existence of the floating-point pre-training model, the learning rate Lr for quantitative training can be very small. Generally, you can start from 0.001 or 0.0001, and you can do one or two times Lr adjustments of scale=0.1 with StepLrUpdater without prolonging the training time. In addition, weight decay will also have some effect on the training results.

The quantitative training strategy for the GaNet example model can be found in the configs/lane_pred/ganet/ganet_mixvargenet_culane.py file.

On This Page

The hb_compile Tool

QAT

Model Export

Horizon Operator

Vision Algorithm

Sample

VP API Introduction

Data Structure

Function Interface

Model Inference API Introduction

Data Type and Data Structure

API Interface

Model Inference Tool Introduction

High Performance Algorithm

HPL API Introduction

Data Structure

Functional Interface

Data Structure

Functional Interface

FRAMEWORK

TUTORIALS

EXAMPLES

API REFERENCE

#GaNet Lane Line Detection Model Training

#Training Process

#Dataset Preparation

#Model Training

#Export FixedPoint Model

#Model Verification

#Model Inference

#Simulation Board Accuracy Verification

#Fixed-point Model Checking and Compilation

#Training Details

#Model Construction

#Data Enhancement

#Training Strategy

#Quantization Training

GaNet Lane Line Detection Model Training

Training Process

Dataset Preparation

Model Training

Export FixedPoint Model

Model Verification

Model Inference

Simulation Board Accuracy Verification

Fixed-point Model Checking and Compilation

Training Details

Model Construction

Data Enhancement

Training Strategy

Quantization Training