English

Deformable DETR Detection Model Training

This tutorial uses Deform-detr as an example of how to train a detection model for a fixed point using the HAT algorithm package. Before starting Quantized Awareness Training (QAT), which is also known as fixed-point model training, you first need to train a pure floating-point model with high accuracy, and then you can quickly train a fixed-point model by doing finetune based on this pure floating-point model. So we start by training a pure floating-point Deform-detr model.

Training Process

As with other tasks, for all training and evaluation tasks, HAT is unified in the form of tools + config. After preparing the raw dataset, the following process can be used to conveniently completes the entire training process.

Dataset Preparation

Before we start training the model, the first step is to prepare the dataset, here we download MSCOCO's train2017.zip and val2017.zip as the training and validation sets for the network, and we also need to download the corresponding labeled data annotations_trainval2017.zip , and the structure of the data directory after unzipping is shown below:

tmp_data
|-- mscoco
  -- annotations_trainval2017.zip
  |-- train2017.zip
  |-- val2017.zip
  |-- annotations
  |-- train2017
  |-- val2017

Meanwhile, in order to improve the speed of training, we have done a packing of the original dataset in jpg format and converted it to lmdb format dataset. Simply run the following script for successful conversion:

python3 tools/datasets/mscoco_packer.py --src-data-dir ./tmp_data/mscoco/ --target-data-dir ./tmp_data/mscoco --split-name train --pack-type lmdb
python3 tools/datasets/mscoco_packer.py --src-data-dir ./tmp_data/mscoco/ --target-data-dir ./tmp_data/mscoco --split-name val --pack-type lmdb

The above two commands correspond to converting the training dataset and the validation dataset, respectively. After the packaging is completed, the file structure in the data directory should be as follows:

tmp_data
|-- mscoco
  |-- annotations
  |-- train2017
  |-- train_lmdb
  |-- val2017
  |-- val_lmdb

train_lmdb and val_lmdb are the packaged training dataset and validation dataset, and the final dataset read by the network.

Model Training

Once the dataset is ready, it is time to start training the floating-point Deformable Detr detection network. You can use the following commands to test the amount of computation and the number of parameters of the network first before the network training starts:

python3 tools/calops.py --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py --input-shape "1,3,800,1332"

If you simply want to start such a training task, just run the following command:

python3 tools/train.py --stage float --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py
python3 tools/train.py --stage calibration --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py
python3 tools/train.py --stage qat --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py

The above commands complete the training of floating-point model and fixed-point model respectively, where the training of fixed-point model needs to be based on the trained floating-point model. For details, please refer to Quantized Awareness Training (QAT) section.

Exporting Fixed-point Model

After completing the quantization training, you can start exporting the fixed-point model. It can be exported with the following command:

python3 tools/export_hbir.py --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py

Model Validation

After completing the training, the trained floating-point, quantized or fixed-point model can be obtained. Similar to the training method, we can use the same method to do metrics validation on the trained model, obtaining metrics as Float, Calibration, and Quantized, which are floating-point, quantized, and fully fixed-point metrics, respectively.

python3 tools/predict.py --stage "float" --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py

python3 tools/predict.py --stage "qat" --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py

Similar to training models, --stage followed by "float" and "calibration" can be used to validate trained floating-point models and quantized models, respectively.

Model Inference

HAT provides the infer_hbir.py script to provide a visual presentation of the inference results from the fixed-point model:

python3 tools/infer_hbir.py --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py --model-inputs img:${img-path} --save-path ${save_path}

Simulation on Board Accuracy Verification

In addition to the model validation described above, we also provide an accuracy validation method that is identical to that of the upper board, which can be accomplished in the following manner:

python3 tools/validation_hbir.py --stage "align_bpu" --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py

Fixed-point Model Checking and Compilation

The quantization training toolchain integrated in HAT is primarily intended for Horizon's computing platforms, and is therefore necessary for the checking and compilation of quantization models. We provide an interface for model checking in HAT, which allows the user to define a quantitative model and then check whether it can work properly on BPU:

python3 tools/model_checker.py --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py

After the model has been trained, the compile_perf_hbir script can be used to compile the quantization model into a hbm file that can be run on the board, and the tool can also predict the performance of the run on the BPU:

python3 tools/compile_perf_hbir.py --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py

The above is the whole process from data preparation to generation of quantized deployable models.

Training Details

Model Structure

The network structure of Deformable DETR can be found in Paper, which is not described in detail here. We can easily define and modify the model by defining a dict variable like model in the config file.

model = dict(
    type="DeformableDETR",
    backbone=dict(
        type="ResNet50",
        bn_kwargs={},
        num_classes=num_classes,
        include_top=False,
    ),
    neck=dict(
        type="ChannelMapperNeck",
        in_channels=[512, 1024, 2048],
        out_indices=[2, 3, 4],
        out_channel=256,
        kernel_size=1,
        extra_convs=1,
        norm_layer=dict(
            type=nn.BatchNorm2d,
            num_features=256,
        ),
    ),
    position_embedding=dict(
        type="PositionEmbeddingSine",
        num_pos_feats=128,
        temperature=10000,
        normalize=True,
        offset=-0.5,
    ),
    transformer=dict(
        type="DeformableDetrTransformer",
        encoder=dict(
            type="DeformableDetrTransformerEncoder",
            embed_dim=256,
            num_heads=8,
            feedforward_dim=1024,
            attn_dropout=0.1,
            ffn_dropout=0.1,
            num_layers=6,
            post_norm=False,
            num_feature_levels=4,
        ),
        decoder=dict(
            type="DeformableDetrTransformerDecoder",
            embed_dim=256,
            num_heads=8,
            feedforward_dim=1024,
            attn_dropout=0.1,
            ffn_dropout=0.1,
            num_layers=6,
            return_intermediate=True,
            num_feature_levels=4,
            as_two_stage=as_two_stage,
            with_box_refine=with_box_refine,
        ),
        num_feature_levels=4,
        two_stage_num_proposals=num_queries,
        as_two_stage=as_two_stage,
    ),
    embed_dim=256,
    num_classes=num_classes,
    num_queries=num_queries,
    aux_loss=True,
    post_process=dict(
        type="DeformDetrPostProcess", select_box_nums_for_evaluation=300
    ),
    criterion=dict(
        type="DeformableCriterion",
        num_classes=num_classes,
        matcher=dict(
            type="HungarianMatcher",
            cost_class=2.0,
            cost_bbox=5.0,
            cost_giou=2.0,
            use_focal=True,
            alpha=0.25,
            gamma=2.0,
        ),
        weight_dict={
            "loss_class": 1.0,
            "loss_bbox": 5.0,
            "loss_giou": 2.0,
        },
        loss_class_type="focal_loss",
        alpha=0.25,
        gamma=2.0,
    ),
    as_two_stage=as_two_stage,
    with_box_refine=with_box_refine,
)

Where type under model indicates the name of the defined model, and the remaining variables indicate the other components of the model. The advantage of defining the model this way is that we can easily replace the structure we want. For example, if we want to train a model where backbone is efficientnet, just replace backbone under model and set neck accordingly.

Data Enhancement

As with the definition of model, the process of data enhancement is realized by defining the dicts data_loader and val_data_loader in the config configuration file, which correspond to the processing flow of the training set and validation set, respectively. Take the data_loader as an example:

data_loader = dict(
    type=torch.utils.data.DataLoader,
    dataset=dict(
        type="Coco",
        data_path="./tmp_data/mscoco/train_lmdb/",
        transforms=[
            dict(
                type="RandomFlip",
                px=0.5,
                py=0,
            ),
            dict(
                type="RandomSelectOne",
                p=1.0,
                transforms=[
                    dict(
                        type="Resize",
                        img_scale=[
                            (480, 1333),
                            (512, 1333),
                            (544, 1333),
                            (576, 1333),
                            (608, 1333),
                            (640, 1333),
                            (672, 1333),
                            (704, 1333),
                            (736, 1333),
                            (768, 1333),
                            (800, 1333),
                        ],
                        multiscale_mode="value",
                        keep_ratio=True,
                    ),
                    dict(
                        type=Compose,
                        transforms=[
                            dict(
                                type="Resize",
                                img_scale=[
                                    (400, 9999999),
                                    (500, 9999999),
                                    (600, 9999999),
                                ],
                                multiscale_mode="value",
                                keep_ratio=True,
                            ),
                            dict(
                                type="RandomSizeCrop",
                                min_size=384,
                                max_size=600,
                                filter_area=False,
                            ),
                            dict(
                                type="Resize",
                                img_scale=[
                                    (480, 1333),
                                    (512, 1333),
                                    (544, 1333),
                                    (576, 1333),
                                    (608, 1333),
                                    (640, 1333),
                                    (672, 1333),
                                    (704, 1333),
                                    (736, 1333),
                                    (768, 1333),
                                    (800, 1333),
                                ],
                                multiscale_mode="value",
                                keep_ratio=True,
                            ),
                        ],
                    ),
                ],
            ),
            dict(
                type="ToTensor",
                to_yuv=True,
                use_yuv_v2=False,
            ),
            dict(
                type="Normalize",
                mean=128,
                std=128,
            ),
        ],
    ),
    sampler=dict(type=torch.utils.data.DistributedSampler),
    batch_size=batch_size_per_gpu,
    shuffle=True,
    num_workers=4,
    pin_memory=True,
    collate_fn=collate_2d_pad,
)

Where type directly uses pytorch's own interface torch.utils.data.DataLoader, which represents the combination of pictures of batch_size size. The only thing to focus on here is probably the dataset variable, CocoFromLMDB, which indicates the image to be read from the lmdb dataset. The path is the same path we mentioned in the first part of the dataset preparation. A number of data enhancements are included under transforms. The val_data_loader imposes no data enhancements. You can also achieve your desired data enhancement by inserting new dicts into transforms.

Training Strategy

In order to train a model with high accuracy, a good training strategy is essential. For each training task, the corresponding training strategy is also defined in one of the config files. This can be seen in the variable float_trainer.

float_trainer = dict(
    type="distributed_data_parallel_trainer",
    model=model,
    model_convert_pipeline=dict(
        type="ModelConvertPipeline",
        qat_mode="fuse_bn",
        converters=[
            dict(
                type="LoadCheckpoint",
                checkpoint_path=(
                    "./tmp_pretrained_models/resnet50_imagenet/float-checkpoint-best.pth.tar" 
                ),
                allow_miss=True,
                ignore_extra=True,
                verbose=True,
            ),
        ],
    ),
    find_unused_parameters=False,
    data_loader=data_loader,
    optimizer=dict(
        type="custom_param_optimizer",
        optim_cls=torch.optim.AdamW,
        model=model,
        optim_cfgs=dict(lr=1e-4, weight_decay=1e-4),
        custom_param_mapper=dict(
            norm_types={"weight_decay": 0.0}, backbone={"lr": 1e-5}
        ),
    ),
    batch_processor=batch_processor,
    num_epochs=50,
    device=None,
    callbacks=[
        stat_callback,
        loss_show_update,
        grad_callback,
        dict(
            type="StepDecayLrUpdater",
            lr_decay_id=[40],
            step_log_interval=100,
        ),
        bn_callback,
        val_callback,
        ckpt_callback,
    ],
    train_metrics=dict(
        type="LossShow",
    ),
    val_metrics=dict(
        type="COCODetectionMetric",
        ann_file="./tmp_data/mscoco/instances_val2017.json",
    ),
)

The float_trainer defines our training approach in the big picture, including the use of multi-card distributed training (distributed_data_parallel_trainer), the number of epochs for which the model will be trained, and the choice of optimizer. One of the optimizers is the AdamW optimizer wrapped by custom_param_optimizer, which allows for finer control of optimization parameters for backbone and norm layers in the model. The callbacks also represent the small strategies used by the model during training and the actions that the user wants to implement, including the way the learning rate is transformed (StepDecayLrUpdater), the way the model is validated during training (Validation), and the action of saving (Checkpoint) the model. Of course, if you have your own operations that you want the model to implement during training, you can also add them in this dict way.

Note

If you need to reproduce the accuracy, the training strategy in config is best left unchanged. Otherwise unexpected training situations may occur.

With the above introduction, you should have a clearer understanding of the function of the config file. Then with the training script mentioned earlier, a highly accurate pure floating-point detection model can be trained. Of course, training a good detection model is not our ultimate goal, it just serves as a pretrain for us to train the fixed-point model later.

Quantitative Model Training

Once we have a pure floating-point model, we can start training the corresponding fixed-point model. In the same way as the floating point training, we can train the fixed point model simply by running the following script.

python3 tools/train.py --stage calibration --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py

python3 tools/train.py --stage qat --config configs/detection/deform_detr/deform_detr_resnet50_mscoco.py

As you can see, our configuration file has not changed, only the type of stage. Deformable DETR is recommended to be added to the calibration process during quantization training. Calibration can provide a better initialization parameter for quantization training of QAT. At this point we use the training strategy from the calibration_trainer and qat_trainer in the config file.

calibration_batch_processor = copy.deepcopy(val_batch_processor)
calibration_step = 100

calibration_trainer = dict(
    type="Calibrator",
    model=model,
    model_convert_pipeline=dict(
        type="ModelConvertPipeline",
        qat_mode="fuse_bn",
        qconfig_params=dict(
            activation_calibration_observer="mse",
        ),
        converters=[
            dict(
                type="LoadCheckpoint",
                checkpoint_path=os.path.join(
                    ckpt_dir, "float-checkpoint-best.pth.tar"
                ),
            ),
            dict(type="Float2Calibration", convert_mode=convert_mode),
        ],
    ),
    data_loader=calibration_data_loader,
    batch_processor=calibration_batch_processor,
    num_steps=calibration_step,
    device=None,
    callbacks=[
        stat_callback,
        val_callback,
        ckpt_callback,
    ],
    val_metrics=dict(
        type="COCODetectionMetric",
        ann_file="./tmp_data/mscoco/instances_val2017.json",
    ),
    log_interval=calibration_step / 10,
)

We first obtain a good initialization parameter for QAT through calibration, calibration loads the trained floating-point model, and then searches for quantization parameters using mse's calibration, where Float2Calibration converts the model from a floating-point model to a calibrated model.

qat_trainer = dict(
    type="distributed_data_parallel_trainer",
    model=model,
    model_convert_pipeline=dict(
        type="ModelConvertPipeline",
        qat_mode="fuse_bn",
        qconfig_params=dict(
            activation_qat_qkwargs=dict(
                averaging_constant=0,
            ),
            weight_qat_qkwargs=dict(
                averaging_constant=1,
            ),
        ),
        converters=[
            dict(type="Float2QAT", convert_mode=convert_mode),
            dict(
                type="LoadCheckpoint",
                checkpoint_path=os.path.join(
                    ckpt_dir, "calibration-checkpoint-best.pth.tar"
                ),
            ),
        ],
    ),
    data_loader=qat_data_loader,
    optimizer=dict(
        type="custom_param_optimizer",
        optim_cls=torch.optim.AdamW,
        model=model,
        optim_cfgs=dict(lr=1e-5, weight_decay=0),
        custom_param_mapper=dict(
            norm_types={"weight_decay": 0.0},
        ),
    ),
    batch_processor=batch_processor,
    num_epochs=6,
    device=None,
    callbacks=[
        stat_callback,
        loss_show_update,
        grad_callback,
        dict(
            type="StepDecayLrUpdater",
            lr_decay_id=[5],
            step_log_interval=500,
        ),
        val_callback,
        ckpt_callback,
    ],
    train_metrics=dict(
        type="LossShow",
    ),
    val_metrics=dict(
        type="COCODetectionMetric",
        ann_file="./tmp_data/mscoco/instances_val2017.json",
    ),
)

Next, we perform QAT quantization training based on the calibration model. The initial learning rate is set to one-tenth of the floating-point training, and the number of training epochs is greatly reduced. Notice that Float2QAT in converter converts the model from a floating point model to a QAT model and then the loaded calibration model weights. After quantization training, the accuracy of the Deformable DETR quantization model can reach more than 99% of the accuracy of the floating-point model.

On This Page

The hb_compile Tool

QAT

Model Export

Horizon Operator

Vision Algorithm

Sample

VP API Introduction

Data Structure

Function Interface

Model Inference API Introduction

Data Type and Data Structure

API Interface

Model Inference Tool Introduction

High Performance Algorithm

HPL API Introduction

Data Structure

Functional Interface

Data Structure

Functional Interface

FRAMEWORK

TUTORIALS

EXAMPLES

API REFERENCE

#Deformable DETR Detection Model Training

#Training Process

#Dataset Preparation

#Model Training

#Exporting Fixed-point Model

#Model Validation

#Model Inference

#Simulation on Board Accuracy Verification

#Fixed-point Model Checking and Compilation

#Training Details

#Model Structure

#Data Enhancement

#Training Strategy

#Quantitative Model Training

Deformable DETR Detection Model Training

Training Process

Dataset Preparation

Model Training

Exporting Fixed-point Model

Model Validation

Model Inference

Simulation on Board Accuracy Verification

Fixed-point Model Checking and Compilation

Training Details

Model Structure

Data Enhancement

Training Strategy

Quantitative Model Training