#Bev Multi-task Model Training

The BEV reference algorithm is developed based on Horizon Torch Samples (Horizon's own deep learning framework), and you can refer to the Horizon Torch Samples usage documentation for an introduction to the use of Horizon Torch Samples. The training config for the BEV reference algorithm is located under the HAT/configs/bev/ path. The following part takes HAT/configs/bev/bev_ipm_efficientnetb0_multitask_nuscenes.py as an example to describe how to configure and train the BEV reference algorithm.

#Training Process

If you just want to simply train the BEV model, then you can read this section first.

Similar to other tasks, HAT performs all training tasks and evaluation tasks in the form of tools + config.

After preparing the original dataset, take the following process to complete the whole training process.

#Dataset Preparation

Here is an example of the nuscense dataset, which can be downloaded from https://www.nuscenes.org/nuscenes . Also, in order to improve the speed of training, we have done a packing of the original jpg format dataset to convert it to lmdb format. Just run the following script and it will be successful to achieve the conversion.

The above two commands correspond to transforming the training dataset and the validation dataset, respectively. After the packing is completed, the file structure in the data directory should look as follows.

The train_lmdb and val_lmdb are the training and validation datasets after packaging, and are the datasets that the network will eventually read. metas is the map information needed for the segmentation model.

#Model Training

Before the network starts the training, you can first calculate the number of network operations and parameters using the following command:

The next step is to start the training. Training can also be done with the following script. Before training, you need to make sure that the dataset path specified in the configuration has already been changed to the path of the packaged dataset.

Since the HAT algorithm package uses the registration mechanism, it allows each training task to be started in the form of train.py plus a config file. The train.py is a uniform training script and independent of the task, and the tasks we need to train, the datasets we need to use, and the hyperparameter settings related to training are all in the specified config file.

The parameters after --stage in the above command can be "float", "calibration", which, respectively, indicates the training of the floating-point model and the quantitative model, and the conversion of the quantitative model to the fixed-point model, where the training of the quantitative model depends on the floating-point model produced by the previous floating-point training.

#Export FixedPoint Model

Once you've completed your quantization training, you can start exporting your fixed-point model. You can export it with the following command:

#Model Verification

After completing the training, we get the trained floating-point, quantitative, or fixed-point model. Similar to the training method, we can use the same method to complete metrics validation on the trained model and get the metrics of Float, Calibration, and Quantized, which are floating-point, quantitative, and fully fixed-point metrics, respectively.

Similar to the model training, we can use --stage followed by "float", "calibration", to validate the trained floating-point model, and quantitative model, respectively.

The following command can be used to verify the accuracy of a fixed-point model, but it should be noted that hbir must be exported first:

#Model Inference

HAT provides the infer_hbir.py script to visualize the inference results for the fixed-point model:

#Simulation Board Accuracy Verification

In addition to the above model validation, we provide an accuracy validation method identical to the on-board environment, which can be accomplished by:

#Fixed-point Model Checking and Compilation

As the quantitative training toolchain integrated in HAT is mainly prepared for Horizon's processors, it is a must to check and compile the quantitative models.

We provide an interface for model checking in HAT, which allows the user to define a quantitative model and then check whether it can work properly on the BPU first.

After the model is trained, you can use the compile_perf_hbir script to compile the quantitative model into an HBM file that supports on-board running. The tool can also predict the performance on the BPU.

The above is the whole process from data preparation to the generation of quantitative and deployable models.

#Training Details

In this note, we explain some things that need to be considered for model training, mainly including settings related to config.

#Model Construction

Where type under model indicates the name of the defined model, and the remaining variables indicate the other components of the model. The advantage of defining the model this way is that we can easily replace the structure we want. For example, if we want to train a model with a backbone of resnet50, we just need to replace backbone under model .

#Data Augmentation

Like the definition of model, the data enhancement process is implemented by defining two dicts data_loader and val_data_loader in the config configuration file, corresponding to training set and the processing flow of the validation set. Take data_loader as an example.

Where type directly uses the interface torch.utils.data.DataLoader that comes with pytorch, which represents the combination of batch_size size images together. The only thing to be concerned about here is probably the dataset variable, CocoFromLMDB means reads the image from the lmdb dataset, and the path is the same path we mentioned in the first part of the dataset preparation. transforms contains a series of data enhancements underneath. Except for the image flip (RandomFlip), the other data transformations of the val_data_loader are the same as data_loader . You can also achieve the data augmentation you want by inserting a new dict in transforms.

#Training Strategies

In order to train a model with high accuracy, a good training strategy is essential. For each training task, the corresponding training strategy is defined in the config file as well, as can be seen from the variable float_trainer.

The float_trainer defines our training approach in the big picture, including the use of distributed_data_parallel_trainer, the number of epochs for model training, and the choice of optimizer. Also, the callbacks reflect the small strategies used by the model during training and the operations that the user wants to implement, including the way to transform the learning rate (WarmupStepLrUpdater), the metrics to validate the model during training (Validation), and the operations to save (Checkpoint) the model. Of course, if you have operations that you want the model to implement during training, you can also add them in this dict way.

With the above introduction, you should have a clearer understanding of the functions of the config file. Then you can train a high-precision pure floating-point detection model by the training script mentioned earlier. Of course, training a good detection model is not our ultimate goal, it is only used as a pretrain for us to train a fixed-point model later.

#Quantized Model Training

Once we have a pure floating-point model, we can start training the corresponding fixed-point model. As with floating-point training, we only need to run the following script to get a pseudo-quantization model, which can achieve the goal using only calibration.

As you can see, our config file has not changed, only the type of stage has been changed. The training strategy we use at this point comes from calibration_trainer in the config file.

#The Value of Quantize Parameter is Different

When we train the quantized model, we need to set quantize=True, at this time the corresponding floating point model will be converted into a quantized model, the relevant code is as follows.

For key steps in quantization training, such as preparing the floating-point model, operator substitution, inserting quantization and inverse quantization nodes, setting quantization parameters, and fusing operators, please read the Quantized Awareness Training (QAT) section.

#Different Training Strategies

As we said before, quantization training is in fact finetune based on pure floating-point training, so when quantization training, our initial learning rate is set to one-tenth of the floating-point training, the number of epochs for training is largely reduced, most importantly, when defining the model , our pretrained needs to be set to the address of a pure floating-point model that has already been trained.

After making these simple adjustments, we can start training our quantitative model.