#lidarMultiTask Model Training

This tutorial focuses on how to use HAT to train a lidarMultiTask model on the lidar point cloud dataset nuscenes from scratch, including floating-point, QAT and quantized models.

#Dataset Preparation

Before starting to train the model, the first step is to prepare the dataset, download the full dataset (v1.0) and nuScenes-lidarseg from nuscenes dataset.

After downloading, unzip and organize the folder structure as follows which can refer to nuscenes Tutorials

To improve the speed of training, we did a package of data information files to convert them into lmdb format datasets from nuscenes lidar files. The conversion can be successfully achieved by simply running the following script:

The above two commands correspond to transforming the training dataset and the validation dataset, respectively. After the packing is completed, the file structure in the data directory should look as follows:

train_lmdb and val_lmdb are the packaged training and validation datasets, which are also the final datasets read by the network. meta contains information for metrics initialization, copying from nuscenes.

Also, for nuscenes point cloud data training, it is necessary to generate database files for each individual training target in the dataset and store it using a .bin format file in tmp_nuscenes/lidar/nuscenes_gt_database. The path can be modified as needed. At the same time, a file containing these database information in .pkl format needs to be generated. In addition, we need to collect category information of all samples in train dataset and resample the whole dataset, thus we can generate these information and save them in .pkl format to accelerate training. Then, create these files by running the following command:

After executing the above command, the following file directory is generated:

nuscenes_gt_database and nuscenes_dbinfos_train.pkl are the samples that are used for sampling during training, and nuscenes_infos_train.pkl can be used to accelerate train dataset initialization.

#Floating-point Model Training

Once the dataset is ready, you can start training the floating-point lidarMultiTask network. Before the network training starts, you can test the computation and the number of parameters of the network by using the following command.

If you simply want to start such a training task, just run the following command:

Since the HAT algorithm package uses an ingenious registration mechanism, each training task can be started in the form of this train.py plus the config configuration file. The train.py is the unified training script has nothing to do with the task. What kind of task we need to train, what kind of dataset to use, and training-related hyperparameter settings are all in the specified config configuration file. The config file provides key dicts such as model building and data reading.

#Model Building

The network structure of lidarMultiTask can refer to CenterPoint model with modified neck and additional segmentation head. You can find details in the config file. We can easily define and modify the model by defining a dict-type variable such as model in the config configuration file.

Among them, the type under model means the name of the defined model, and the remaining variables mean the other components of the model. The advantage of defining the model this way is that we can easily replace the structure we want. After starting, the training script calls the build_model interface to convert such a model of type dict into a model of type torch.nn.Module.

#Data Enhancement

Like the definition of model, the data enhancement process is implemented by defining two dicts data_loader and val_data_loader in the config file, which correspond to the processing of the training and validation sets, respectively. Here we take data_loader as an example:

Here, the type directly uses the interface torch.utils.data.DataLoader that comes with pytorch, which represents the combination of batch_size size samples together. Here you may only need to pay attention the dataset variable, and the data_path path is the path we mentioned in the first part of the dataset preparation. transforms contains a series of data augmentations. In val_data_loader, only point cloud reading, segmentation labels generation and data reformat are available. You can also implement your desired data augmentation operations by inserting a new dict in transforms.

#Training Strategy

To train a model with high accuracy, a good training strategy is essential. For each training task, the corresponding training strategy is also defined in the config file, which can be seen from the variable float_trainer.

The float_trainer defines our training approach from the big picture, including the use of distributed_data_parallel_trainer, the number of epochs for model training, and the choice of optimizer. At the same time, callbacks reflects the small strategies used by the model in the training process and the operations that the user wants to implement, including the transformation method of the learning rate (CyclicLrUpdater), the indicator (Validation), and save (Checkpoint) the operation of the model. Of course, if you have operations that you want the model to implement during training, you can also add it in this dict way. float_trainer is responsible for concatenating the entire training logic, which is also responsible for model pretraining.

Through the above introduction, you should have a clear understanding of the role of the config file. Then, through the training script mentioned above, a high-precision pure floating-point model can be trained. Of course, training a good model is not our ultimate goal, it is just a pre-training for our future training of quantitative models.

#Quantitative Model Training

When we have a floating-point model, we can start training the corresponding QAT model. In the same way as floating-point training, we can train a QAT model just by running the following script: BTW, it is recommended to add a calibration stage before quantization aware training. Calibration can provide better initialization parameters for QAT.

As you can see, our configuration file has not changed, only the type of stage has been changed. At this point, the training strategy we use comes from the qat_trainer in the config file.

#With Different model_convert_pipeline Parameters

By setting model_convert_pipeline when training quantitative models, the corresponding floating-point model can be converted into a quantitative model, as below:

For the key steps in quantitative training, such as preparing floating-point models, operator replacement, inserting quantization and dequantization nodes, setting quantization parameters, and operator fusion, please read the Quantized Awareness Training (QAT) section.

#With Different Training Strategies

As we said before, the quantitative training is actually finetue on the basis of pure floating-point training. Therefore, when quantized training, our initial learning rate is set to half of the floating-point training, the number of epochs for training is also reduced, and most importantly, when model is defined, our pretrained needs to be set to the address of the pure floating-point model that has been trained or the calibration model.

After making these simple adjustments, we can start training our quantitative model.

#Export FixedPoint Model

Once you've completed your quantization training, you can start exporting your fixed-point model. You can export it with the following command:

#Model Validation

After the model is trained, we can also validate the performance of the trained model. Since we provide two stages of training process, float and qat, we can validate the performance of the model trained in these two stages. Run the following two commands:

At the same time, we also provide a performance test of the quantization model, just run the following command, but it should be noted that hbir must be exported first:

The displayed accuracy is the real accuracy of the final int8 model. Of course, this accuracy should be very close to the accuracy of the qat verification stage.

#Simulation of On-board Accuracy Validation

In addition to the model validation described above, we offer an accuracy validation method that is identical to the board side, you can refer to the following:

#Model Inference and Results Visualization

If you want to see the detection effect of the trained model for a lidar point cloud file, we also provide point cloud prediction and visualization scripts in our tools folder, you just need to run the following script.

#Model Checking and Compilation

After training, the quantized model can be compiled into an hbm file that can be run on the board by using the compile_perf_hbir tool. At the same time, the tool can also estimate the running performance on the BPU. The following scripts can be used: