#Calibration

In QAT, an important stage is to calculate the quantitative parameter scale. A good scale can significantly improve the accuracy of the model training results and speed up the convergence of the model. The calibration process is to run a few batch data on floating-point models (only run the forward process, not backward), count the distribution histogram and get min_value and max_value. Then, use the min_value and max_value to calculate the scale. When the QAT accuracy is low, calibrating the quantitative parameters like this before QAT can provide better quantitative initilization parameters.

#How to Define Calibration Model

• No need to change existing models by default

  Similar to the setting of QAT QConfig when defining a quantitative model, it is also necessary to set Calibration QConfig when defining a calibration model. However, the latter is simpler as the HAT has already implemented the default settings for Calibration QConfig for direct use without any modification to the model.

• Define submodule Calibration QConfig

  By default, Calibration QConfig is set for all modules (inherited from nn.Module) of the model. Therefore, the calibration will count the feature distribution of all modules. If you have special needs, you can customize the implementation of the set_calibration_qconfig method in the model:

  class Classifier(nn.Module): def __init__(self,): ... def forward(self, x): ... # Customize the module to be calibrated def set_calibration_qconfig(self, ): # e.g., you can set the qconfig of loss to None to skip # the calibration on loss. # It can relatively reduce unnecessary statistics, reduce # the video memory usage and speed up the calibration. if self.loss is not None: self.loss.qconfig = None

#Run Calibration on Floating-point Models

HAT comes with built-in calibration functions, whose commands are similar to those in normal training. Simply run the following command:

See the calibration_trainer settings in the config file:

1. Setting Dataset:

The datasets for the calibration cannot be testing datasets (can be training datasets or others). There is no definite conclusions for now about the transforms for data enhancement, you can try either using the transforms consistent with those in normal training or validation, or using the customized transforms.

2. Number of images to be iterated in calibration (for reference):

• classification: 500~1500 images
• segmentation && detection: 100~300 images

#QAT with Calibration Model

Setting of averaging_constant:

In the QAT, the update rule for the scale parameter is scale = (1 - averaging_constant) * scale + averaging_constant * current_scale .

It is found in some existing experiments that it may lead to higher accuracy by fixing the scale of the activation after calibration, i.e., by setting averaging_constant=0 of the activation and averaging_constant=1 of the weight.

Next, you only need to start the training by executing the normal QAT command: