English

Eager Mode

As with PyTorch, we recommend using the fx quantization mode as your preferred mode. For quantization in eager mode, currently you are still supported by our horizon_plugin_pytorch. The overall flow of the Eager mode is also based on PyTorch's quantization interface and ideas, so we recommend that you read the Eager mode section of the PyTorch official document.

Difference from fx mode

The main differences between using eager mode and fx mode in horizon_plugin_pytorch are:

The eager mode only supports module operators. Before carryout model quantization, you need to manually replace the functional operators in the floating-point model with Module-type operators in PyTorch or proprietary operators defined in horizon_plugin_pytorch, including but not limited to:

Original floating-point operator	Operator to be replaced
torch.nn.functional.relu	torch.nn.ReLU()
a + b / torch.add	horizon.nn.quantized.FloatFunctional().add
Tensor.exp	horizon.nn.Exp()
torch.nn.functional.interpolate	horizon.nn.Interpolate()

You must manually define the operators to be fused and explicitly call the fusion function before carryout model quantization, and also specify the use of the fuser_func provided in horizon_plugin_pytorch when calling it. As shown below:

import torch
from torch import nn
import horizon_plugin_pytorch as horizon


class ConvBNReLU(nn.Sequential):
    def __init__(self, in_channels, out_channels, kernel_size):
        super(ConvBNReLU, self).__init__(
            nn.Conv2d(
            in_channels=in_channels,
            out_channels=out_channels,
            kernel_size=kernel_size
            ),
            nn.BatchNorm2d(num_features=out_channels),
            nn.ReLU()
        )

    # specify the operators that can be fuse
    def fuse_model(self):
        torch.quantization.fuse_modules(
            self,
            ['0', '1', '2'],
            inplace=True,
            # specify the fuse function provided in horizon_plugin_pytorch
            fuser_func=horizon.quantization.fuse_known_modules,
        )

float_model = ConvBNReLU(1, 1, 1)
# call the fuse function explicitly
float_model.fuse_model()

print(float_model)
# ConvBNReLU(
#   (0): ConvReLU2d(
#     (0): Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1))
#     (1): ReLU()
#   )
#   (1): Identity()
#   (2): Identity()
# )

The other steps are consistent with FX mode, please refer to the "Quick Start" chapter.

On This Page

PTQ Conversion Tools

The hb_compile Tool

PTQ Conversion Steps

PTQ Conversion Samples

FAQ and Common Failure Resolutions

Appendix

Tutorial

Advanced Tutorial

API Reference

QAT

Model Export

Horizon Operator

Vision Process

Vision Algorithm

Sample

VP API Introduction

Data Structure

Function Interface

Model Inference Development

Model Inference API Introduction

Data Type and Data Structure

API Interface

Model Inference Tool Introduction

High Performance Library

High Performance Algorithm

HPL API Introduction

Data Structure

Functional Interface

UCP Common API Introduction

Data Structure

Functional Interface

UCP Performance Analysis Tools

Horizon Torch Samples

FRAMEWORK

TUTORIALS

EXAMPLES

API REFERENCE

Toolchain Operator Support Constraint List

Eager Mode

Difference from fx mode

The hb_compile Tool

QAT

Model Export

Horizon Operator

Vision Algorithm

Sample

VP API Introduction

Data Structure

Function Interface

Model Inference API Introduction

Data Type and Data Structure

API Interface

Model Inference Tool Introduction

High Performance Algorithm

HPL API Introduction

Data Structure

Functional Interface

Data Structure

Functional Interface

FRAMEWORK

TUTORIALS

EXAMPLES

API REFERENCE

#Eager Mode

#Difference from fx mode

Eager Mode

Difference from fx mode