#nn.Correlation

#class horizon_plugin_pytorch.nn.Correlation(kernel_size: int = 1, max_displacement: int = 1, stride1: int = 1, stride2: int = 1, pad_size: int = 0, is_multiply: bool = True)

Perform multiplicative patch comparisons between two feature maps.

Correlation performs multiplicative patch comparisons between two feature maps. Given two multi-channel feature maps $f_{1}, f_{2}$, with $w$, $h$, and $c$ being their width, height, and number of channels, the correlation layer lets the network compare each patch from $f_{1}$ with each patch from $f_{2}$.

For now we consider only a single comparison of two patches. The ‘correlation’ of two patches centered at $x_{1}$ in the first map and $x_{2}$ in the second map is then defined as:

$c(x_{1}, x_{2}) = \displaystyle\sum_{o \in [-k,k] \times [-k,k]} <f_{1}(x_{1} + o), f_{2}(x_{2} + o)>$

for a square patch of size $K:=2k+1$.

Note that the equation above is identical to one step of a convolution in neural networks, but instead of convolving data with a filter, it convolves data with other data. For this reason, it has no training weights.

Computing $c(x_{1}, x_{2})$ involves $c * K^{2}$ multiplications. Comparing all patch combinations involves $w^{2}*h^{2}$ such computations.

Given a maximum displacement $d$, for each location $x_{1}$ it computes correlations $c(x_{1}, x_{2})$ only in a neighborhood of size $D:=2d+1$, by limiting the range of $x_{2}$. We use strides $s_{1}, s_{2}$, to quantize $x_{1}$ globally and to quantize $x_{2}$ within the neighborhood centered around $x_{1}$.

The final output is defined by the following expression:

$out[n, q, i, j] = c(x_{i, j}, x_{q})$

where $i$ and $j$ enumerate spatial locations in $f_{1}$, and $q$ denotes the $q^{th}$ neighborhood of $x_{i,j}$.

• Parameters:
  • kernel_size – kernel size for Correlation must be an odd number
  • max_displacement – Max displacement of Correlation
  • stride1 – stride1 quantize data1 globally
  • stride2 – stride2 quantize data2 within neighborhood centered around data1
  • pad_size – pad for Correlation
  • is_multiply – operation type is either multiplication or subduction, only support True now

#forward(data1: Tensor | QTensor, data2: Tensor | QTensor)

Forward for Horizon Correlation.

• Parameters:
  • data1 – shape of [N,C,H,W]
  • data2 – shape of [N,C,H,W]
• Returns: output
• Return type: Tensor