Once BatchNorm Becomes Multiply-Add, the Numerical Work Begins

Series contents · Numerics and Quantization · 阅读中文版

“BatchNorm has already been folded into y=ax+b. Surely all that remains is a multiplication and an addition?” Everyone nods. Minutes later, the questions are: “How many bits for a? Whose scale does b use? Do we truncate after multiplying? If a is close to 1, can we skip it?”

The formula has not become more complicated. It has finally met finite precision.

1. Inference-Time Normalization Can Fold into an Affine Transform

With fixed inference statistics, per-channel BatchNorm can be written as:

1
2
3
4

y_c = gamma_c * (x_c - mean_c) / sqrt(var_c + epsilon) + beta_c
a_c = gamma_c / sqrt(var_c + epsilon)
b_c = beta_c - mean_c * a_c
y_c = a_c * x_c + b_c

Fixed statistics are a prerequisite. During training, statistics must be calculated from the current batch and state updated; this folding cannot simply be applied.

The relevant historical implementation accepted this affine form: it read multiplication and bias constants, produced integer constants and quantized types, and lowered them into target computations. An earlier approach used a dedicated affine operation; later work moved to general multiplication and addition. Both approaches owe the same numerical contract.

2. Parameters Have Quantization Domains Too

Let the input be x=s_x(q_x-z_x) and the coefficient a=s_a(q_a-z_a). Being a model constant does not allow a to be truncated to an integer and multiplied directly.

Take teaching coefficients [-0.8,0.25,1.6]. With symmetric signed storage and a selected positive range Q, a common construction is:

1
2

s_a = max(abs(a)) / Q
q_a = clamp(round(a/s_a))

A symmetric signed domain normally uses zero point zero. Unsigned storage accommodating both positive and negative coefficients needs real zero placed inside the available range. Historical constant-quantization logic chose the zero point from storage signedness and range, derived scale from the absolute-value range, then rounded and clamped.

This maximum-absolute-value approach is straightforward, but one large channel forces smaller coefficients to share a coarser grid. Per-channel scales may reduce error while requiring more complex backend parameters and broadcasting. Precision comparisons should not stop at “8 bits or 16 bits.”

All-zero parameters also need a defined scale to avoid division by zero. A positive fallback scale can represent them, but the generated integers must still equal the zero point representing real zero. Writing integer 0 into every unsigned element is not automatically correct.

3. Where the Intermediate Product Lands Determines Whether Bias Can Repair It

If the multiplication result is immediately quantized into the final output type before adding bias, the computation is:

1
2

q_mid = Q_out(a*x)
q_y   = Q_out(D_out(q_mid) + b)

Q and D denote quantization and dequantization. Retaining a wider intermediate and quantizing only at the end instead gives:

1

q_y = Q_out(a*x + b)

These are generally different. Consider an output real-value upper bound of 10, a product of 12, and bias -4. Saturating after multiplication produces 10, then adding -4 gives 6. Saturating only at the end gives 8. A later negative bias cannot recover the 2 already clipped away.

Historical implementation paths used the output type for the multiplication intermediate. A public analysis should therefore ask about intermediate ranges and rounding locations, not treat algebraic equality as proof of bitwise equality. Acceptability depends on the target’s quantization specification and the model’s error budget.

A wider intermediate increases storage and bandwidth. A fused affine operation may avoid one quantization step, but needs execution-unit or kernel support for the semantics. Choosing a route requires accounting for accuracy, resources, and maintenance together.

4. “Almost One” Does Not Always Make Multiplication Removable

Some historical versions selected simplified paths when coefficients were close to 1 and biases close to 0. Mathematically, a=1 and b=0 leave only requantization from the input to the output domain. With a=1 and b≠0, retaining only addition may be possible.

“Close to 1,” however, is an approximate optimization whose error depends on input magnitude:

1

real-value error from omitting multiplication = (a-1)*x

A small |a-1| multiplied by a large |x| can still cross one or more output quantization steps. A constant tolerance should relate to the data range or quantized equivalence, rather than being chosen in isolation.

A stronger proof would show that the original and simplified expressions produce identical output integers for every permitted input integer. Small widths permit exhaustive checking; larger spaces require interval reasoning or explicitly bounded error analysis.

Later historical versions changed these shortcuts. An approximate-identity optimization in an intermediate version should not be described as a permanent final capability. Reading the history helps avoid mistaking an old signpost for today’s road.

5. Power-of-Two Scales: Trading Precision for Implementation Constraints

Another historical change snapped a constant scale to a power of two under a configuration condition. A general motivation is to fit shifts or restricted multiplier paths, but the concrete benefit depends on the target’s execution model.

If an original scale s is increased to a nearby power of two s’, coverage generally widens and the grid becomes coarser. With nearest rounding and no saturation, the absolute quantization error of one constant is roughly bounded by s’/2. That is only the constant’s own error; multiplying it by an activation can amplify it.

Compare three strategies: unrestricted scales, power-of-two scales, and per-channel scales. Look beyond final mean error to maximum error, extreme channels, saturation frequency, and whether the multiplier is legally representable. “Easier to generate” is not a cost-free advantage.

6. A Numerical Regression Matrix

The history includes compilation entry points and constant processing; this does not establish that all numerical regressions passed. A reference implementation must specify the same inference statistics, epsilon, rounding, and saturation order. Otherwise a specification difference may be mistaken for a kernel error.

Affine computation generally reads and writes elements individually. Reusing parameters per channel makes additional parameter volume grow with C; expanding them into a full feature map makes it grow with NCHW. The next article examines why broadcasting code written for simplicity can deserve more optimization attention than the arithmetic itself.

A short multiply-add formula does not mean the engineering is nearly finished. It concentrates the numerical responsibilities into something we can state and test: how constants are represented, where intermediates are rounded, and where the result saturates.

7. Propagate Parameter Error through the Formula

Write the quantized coefficient as a plus error da, and the bias as b plus error db. Ignoring later rounding and saturation, the output error is da times the input plus db. Small coefficient errors therefore grow with input magnitude, while bias errors shift the entire channel’s curve.

This suggests a concrete test design: sample both ends of the input interval and values near zero, rather than relying only on random activations. Interval endpoints amplify coefficient errors; values near zero reveal bias and zero-point problems. If a channel’s error is nearly constant, inspect bias first. If it grows linearly with input, inspect the coefficient. If it breaks abruptly at a magnitude threshold, inspect saturation. These patterns provide more diagnostic information than maximum absolute error alone.

With one common parameter scale, plot the coefficient differences before and after quantization by channel. Large coefficients typically occupy most of the dynamic range, while small ones may quantize to zero. Overall mean squared error need not be large, yet some channels can change qualitatively. Channel-wise error analysis is a natural requirement for per-channel affine transforms, not something replaceable by one whole-tensor average.

Bias range may also be poorly served by simply copying the activation storage width. Bias can cancel a large mean term and have a magnitude beyond the final output range. If the bias constant saturates during materialization, even a perfectly correct later addition operates on damaged parameters. Distinguishing saturation during constant storage from saturation of a computed result helps locate the first stage at which error appears.

A design review can request four visible quantities: coefficient dequantization error, bias dequantization error, multiplication intermediate range, and final output saturation rate. The first two can be checked during constant generation; the latter two need input ranges or runtime samples. Passing one does not substitute for the other three.

The same error-propagation formula can set a threshold for approximately skipping multiplication by one. Given a bound on absolute input magnitude, the worst real-value error from the coefficient deviation must fit the allowed budget. If bitwise integer identity is required, rounding boundaries matter too: being below half an output step is not always sufficient, because the original result may lie immediately next to a boundary.

Back to series contents · Previous · Next

• ← Previous Post
• Next Post →