Floating Point Passes, Quantization Fails: A Rewrite Gives the Output the Input’s Identity

Series contents · Numerics and Quantization · 阅读中文版

“The floating-point model compiles. The quantized one cannot get through the first stage.” Someone starts suspecting backend integer instructions. But the failure happens too early: before instruction emission, the function’s return type already mismatches.

The cause is small: a new node takes the correct shape but the wrong element type. It is like updating an address after moving house, then accidentally copying the neighbor’s name onto the record.

1. A Fold That Looks Entirely Reasonable

A graph may contain:

1
2
3
4

original channel tensor
  → reshape into a tensor organized by groups
  → InstanceNorm
  → reshape back to the original shape

When grouping, parameter broadcasting, and shape conditions hold, this can fold into GroupNorm directly on the original tensor. Removing the helper reshapes makes the graph more direct.

For example, take input [1,10,3,7] divided into 5 groups of 2 channels. Each group’s spatial and channel elements can be organized together for statistics. The folded output shape should return to [1,10,3,7].

But restoring the shape does not answer which quantization domain to use. Normalization can substantially change the data distribution, so its output scale need not equal its input scale.

2. Floating-Point Types Hide the Mistake

In an ordinary floating-point path, the input uses f32, and the normalization output also uses f32. A type-construction function obtains f32 whether it takes the element type from the input or the original output. The test passes.

In a quantized path, the element type may include the storage integer type, expressed real type, scale, and zero point. Two tensors both using 8-bit integers need not have equal types.

For example:

1
2

input element type: signed 8-bit, scale 0.02, zero 0
normalization output: signed 8-bit, scale 0.08, zero 0

If the rewritten GroupNorm takes the input element type, its result carries 0.02 while the function promises 0.08. A verifier may detect the error immediately, or permissive intermediate conversions may hide it until a later node.

Historical diffs show that quantization information had already been imported before canonicalize. The assumption that an early high-level graph cannot yet contain quantized types was false. A stage’s name is not a guarantee; its contract is.

3. A Result Type Is an Obligation, Not a Guess from the Input

A central equivalence condition for graph rewriting is that the new expression satisfies the old expression’s observable result contract. That includes numerical representation, layout or encoding information, and external interface requirements—not just element count.

A source table for constructing a folded result is:

“Copy the input tensor type, then change its shape” is convenient and common, but unsuitable for an operation that changes numerical domains. Better helper names or parameters should force the caller to state exactly what is inherited.

This does not mean always copying the full output type. Intermediate reshapes and permutations may need shapes in the current coordinate system, and some layout encodings change with dimension order. Derive each component; do not replace “the input type was wrong” with “blindly use the output type everywhere.”

4. Tests Must Deliberately Distinguish Input and Output

An existing historical test used the same scale for input, normalization output, and function return. It validated the graph fold but could not distinguish where the element type came from.

The added regression used different input and output scales and explicitly checked the folded GroupNorm’s quantized result type. Correct and incorrect implementations then produced different observable results—the central purpose of a regression test.

Think of the test as creating a fork:

1
2
3
4

If scale_in == scale_out:
    copying the input type and copying the output type give the same result.
If scale_in != scale_out:
    the implementations diverge, and the test can distinguish them.

Likewise, to expose incorrect parameter broadcasting across groups, do not give every group the same gamma. To expose axis permutations, do not make every dimension equal. To expose lost zero points, do not test only zero point zero. Tests should deny errors the shelter of coincidence.

5. A Compilation Error Can Be the Cheapest Numerical Safeguard

A return-type mismatch sounds like a compilation-experience problem. In fact, it prevents a numerical-domain error from entering the backend. Once an input integer is interpreted with the wrong scale, every downstream operator may follow its own type correctly while the overall computation deviates from the model.

Run type verification early and after important rewrites. Compiler developers are often tempted to insert a cast simply to silence a mismatch. That is justified only when its numerical conversion is understood and preserves the original semantics. Forcing the type to match does not automatically change the meaning of the stored integers.

If the output contract already specifies the original output type, construct the right result directly rather than disguising the mismatch afterward. The historical patch did the former.

6. Test the Pattern’s Relationship with Its Neighbors

The problem crossed two stages: quantization import and canonicalization. Testing a pattern only on floating-point IR rarely exposes it; testing quantization import without folding does not trigger it either. Keep isolated stage tests and small composition tests.

Also inspect whether a failed path modifies the original operation before returning failure. A rewrite should ideally validate enough information before committing changes; otherwise even a pattern that reports no match may leave unexpected attributes. Related historical work changed some attribute handling to construct attributes on the new node instead of modifying the old one first. Such details help maintain rewrite discipline.

Correcting a type source does not usually increase execution speed directly; it restores correctness and compilability. Folding may remove helper nodes, but a reshape that was only a view did not move data, so fewer nodes do not establish fewer memory accesses.

The revealing point is that no elaborate mathematical derivation went wrong. Identical storage width concealed different numerical types. Once types carry scales, copying a type is no longer clerical work: it copies a numerical promise.

7. Make Correct Type Propagation a Lasting Rule

A fix can easily stop at replacing “input” with “output” on one line. To prevent the same error returning under another operator, classify rewrite helpers semantically: pure views preserve numerical element types; actual computation nodes use their result contracts; transformations that alter layout encodings explicitly remap them. A caller should see the category in the parameters, rather than guessing from “like” in a function name.

This does not require long helpers. It means avoiding a convenience interface that promises too much at once. A helper copying shape, element type, and encoding can easily be misused by a caller intending to inherit only one. Expressing independent choices is often more reliable than a comment saying “watch quantized types.”

Tests can vary type provenance directly. Hold the graph structure fixed and change only the original computation’s output scale; the rewritten result type must follow. Hold the output contract fixed and change only the input scale; the folded result must not follow it without justification. These two tests pull on different threads and reveal which source controls the result. The same method applies separately to zero points or storage signedness.

At a function-boundary error, trace backward from the return value to the first place the type changed, instead of initially examining every quantization parameter. If the original normalization output is correct and the folded node is wrong, the search has narrowed to the rewrite. If the original output is already wrong, return to import. This uses discrete evidence from the type system before undertaking a complete numerical error analysis.

Check every external consumer’s input requirements before and after rewriting. A result may be both returned and consumed by another computation. Fixing only a conversion at the function return can leave the other path incorrect. Preserving the replaced value’s complete result contract protects all consumers, instead of patching each edge independently.

Finally, avoid tests that rely on an accidental later fold. If the final type becomes visible only after running the entire optimization pipeline, identifying which stage repaired or damaged it is difficult. An isolated regression for the core pattern and a small regression for the stage combination provide both localization and assurance that the actual ordering still works. They serve different purposes; not every model needs to become an enormous end-to-end test.

Back to series contents · Previous · Next

• ← Previous Post
• Next Post →