The Importer Got Clever Too Soon: Algebraic Optimization before Quantization Metadata

Series contents · Numerics and Quantization · 阅读中文版

“A constant divided by a tensor—wouldn’t a scaled reciprocal be more efficient?” The enthusiastic importer tidies the algebra as soon as the model arrives. Later, a module checking quantization information asks, “Where did the original division go?”

The formula need not be wrong. Its timing may disrupt correspondences that other stages still require.

1. Import, Annotation, and Optimization Cannot Be Arbitrarily Reordered

Suppose the model contains c/x. With suitable numerical-domain and exceptional-value semantics, it can become c*reciprocal(x), or c can become a coefficient attribute of the reciprocal operation.

Historical import code used a special path when the left operand was a single-element weight and the right was not, directly generating the reciprocal form. A fix removed that special case, allowing such inputs to initially use ordinary division.

The diff establishes that the transformation was deferred or, at least, no longer performed at that entry point. It does not independently provide a complete failing example, so a particular quantization-configuration mismatch cannot be described as reproduced fact. A reasonable inference from the stage relationships is that changing the node kind, input count, or constant position too early complicates quantization-annotation matching. That is a hypothesis explaining a design tradeoff, still requiring upstream/downstream code and test verification.

Import can first represent model semantics faithfully, leaving equivalent transformations to dedicated optimization after identity mappings, types, and necessary annotations stabilize. This does not exclude optimization; it gives optimization a better-informed moment to act.

2. Mathematical Equivalence Is Not Metadata Equivalence

Quantization information may bind through node names, tensor names, input positions, or type relationships. Replacing binary division with unary reciprocal plus an attribute changes where metadata lives, even if the real-valued function stays the same.

A complete rewrite contract should answer at least:

1
2
3
4
5

Which new result corresponds to each old result?
Which new input receives each old input's quantization domain?
When a constant becomes an attribute instead of a tensor, must its scale remain?
How is node provenance tracked?
Are exceptional-input and rounding behaviors preserved?

Without answers, an early simplification can force downstream stages to guess through special cases. As those accumulate, importer, quantizer, and lowering patch around one another until no stage can independently explain the complete semantics.

When x approaches zero, reciprocal approximation, division implementation, and quantized saturation may also have different boundary behavior. This is a general numerical risk; real-number algebra alone does not establish bitwise equivalence. State the valid domain and exceptional semantics explicitly.

3. Transposing a Constant Matrix Moves Data; Its Type Must Follow

Another historical change concerned matrix multiplication with a constant left operand. Mathematically:

1

A * B = transpose(transpose(B) * transpose(A))

This is a common transformation when a target conveniently handles constants only in a particular operand position. If A is constant, its data can be transposed at compile time instead of doing that work at runtime.

Creating a new floating-point constant, or taking only the original constant’s storage type, can lose its quantized element type and associated encoding. The historical fix constructed the transposed shape while preserving the original tensor element type and encoding, and updated the corresponding weight data. It also removed some earlier special handling in quantization import.

The diff supports the observation that quantization-import special cases were reduced while type information was preserved at the constant transformation. It does not automatically prove safety for every shared-constant use, or correct remapping of every per-axis quantization encoding.

4. Preserving an Encoding Does Not Prove It Still Names the Right Axis

Let a teaching constant A have shape [M,K], with row-wise quantization parameters along M. After transposition, Aᵀ has shape [K,M], so the same parameter group should correspond to columns.

If the encoding contains a quantization-axis number, copying it can preserve its data while leaving the axis identity unchanged. If it contains channel parameters governed by another convention, verify where that convention is updated. One patch is not a proof of the entire quantization system, so this is a review question, not an assertion of a historical unfixed bug.

Shared uses need the same care. Updating a constant’s shape and data in place affects both consumers if two operators use it. A legal rewrite needs conditions such as a single use, coordinated transformation of every consumer, or a separate constant for the transformed path. Determining which applies requires the full matching preconditions; a few update lines alone establish neither safety nor danger.

Preserving identity prevents metadata loss. Managing sharing controls the scope of a mutation. Both matter, and neither substitutes for the other.

5. When Should a Rewrite Create a Constant or Update One?

A new constant lets different consumers retain their own layouts and fits an immutable-IR style. It requires complete migration of quantized types, encodings, provenance, and data. Updating an existing constant reduces identity changes but depends more heavily on use relationships and transactional handling of mutations.

Possible decision principles are:

These are design recommendations, not substitute descriptions of unread implementation details. They turn “should we keep this name?” into a verifiable question about identity and uses.

6. Cross Stage Boundaries Deliberately in Tests

Division values, quantization import, and matrix transposition can each pass isolated tests while their combination fails. Small, discriminating pipeline tests are useful:

Neither historical fix added the full matrix proposed here in its diff. These are recommendations, not completed validations.

Deferring algebraic transformation may give later optimizations more information, or leave early IR slightly larger. Compilation-time effects require measurement. Constant transposition moves work to compilation and may reduce runtime reordering, while adding constant-storage or compiler-memory costs. If the backend can already avoid reordering through layout interpretation, an explicit transpose may not be preferable.

An importer’s greatest contribution may be handing complete information to the next stage, rather than appearing clever immediately. Once the compiler knows what every number means and who uses each constant, cleverness becomes easier to prove harmless.

7. Give Every Rewrite a Metadata Migration Sheet

A small table can record how many data inputs, parameter inputs, and results the old node has, what the new nodes are, and where each quantization annotation and provenance item moves. If an item has no destination, retain the original representation until the mapping is understood. This is particularly useful when constants become attributes, several operators fuse, or one operator decomposes into a subgraph.

The sheet should also separate information that can be inferred again from information that must be preserved. Shapes can often be inferred from inputs and operations; calibration scales cannot necessarily be recovered from shapes. Constant data can be rearranged, but a name’s relationship to an external quantization table may be impossible to reconstruct from data content. Separating inferable and unrecoverable information helps identify a safe time for rewriting.

For matrix transposition, prove each part separately: the new constant at (k,m) equals the old constant at (m,k); the new shape retains the element count; quantization parameters preserve the corresponding real values; and transposes of the right operand and result restore output indexing. Equal element counts alone prove very little about the computation. They only rule out one particularly crude error.

If a constant has several consumers, list whether each participates in the transformation. A unique name does not imply a unique use: many nodes can refer to one SSA value or weight object. Establishing use relationships before an in-place update is more direct than later investigating why a bypass path changed.

Stage ordering can likewise have explicit preconditions and postconditions. If an optimization requires quantization annotations to have become types, make that an input condition. If a later stage requires parameters to remain tensors, do not absorb them into attributes beforehand. Checking these conditions turns “try swapping the passes” from an empirical maneuver into an explainable design change.

This method does not guarantee performance gains from every transformation. It first guards against information loss. Performance evaluation can then ask how much compilation work preprocessing adds, which runtime operations it removes, and whether it creates additional data copies. Separating correctness of migration from cost comparison makes the discussion much clearer.

Back to series contents · Previous

• ← Previous Post
• Next Post →