Max, Min, and Sum: Similar Operators, Different Mathematical Homework

Series contents · Numerics and Quantization · 阅读中文版

“We already have mean. For sum, remove the division; for max and min, change a symbol. Finished by this afternoon?”

If an operator support list were a restaurant menu, that would be an attractive proposal. But adding an operator to a compiler requires more than a kitchen that can cook it. The ordering system must recognize it, the storeroom must know where its ingredients belong, and the waiter must deliver it to the right table. Three new dishes can touch a dozen integration points.

1. The Gates between a Model and an Instruction

The historical sum, maximum, and minimum additions were more than new kernel functions. Each added emission entry points, low-level instructions, backend registration, axis legalization, and tests. Minimum also added a mapping from a high-level reduction mode to the target operation.

These diffs show an integration chain becoming more complete; file counts alone do not establish end-to-end correctness. Each gate has its own failure modes:

“Operator support” is best defined in layers. Being able to print an IR node is one step. Passing lowering is another. Numerical validation and boundary tests move the implementation closer to complete support.

2. Share the Geometry, Distinguish the Arithmetic

For an input [A,B,C] reduced along the last axis, all three operations produce shape [A,B,1]. Axis permutation, effective-rank checks, output-element-count checks, and address lookup can share geometric rules. Historical changes generalized reduction-axis legalization into a pattern shared by several operations, reusing precisely this part.

Numerically, however, replacing a mnemonic is insufficient. Let x=s(q-z), with one common positive scale within a reduction group:

1
2
3

sum(x_i) = s * (sum(q_i) - R*z)
max(x_i) = s * (max(q_i) - z)
min(x_i) = s * (min(q_i) - z)

Sum subtracts the zero point R times; max and min subtract it once from the selected value. Hardware might subtract it from every element before accumulation, or apply a correction afterward. The compiler must know which. Seeing z in a parameter register does not reveal the execution unit’s complete semantics.

A positive scale preserves order, so maximum and minimum can compare integers within a common quantization domain. With different scales inside one reduction group, integer order no longer implies real-value order. For example, integer 20 at scale 0.1 represents 2, while integer 12 at scale 0.5 represents 6. Choosing 20 directly selects the wrong real maximum. Per-channel quantization particularly requires care when reducing across channels.

3. Sums Grow; Extrema Have Different Boundaries

For signed b-bit input integers with bounded magnitudes, the width needed to sum R terms generally grows with ceil(log2 R), although precise lower and upper bounds must treat the minimum and maximum separately. Zero-point corrections, coefficient multiplication, and temporary values before rounding may require more width.

Maximum and minimum do not normally expand the selected value’s range as R grows, but have other traps. Does initialization cover the full input domain? Is a signed input accidentally interpreted as unsigned? If the target supports floating point, how are NaNs and signed zeros handled? The integer quantization paths examined here do not establish floating-point special-value semantics; those rules must be specified when extending the implementation.

Another practical question is how to combine block reductions. Sum adds the block sums; max and min select among the block extrema. Mean must weight block means by element counts when block sizes differ. Abstracting every reduction as “reduce each block, then reduce the block results” fails for such means. The siblings can share a schoolbag, but each must do its own homework.

4. What Changes When Three Instructions Share One Implementation?

A later historical change consolidated three duplicate instruction classes into a common implementation with a kind enum. It unified assembly printing, binary fields, and copying while retaining the distinct operation kinds.

This is a refactor that reduces duplicated implementation. Its main benefit is lowering the chance that a field fix misses one operation. With three copies, a new mode check can easily reach only two; common encoding gives all three the same constraint.

It does not mean the hardware fused three instructions into one, or that a model executes fewer reductions. Performance wording in a commit title must be interpreted through the actual diff. What is directly visible here is consolidation of the software representation. Runtime, binary size, and compilation-time improvements need separate measurements. A public article should not turn “merged classes” into “operator fusion accelerates execution.”

Common implementations also have limits. If mean has a different register layout, reciprocal treatment, or control fields, forcing it into the same parameter package may create more conditionals. Build abstractions around stable shared structure, not merely around names containing Reduce.

5. Why Test Both Assembly and Machine Words?

Historical tests checked both emitted text and binary encoding. The two are complementary:

• Text checks help detect wrong parameter order, operation kinds, and default modes.
• Binary checks help detect wrong shifts, masks, sign bits, and field widths.

An encoder can print “maximum” while the binary selects “minimum.” If the test uses the same encoding helper to generate its expected value, the two errors may agree. More trustworthy golden values are calculated independently of the tested path or cross-checked with a trusted decoder.

One machine word is not enough, either. Combinations of mode, width, signedness, accumulation flags, and other fields need coverage. If the shared implementation accepts an integer enum, test rejection of invalid values too, so an unknown kind cannot silently enter a default branch.

6. A Proposed Regression Matrix and Performance Hypotheses

Existing historical test files show attention to encoding and emission structure; they do not constitute a passing execution report obtained here.

Performance can first be split into three costs: input traffic, reduction-tree or serial-accumulation work, and invocation/configuration overhead. Extrema and sum read the same elements, but execution-unit throughput, dependencies, and type widening may differ. Small tensors are more likely to be dominated by configuration; large tensors may be limited by bandwidth or reduction throughput. The number of shared C++ classes is not a key term in that runtime model.

The most useful outcome is not “three more supported operators,” but a clear map of responsibilities: what can be shared and what needs its own proof. The next member of the reduction family then knows which desk to visit.

7. Challenge the Abstraction with a Zero-Point Counterexample

Give a teaching group the nonzero zero point z=5 and integer values 6, 7, 8. In the centered integer domain, these are 1, 2, 3. Their centered sum is 6, but summing the raw integers gives 21. Subtracting the zero point once gives 16; correctness requires subtracting it per element or subtracting three times the zero point in one step. Maximum instead selects 8 and subtracts the zero point once, giving 3. The two parameter tables may look identical, yet their internal interpretation must differ.

If the output scale differs from the input scale, an appropriate scale conversion is also required after reduction. Max and min can first select a value in a common quantization domain and then convert it, because a common positive scale preserves ordering. Sum must ensure that accumulation does not overflow before conversion. Reusing extrema code while also restricting the sum accumulator to input width produces a characteristic failure: short vectors work, while longer vectors suddenly wrap or saturate.

Block reductions also let us test initialization through identity elements. A sum starts from zero; a maximum needs an initial value no greater than any legal input; a minimum needs the reverse. Initializing every operation to integer zero makes an all-negative maximum incorrectly return zero, and can make an all-positive minimum incorrectly return zero. This example does not depend on a particular instruction format and belongs in any backend’s numerical regression suite.

For signed and unsigned comparisons, choose values with the same bit patterns but different interpretations, and verify that the emitter’s signedness mode reaches the machine word. Small positive values cannot distinguish the interpretations. Applying this discriminating principle to each control field yields a modest number of tests, each excluding a specific incorrect implementation.

These counterexamples do not assert that the historical implementation made those mistakes. They explain why abstraction review cannot stop at code similarity. A shared class should centralize stable representation rules while keeping mathematical differences explicit. Hiding differences in defaults to save lines only makes the next extension harder to prove correct.

Back to series contents · Previous · Next

• ← Previous Post
• Next Post →