Same Output Shape, Different Answers: Pooling, Resizing, and Index Reductions

Series contents · Engineering and Delivery · 阅读中文版

“The shape, type, and memory size all match. Why are we still off by a pixel?”

An investigator enlarges a feature map and notices a small discrepancy every few columns. Elsewhere, a pooling test differs only at the edges. The team initially treats them as separate failures before recognizing the shared question: how does an output coordinate map back to the input?

Shape says how many answers there are, not how to compute each one. Window boundaries, sampling coordinates, rounding rules, and index selection decide what fills that shape.

Padding can hide the denominator of an average

Consider a one-dimensional teaching input [2,6], a window size of 3, and one padded position on the left. The first window can be viewed as [padding,2,6].

If padding contributes to the average and represents mathematical zero, the result is 8/3. If only real input elements count, it is 8/2=4. The output shape is identical, both operations may be called average pooling, and the difference is not floating-point noise.

Several counts are easy to confuse:

• The logical window size.
• The number of actual input elements covered by this window.
• The number of aligned execution slots processed by the unit.

The third count generally should not automatically enter the mathematical denominator. Extra slots introduced for vectorization must not accidentally affect the sum or count.

MaxPool has a related problem. Padding with mathematical zero can make the padding win when all real inputs are negative. An implementation may use masks, a sufficiently small value, or hardware boundary handling consistent with the dtype and specification. Clearing memory is not a universal padding strategy.

Global pooling and ReduceMean cannot be unified by name alone

For some layouts and axis sets, global average pooling and ReduceMean over spatial axes express the same mathematics. Equivalence requires matching reduction axes, output-dimension rules, numerical types, and compatible rounding.

If one operation retains reduced dimensions of size 1 while another removes them, later broadcasting may differ. In quantized computation, accumulating first and scaling once can also differ in precision from another order.

The historical specification separated pooling and reduction. That organization is a useful reminder: underlying kernels may be shared, but the semantic correspondence needs a proof first. Translating similarly named operations into one low-level instruction is an implementation decision, not the proof.

Nearest neighbor: nearest to which coordinate?

Consider resizing a sequence of length 3 to length 5. Two common continuous-coordinate ideas are:

1
2

Align endpoints: x = j * (3 - 1) / (5 - 1)
Half-pixel:      x = (j + 0.5) * 3 / 5 - 0.5

At j=1, the first gives 0.5 and the second 0.4. The implementation must then convert the continuous coordinate into an integer index: floor, ceiling, nearest rounding, or a specified tie rule.

A compiler must read the particular operator version and attributes instead of seeing “nearest” and choosing a familiar formula.

An output length of 1 also exposes the endpoint formula’s denominator. Production code needs an explicit rule rather than relying on division by zero followed by a hopeful integer conversion.

Bilinear interpolation needs a coordinate system before it needs four neighbors

In two dimensions, bilinear interpolation usually combines four neighboring points using horizontal and vertical weights. “Weight four points” describes only the second half of the computation.

The first half still asks:

1. Where does the output pixel center map in the input?
2. Are out-of-bounds coordinates clamped first or handled by a boundary policy?
3. How are neighboring indices selected?
4. What precision represents the weights?
5. If the two directions are computed separately, where does rounding occur?

Mathematically equivalent staged expressions need not be bit-identical in fixed point. Horizontal interpolation followed by rounding and vertical interpolation can differ from retaining a wider intermediate and rounding once.

Quantized integer inputs add the contribution of zero_point. Exactly unit-sum weights provide useful affine properties; finite-precision weights that no longer sum to 1 need further error analysis. Integer output does not reveal the internal precision used.

The performance questions are concrete too: can weights and indices be reused? Do boundary cases add branches? Are accesses contiguous? Without benchmarks, one computation order should not simply be labeled faster.

ArgMax returns a position

ReduceMax propagates a maximum value. ArgMax also propagates its location. Its reduction state is closer to:

1

(best_value, best_index)

The operator contract must decide which index wins a tie. For [7,9,9,4], the maximum is 9, but the returned index can be 1 or 2 according to the rule. A parallel reduction tree that changes left/right comparison order carelessly may return a wrong index while preserving the correct maximum value.

NaNs, signed zero, and the output index type need definitions too. A compiler must preserve the incoming contract and explain limits when it cannot implement it.

Keeping dimensions and normalizing negative axes also belong to semantics. Value and index reductions can share traversal structures, but “remember a subscript at the end” is not a correctness argument.

Why random tests often miss these failures

Exactly tied maxima may be rare in random input, allowing tie-breaking defects to hide. Large images dominated by interior pixels can dilute edge discrepancies. An aggregate mean-squared-error threshold may even tolerate a small but systematic coordinate shift.

More targeted inputs include:

Their purpose is to turn “a little numerical error” into hypotheses that specific counterexamples can test.

One more mapping is needed between specification and implementation

A useful operator specification includes more than formulas. It connects frontend attributes, normalized internal semantics, backend capabilities, and rejection conditions.

If hardware supports only one rounding mode, determine whether preprocessing can compensate, another kernel is available, or some modes must be rejected. If a field is still unspecified, retaining “to be confirmed” prevents a later reader from turning a sketch into an implementation guarantee.

The discrepancy every few columns may ultimately require only one corrected coordinate formula. The expensive part is the period before that fix, when the team treats equal output shapes as evidence that two execution paths are equivalent.

Series contents · Previous · Next

• ← Previous Post
• Next Post →