It Only Moves Data. How Can It Still Be Wrong?

Series contents · Engineering and Delivery · 阅读中文版

“This patch should be safe. It does no computation. It only moves data.”

Half an hour later, downstream matrix multiplication returns different results. The team checks multiplication, then quantization scales, before discovering that the copy preserved every byte while changing the position each byte represented.

Data movement easily receives undeserved trust: if values did not change, semantics must be unchanged. Tensor semantics also include coordinates, shape, layout, aliasing, and quantization axes. Correctly moving bytes does not necessarily mean correctly moving a tensor.

The same memory can tell two different stories

Consider a matrix of shape [2,3]:

1
2

a b c
d e f

Its contiguous storage is a b c d e f. Changing the shape directly to [3,2] gives:

1
2
3

a b
c d
e f

Transposing it gives a different logical result:

1
2
3

a d
b e
c f

The first operation may change metadata only. The second may be represented as a view with noncontiguous strides or implemented by physically rearranging data. The choice depends on consumer layout support, buffer lifetimes, and aliasing rules.

A “free transformation” therefore needs at least three answers. Is the element mapping correct? Do later operations accept the new strides? Can writes affect the shared storage? Leave any one unanswered and zero-copy can become a delayed failure.

The Concat axis determines whether whole blocks can be copied

Concatenating two [2,2] tensors along axis zero often allows two sequential block copies in a contiguous layout. Along axis one, their rows must instead be interleaved:

1
2
3

Left input    Right input   Output
a b           e f           a b e f
c d           g h           c d g h

Copying all bytes of the left tensor followed by all bytes of the right tensor produces the wrong order.

A general implementation often divides dimensions into outer, axis, and inner portions, then copies each input’s axis×inner block for every outer position. This explains why some concatenations need a few large copies while others produce many small transfers.

The specification work also exposed a gap that a neat document can obscure: the graph may allow a variable number of inputs while a hardware description provides only a limited set of source addresses. Bridging them requires staged concatenation, temporary buffers, or another strategy. A two-input example cannot justify a promise of arbitrary input count.

Split and Slice: multiple results mean more than multiple names

Splitting an input can create independent buffers or several views. Views avoid copies but introduce a lifetime question: after the input’s logical operation is finished, do output views still need its underlying storage?

Slice also needs start/end rules, steps, negative-index handling, and boundary normalization. A contiguous example with step 1 does not establish that negative steps or strided slices can use the same transfer path.

For example, selecting positions 1, 3, and 5 from [0,1,2,3,4,5] produces [1,3,5]. The result contains 3 elements, but they are not a contiguous 3-element input region. Constructing a memcpy solely because the output size is correct is a bug that simple tests can easily miss.

Test contiguous segments, strided segments, empty results, and boundary indices separately. Explicitly rejecting unsupported cases is usually easier to maintain than silently producing a plausible-looking contiguous block.

Gather turns addresses into runtime data

Unlike regular slicing, Gather may select positions supplied by another tensor. Reading the indices becomes part of the computation.

For input [8,13,21,34] and indices [2,0,2], the output is [21,8,21]. Duplicate indices must remain, and an optimization must not sort away the specified order.

A Gather contract needs the index type, axis, output-shape rules, negative-index policy, and out-of-range behavior. Frontends and operator versions can differ; a backend must not silently fill the gaps with its favorite array-access convention.

This also raises a performance question. Equal output sizes do not imply equal transfer costs. Consecutive indices, repeated hot indices, and scattered indices can have very different access behavior. A useful experiment would fix output element count, vary index distribution, and observe effective bandwidth and latency.

Which zero does Pad insert?

For affine quantization:

1

real = scale * (integer - zero_point)

If zero_point is 117, integer 117 represents real zero. Writing integer 0 around a quantized tensor’s edge introduces a negative real offset, not mathematical zero.

A convolution’s internal padding path may conceal this issue because the compute unit handles boundaries specially. An explicit Pad operation actually writes buffer contents. Assumptions from those two implementation paths are not interchangeable.

Constant padding, edge replication, and reflection also need distinct rules for empty dimensions, small inputs, and padding widths approaching input size.

Quantization axes move with coordinates

Suppose a tensor uses different scales along its channel axis. If Permute moves that channel axis from the first position to the last, the quantization axis must move too. A Gather along the channel axis may need to select or duplicate scale entries with the same indices.

Concat presents a subtler problem. Equal integer dtypes do not imply equal scales or zero points. Concatenating stored integers preserves those integers, but one common output quantization cannot generally interpret two different input scales at once.

Possible solutions include requiring matching parameters, requantizing first, or using a per-axis output representation when allowed. Each has costs and constraints. The central question is whether moving data also moves the information needed to interpret it.

That is the layer most easily omitted by “just memcpy.”

Make movement errors leave recognizable evidence

Random inputs broaden coverage. Coordinate-encoded inputs are often clearer for layout debugging. For a small two-dimensional example:

1

value(row, column) = 10 * row + column

With dimensions small enough to avoid encoding collisions, wrong outputs can reveal swapped axes, incorrect strides, or truncated boundaries. Higher-rank tests can use a noncolliding coordinate encoding or retain coordinate tuples directly as the reference representation.

Separate the test categories:

The review ended with more than a new address calculation. “It does no computation” became: “It preserves the selected real values, but must also preserve coordinates and interpretation.” A few extra words removed a great deal of wishful thinking.

Series contents · Previous · Next

• ← Previous Post
• Next Post →