Series contents · Operators and Layouts · 阅读中文版
“The first row is perfect. From the second row onward, it looks like a lottery.”
That is often a rather helpful error: the result’s structure is already pointing to a boundary. If the arithmetic formula is wrong, the first row usually suffers too. If the first row is correct and errors begin when crossing rows, inspect address strides, physical padding, and pointer positions at loop exit first. This is a localization hypothesis, of course, and needs evidence.
Specify Layouts and Strides
Our teaching tensor uses HWC order, shape [2,3,5], and 1 byte per element. Suppose each pixel’s channel block is aligned to A=16 bytes. Logically, a pixel contains only 5 bytes; physically, it occupies 16. A row occupies 48 bytes, and the tensor occupies 96. The compact logical data contains only 30 bytes. Code that still advances 5 bytes to the next pixel will walk into padding.
A layout formula is more useful than saying “it needs alignment”:
1 | align(x,A) = ceil(x/A)*A |
The second branch matters greatly. A single-channel layout can place pixels contiguously within a row and pad the whole row, whereas a multichannel layout may align each pixel’s channel block. Both are called HWC, yet their physical strides differ. Multiplying the last dimension by element size is not enough to determine addressing for the whole tensor.
One inspected set of historical changes extracted a simple activation-layout calculation so that memory-size estimates and broadcast strides used the same rules, while also fixing constant serialization. Another historical approach distinguished compact and physical inputs for some operators and handled padding with explicit address loops. Both express the same principle: layout is a contract shared by producers, consumers, and allocators. Dates on commits do not turn separate branch designs into a sequence of merged changes.
Why calculate output strides independently? Input and output element widths may differ. Even with the same shape, a channel taking 1 byte on input and 2 bytes on output can require different alignment after the same C channels. “Conveniently reusing the input stride” may pass for small channel counts that fall in the same alignment bucket, then fail abruptly at the next bucket.
Exactly aligned shapes are especially good at creating this illusion. They make logical_stride == physical_stride, so the old code works too. Teaching tests should select points around a boundary: A-1, A, and A+1 bytes, plus the corresponding boundaries for wider element types. Add C=1 cases so that testing does not cover only the different multichannel layout.
Broadcast Boundaries and Address Rewinds
Broadcasting adds a subtler trap. The main input may have several channels, while the smaller broadcast input is [H,W,1]. Each main-input pixel advances to the next aligned block. The smaller input advances by just one element within a row, then skips padding at the row boundary. A single stride repeated H*W times is no longer enough; inner and outer loops are typically needed.
Suppose the smaller input has W=3 single-byte values per row and row stride 16. After each pixel, the loop increments the address by 1 only if inner-loop work remains. After the third pixel, the address is still at the row start plus 2. Reaching the next row requires an increment of:
1 | boundary_step = row_stride - (W-1)*inner_step |
Not 13, and not 16. The W-1 comes from control flow that omits the inner address increment on the final iteration. Deriving a stride from shape alone while ignoring instruction order easily creates a one-element error. The historical fix explicitly constructed boundary strides and checked for loop completion before performing the corresponding address updates.
When a broadcast vector must restart at the beginning for the next outer iteration, the boundary stride may even be negative. Field validation must therefore use a signed range. Converting a negative value to unsigned before comparison, or treating the field as a larger unsigned positive range, breaks legality checks. The historical changes tightened these checks to signed semantics and added out-of-range rejection tests. No particular device field width needs to be disclosed to explain the general problem.
Constant Padding and Layout Invariants
At this point, many developers would declare the fix complete—only to find that constant inputs are still wrong. Enter the next suspect: constant files usually contain compact logical sequences. If the allocator reserves 96 bytes but serialization places only 30 compact bytes at the beginning, a consumer using the correct physical strides will now read incorrect values more consistently.
The right approach places logical elements according to the same layout when building the constant binary:
1 | physical = padding_buffer(required_physical_bytes) |
The historical code created a buffer of the physical length, checked the logical source length, allocation length, and size representable on the host, then copied by offset. Padding was initialized deterministically for stable artifacts and tests. Padding bytes do not automatically represent quantized real zero: a nonzero zero point makes that distinction necessary. If hardware never consumes padding, a deterministic fill suffices. If padding participates in computation, its value needs a separate semantic justification.
This gives three useful invariants. First, the allocation must cover at least the physical address of the last logical element. Second, the constant producer and operator consumer must calculate the same offset for the same coordinate. Third, addresses at inner- and outer-loop boundaries must agree with the direct coordinate formula. These invariants reduce many apparent “broken operator” cases to a shared layout contract.
Regression, Costs, and Unit Checks
A small address trace makes an effective regression. List (h,w,c), source offset, target offset, and the address after each loop iteration, then compare the jump from the last pixel of one row to the first pixel of the next. Historical tests already checked the arrangement of padded constant data, stride registers for multiple operators, and out-of-range diagnostics.
Alignment’s performance cost should remain visible. The teaching example has 30 bytes of useful data and a 96-byte physical allocation, for a payload ratio of only 30/96. This is arithmetic for the example, not a measurement of device bandwidth utilization: caches, buses, and computational access patterns affect actual transfer. It does show that small channel counts can suffer a substantial allocation increase from padding, and loops may add another burden.
Future optimization could explore merging adjacent blocks, selecting different layouts, packing constants in advance, or removing repeated movement at graph level. These are proposals to measure. Reducing loops must not turn padding back into logical data, and allocation and code generation must not invent independent “faster” layouts. A shared calculation function can reduce bugs while turning performance speculation into costs that can be counted.
Beyond units and boundaries lies an error capable of silencing a review meeting for three seconds: everyone calls a field “stride,” but nobody agreed on its unit. One layer supplies elements and another reads bytes. Single-byte types work perfectly; two-byte types expose the problem. Explicit byte units in variable names, or a distinct byte-quantity type, are more reliable than a comment saying “remember to multiply by size.” The historical fix’s tests across element types help check exactly this.
Low-bit packing further demonstrates why element counts and byte addresses are not freely interchangeable. Suppose each element occupies half a byte. An odd channel count may use only half of its final byte. Whether the next block starts in the other half or must align to a new byte depends on the format contract. An ordinary integer byte width loses that information immediately. Conservatively rejecting unsupported formats gives a checkable capability boundary. Adding support requires bit offsets or an explicit packing description.
Tests for padded constants should distinguish three lengths: logical data length in the source file, physical length required by the layout, and actual capacity supplied by the allocator. Capacity exceeding physical length does not make the extra space valid constant data that must be written. A short source must not be silently zero-filled to manufacture missing logical elements. Historical serialization checked that source length matched expectations and that physical data fit the allocation. That ordering exposes input errors at compile time instead of turning them into random runtime differences.
A simpler address ledger can verify negative-stride rewinds. Suppose a shared vector has three blocks: the inner loop moves from the first to the third, and the next outer iteration must return to the first. Because the final iteration does not advance again, the rewind is negative two block strides. A programmer calculating as if “three steps have already happened” moves before the buffer. Do not check only that a negative number fits the field; a representable wrong address is still wrong. Range checks and semantic checks are complementary defenses.
There is also a question about alignment direction: align the byte count and then convert to elements, or align the element count first? These can be converted under particular conditions, such as element size dividing the alignment unit and no packing. Outside those conditions, the results may differ. A shared layout function should preferably work directly in the final address unit, saving upper layers from repeating the derivation. Supplying it to both memory estimation and data generation also makes consistency tests on identical inputs straightforward.
If layout optimization is planned, retain a slow reference that computes addresses directly from coordinates. It can live only in local tests, outside production code generation. Comparing each optimized loop against the reference on small shapes can cover ordinary advancement, end-of-row padding, and rewinds across groups. The reference should not call the same function that builds the loop plan under test; otherwise, both can share one mistake, and an apparently rigorous test is merely nodding at itself.
When the second row finally returns, the real fix is usually larger than one stride. Several participants in the program have finally agreed on where the next element lives.
Series contents · Previous · Next
If you like this blog or find it useful for you, you are welcome to comment on it. You are also welcome to share this blog, so that more people can participate in it. All the images used in the blog are my original works or AI works, if you want to take it,don't hesitate. Thank you !