The Assembly Looks Right. Does That Mean the Machine Code Is Right?

Series contents · Engineering and Delivery · 阅读中文版

“The generated assembly matches expectations, so the code generator is correct.”

If the assembly printer and binary encoder are separate paths, that conclusion leaves half the problem unchecked. The printer can display the right enumeration name while the encoder writes its field into the wrong bits. Conversely, the machine code may be correct while the debugging assembly misleads the investigator.

Before testing a code generator, establish exactly which output you are observing.

Three output layers, three ways to mislead

A typical emitter may expose readable pseudoinstructions, expanded assembly, and final binary output.

1
2
3
4

Operator request
  → pseudoinstructions: close to developer intent
  → actual instructions: expand immediates, resolve labels
  → encoding: assign bits to each field

Pseudoinstructions alone miss expansion errors. Actual assembly alone misses bit-field encoding errors. Binary length alone misses contents that are equally long and entirely wrong.

The reviewed tests grew from configuration loads and instruction-prefix checks for vector operators into comparisons of pseudoassembly, actual assembly, and binary output for tensor computation, transfers, and debugging output. That expanded the dimensions observed rather than merely adding string snapshots.

The distinction matters: some early cases checked key prefixes and binary word counts. They did not all compare every machine-code bit from the beginning. A filename containing “golden” cannot grant a test evidence it never collected.

Don’t choose only the most convenient addresses

Nearby configuration blocks can use a base address plus a short offset. Move the parameter block farther away, and the immediate range may no longer suffice. The emitter may need to load another base address.

That provides a natural test boundary: exercise both near and far paths. Convenient small addresses may put every test on the same short path, leaving the other half of the implementation unquestioned.

For teaching, imagine an instruction with signed offsets in [-B, B-1]. Test more than 0 and 1:

1

-B-1, -B, -1, 0, B-1, B

Inside the range, check the intended relative addressing. Outside it, check a legal base-loading strategy. These cases explain failures more clearly than dozens of arbitrary random addresses.

A real ISA’s boundaries come from its specification, not from these symbolic examples.

Same operation name, but are the same fields valid?

Binary operators may have vector–vector, vector–scalar, and scalar–vector paths. Swapping operands changes subtraction. For some configuration formats, even a commutative operator still needs its scalar in the correct field.

Signed and unsigned data also deserve separate observation. The same printed integer immediate can be interpreted differently by encoding and execution.

Build the matrix around semantic branches:

Test count is not the goal. Understanding why one case differs from another makes a suite maintainable.

What do negative tests actually test?

“Invalid arguments should fail” sounds simple. If failure is deferred until hardware execution, however, it becomes much harder to explain which shape or dtype was invalid.

The reviewed negative cases covered mismatched input/output shapes, invalid index types, disagreement between index length and output shape, slice bounds, reshape element counts, matrix multiplication’s inner dimension, and convolution channels.

These cases check interface boundaries. They do not prove numerical correctness for every legal input. Positive tests ask whether permitted actions are performed correctly; negative tests ask whether forbidden actions are rejected promptly and clearly.

Diagnostics should be checked precisely enough to identify the failure category. Accepting any exception can let a null dereference or unrelated failure pass the test. Locking an entire sentence byte for byte can make harmless wording improvements noisy. Stable error categories and essential context are usually a useful middle ground.

How a golden test can preserve a bug forever

The most dangerous update workflow is: change the implementation, see the tests fail, and automatically accept the new output.

If expected and actual values come from the same encoding logic, their agreement only proves that the function is loyal to itself. There is no independent judgment.

Golden values need an independent basis: a small example interpreted by hand, a separate decoder, a derivation from specified fields, or verified reference execution. A baseline update should explain which fields changed and why, not merely replace a block of hexadecimal.

An invented encoding expression illustrates the approach:

1

word = opcode | (destination << p) | (source << q) | flag

If changing source should affect only one bit range, a test can also check that other ranges remain unchanged. Such metamorphic properties complement fixed baselines. They make it harder for an unexplained whole-output change to pass unnoticed.

Is byte equality still the right requirement after optimization?

A legal optimization can reduce instruction count, renumber registers, or reorder independent work. An old golden mismatch does not automatically imply a semantic bug.

Keep baselines, but define contracts at appropriate layers. The encoder’s fields for one instruction can be compared strictly. Important dependency order inside a kernel needs structural checks. Regions with scheduling freedom can be constrained by dependencies and reference results.

“Prepare parameters before computation” is a stable invariant. Whether “use exactly this temporary register” deserves a test depends on whether register selection belongs to the interface being preserved.

Keeping pseudoassembly, actual assembly, and machine code together helps locate the layer where a change began. It is more informative than staring at a different final hash.

A code-generation regression is not a performance report

Replacing an extra address load with a relative offset may reduce instruction count. Total time still depends on execution engines, memory traffic, scheduling, and synchronization. A test expecting fewer words is not evidence that a model runs faster.

Performance needs a separate experiment with defined input, cache state, hardware, and statistical method. Functional tests should first establish the intended structure reliably; performance analysis asks whether that structure lies on the critical bottleneck.

The evidence supports a narrower conclusion: observing three output forms and rejecting invalid inputs makes code-generation constraints easier to inspect. A golden test is valuable because it can disagree with the implementation—and explain why.

Series contents · Previous · Next

• ← Previous Post
• Next Post →