Debug Works, but Release “Optimizes Away” a Write

Series contents · Engineering and Delivery · 阅读中文版

“The model compiles, but only the release build cannot find the newly generated constant.”

That sentence can send an investigation down an expensive detour. Is the optimization level too high? Did floating-point precision change? Did the linker discard some logic? Add “Debug works perfectly,” and the optimizer is already in the defendant’s chair.

This time, however, the most useful evidence may be an assertion that looks admirably cautious.

How one line creates two different programs

Imagine that a compiler folds a computation into a constant. It must add the data to a weight store, then create a reference in the IR:

1
2

assert(store.insert(key, values));
auto ref = make_constant_reference(key);

It is easy to read this as “perform the insertion and verify that it succeeded.” The actual semantics include a condition: the assertion expression is evaluated only when assertions are enabled. When the build defines NDEBUG, the standard assertion macro does not evaluate its expression. The insertion disappears with it.

This is fundamentally different from an optimizer incorrectly removing a call with side effects. After macro expansion, the program no longer contains that call. Lowering the optimization level will not restore the necessary action while assertions remain disabled. Conversely, enabling optimization while retaining assertions need not produce the same symptom. Debug and Release are convenient labels; the build macros and actual compiler arguments are what matter.

The two programs now diverge:

The function creating the IR reference does not automatically know that an earlier step was omitted. The failure may surface during serialization, a later read, or even model loading. The longer the delay, the more it resembles another module’s bug.

The reviewed fix moved the operation that adds the weight tensor out of the assertion expression and reported failure explicitly. Although the commit title concerned a quantization-related write, the broader lesson is that any required state change must happen independently of whether debug checks exist.

First question: check the mathematics, or check that the data exists?

When a quantized constant goes missing, it is natural to investigate scales, multipliers, shifts, and value ranges. Those matter, but only after establishing that the object exists.

Separate the investigation into three layers:

1. Existence: was the target name added to storage, with the expected byte count?
2. Structure: do the shape, element type, and name reference agree?
3. Numerics: are the values and their quantized interpretation correct?

Suppose a constant should contain 15 elements. If its entry is absent from the file, arguing about multiplier rounding is premature. Once the entry exists and its length is correct, numerical error becomes the next useful question.

This order is broadly useful: prove that data passed through each stage before explaining how a stage transformed it. It reduces the chance of mistaking “never happened” for “happened incorrectly.”

Separating the action from the assertion is only a start

A minimal structural repair is:

1
2

const bool inserted = store.insert(key, values);
assert(inserted);

The insertion now executes even without assertions. But if it fails, Release still creates the reference and enters another invalid state. The failure policy is unfinished.

A more complete teaching example is:

1
2
3
4
5

auto result = store.insert(key, values);
if (!result.ok()) {
    return diagnostic("cannot materialize constant", key);
}
return make_constant_reference(key);

The actual change used an explicit fatal error. This example returns a diagnostic to illustrate the same control-flow requirement, not to imply identical exception or termination semantics. The right choice depends on whether callers can propagate failure and whether the current IR state is recoverable.

The invariant is simple: a successfully created reference must resolve to data that was successfully registered. A check must do more than inspect a Boolean; it must decide whether the state machine may advance after failure.

If the operation allocates a name before writing data, further questions arise. Does failure leave the name reserved? Will a retry choose a different name? If several files are involved, what happens after partial success? The diff establishes that a required call was moved out of an assertion. It does not establish transactional behavior for the entire weight store. These further questions are design checks motivated by the defect.

A “correct test” that can fool us

Running one Debug compilation and checking for exit code zero misses two dimensions: build configuration and output contents.

A more targeted regression plan uses the same minimal model across a matrix:

Nor should “Debug and Release files are byte-for-byte identical” become an unexamined requirement. Debug information or metadata ordering may legitimately differ. First compare the constant set, shapes, and values that must be semantically identical, then decide whether byte equality is necessary.

Fault injection is especially useful here. Making a test double return an insertion failure is more reliable than waiting for a disk or memory problem to occur by chance. Such injection belongs in a test substitute or isolated environment, not in normal production behavior.

Other ways to hide the same risk

Assertions are not the only switchable facility that can accidentally carry required work. Similar defects arise when state is updated only in a debug logging branch, a map is initialized only under verbose output, or a computation depends on a diagnostic-only variable.

A useful review question is: “If every log and check is disabled, are all actions necessary to complete the task still present?”

The opposite extreme—turning every assertion into a runtime error—is not automatically better. An unrecoverable internal invariant, an expected invalid input, and a recoverable resource failure serve different roles. Classify them before choosing an assertion, error return, or termination. The essential rule is that a diagnostic switch must not decide whether required work happens.

There is also a smaller build issue nearby. Some variables are read only by debug output. Remove that code in a release build and they become unused; if warnings are errors, Debug compiles while Release does not. An appropriate annotation can address genuinely diagnostic-only variables. It cannot repair a necessary action that disappeared.

Why the optimizer is not the villain

The interesting feature of this defect is the mismatch between the scale of the symptom and the size of the cause. Models, quantization, release packages, and compiler flags all seem relevant. The decisive question is whether one expression gets evaluated.

When a configuration change causes inconsistent behavior, divide the suspicion into three questions: is the preprocessed code the same? Do required side effects remain? Does failure handling operate in every configuration? Those questions often narrow the problem more effectively than disabling the entire optimization pipeline.

The evidence establishes that a write depended on assertion evaluation and that the fix made its failure path explicit. It does not establish that all Release failures come from assertions, and it provides no performance result. The improvement is consistency of program semantics—the prerequisite for any meaningful performance analysis.

Series contents · Next

• ← Previous Post
• Next Post →