Why Write the Same Final State Twice?

Series contents · Resources and Scheduling · 阅读中文版

“The final hidden state is already a graph output. Why save it again?”

“Because the graph output goes to the caller. Persistent state goes to the next invocation.”

“Then why not chain the two stores?”

It sounds like ordinary common-subexpression reuse. In stateful computation, however, the two destinations have different lifetimes. Turning them into a chain can confuse a value’s meaning with the side effects that store it.

Separate Three Objects First

A recurrent cell can be described by this teaching relationship:

1

(sequence_result, final_state) = recurrent(input, initial_state)

sequence_result is the current invocation’s sequence output, final_state is the state at the last step, and persistent_slot is the location that retains state between invocations. Similar shapes do not make these the same kind of object.

The final state may serve two purposes: update persistent storage and return as a graph output for this invocation. The natural dependency graph therefore fans out:

1
2
3

                   ┌─ Write to persistent storage
final_state ────────┤
                   └─ Write to graph-output storage → Return to caller

A more error-prone form is:

1

final_state → Result of state writeback → Graph-output writeback

Whether that second form is correct depends on the exact definition of a store’s returned value. A destination alias, a value annotated with a memory space, or a placeholder that preserves a side effect cannot necessarily replace the original final_state unconditionally.

The History Is Not a Straight Line

Commit titles deserve particular caution here. Two commits with the same title contain different changes.

One proposal tried to accept dynamic initial state. When the initial state was not constant, it allowed only a restricted unidirectional case and required its type to match the final state. It also checked whether the final state already had exactly one store user, using that fact to decide whether to create another internal store.

The other same-title proposal did not retain that dynamic-input extension. Instead, it always created internal state writeback while leaving existing consumers attached directly to the original final state. Its accompanying test focused on both the internal state store and the graph-output store reading the final state directly.

A later change removed the dynamic-initial-state path, restored the constant-initialization requirement, and adopted this direct fanout. The Git tree diff between the two compared endpoints was empty. That establishes equal contents at those endpoints, not the absence of the intermediate experiment, nor that both histories reached the current working branch.

The useful lesson is not a slogan about successfully supporting dynamic state. Input-state policy and output-state policy are separate matters. Correct output handling does not automatically establish support for every input form.

Why “There Is Already a Store” Is Insufficient

A value having one store user does not identify the destination. It could be a graph-output buffer or a state slot shared across invocations. Even if their bytes are identical, ownership, address stability, update timing, and reclamation conditions may differ.

Skipping persistence based only on user count mistakes a structural property for a semantic one. A better question is whether the writeback satisfies the persistence contract. That answer should come from destination identity, an effect description, or an explicit lowering convention.

The reviewed final test checks that internal state writeback targets the same address as initial-state storage. It also checks that graph-output writeback directly consumes the final state rather than the result of internal writeback. These are assertions about IR structure and address relationships, not evidence that numerical behavior across repeated inference calls has been validated.

Why Persistence Bugs May Appear Only on the Second Run

In a teaching scenario, the first invocation uses initialized state and computes correctly. If the internal slot is never updated, the second invocation starts from the initialization again. A single-inference test can miss this entirely.

A subtler case is also possible: the graph output contains the correct final state, but the runtime never feeds that output back into the state input. Seeing a correct output file, a developer assumes persistence works, while the next invocation still reads the old location.

Conversely, if the runtime explicitly feeds output into the next invocation while the compiler also updates internal state implicitly, two state-management schemes may coexist. A public interface should say who manages state, whether reset is supported, and whether instances may run concurrently, rather than hiding these semantics in a placeholder marker.

Dynamic Initial State Has a Separate Contract

Dynamic state support requires checking at least direction, batch, hidden dimension, element type, and quantization parameters. Equal tensor size is insufficient: the same element count can represent different directions or layouts.

If input and output states use different quantization, feedback may require rescaling. If the runtime keeps state at a fixed address, aliasing writes must not overwrite input that has not yet been fully read. The final state of a bidirectional recurrence also has different implications from unidirectional streaming inference. Conclusions from one experimental unidirectional path cannot be extended to every form.

These are general design requirements. The appearance and later removal of a restricted dynamic path is a reminder that capability claims belong to specific versions and explicit tests.

Is an Extra Writeback Wasteful?

If the destinations truly differ, both writebacks may be necessary. If the final state already occupies persistent storage, an internal action may express effects and dependencies without copying the same bytes again. Inspect storage aliases and actual transfer instructions after lowering rather than merely counting high-level Store nodes.

Conditional optimizations include exposing persistent state directly as graph output or sharing storage between the final sequence slice and final state. But the caller must not still need the old output view when the next update occurs. Otherwise, saving a copy changes the data the caller observes.

Performance evaluation should therefore track bytes written, copy count, state-slot lifetime, and how long callers retain outputs. Comparing Store-node counts in isolation can reward missing semantics as apparent efficiency.

The Save Must Finish Before the Next Read

Even if both writebacks read the correct final state, ordering between invocations still matters. Suppose return from this invocation guarantees only that graph output is ready, without waiting for internal state writeback. A caller immediately starts another invocation, which may race with the unfinished update.

Persistence thus requires both an address relationship and a completion relationship. A teaching contract is: this invocation’s state writeback completes before the next invocation reads its initial state. Call synchronization, device events, or serial runtime execution may provide the guarantee, but the responsible layer must be identifiable.

Now let two threads share one model instance. If they share its state slot, interleaved calls may form an unintended state sequence. Two independent streams require independent state instances. Merely duplicating the output tensor does not necessarily isolate internal persistent storage.

This does not mean every recurrent operation must implement thread isolation itself. It means interface designers must state responsibility clearly. The historical evidence here is mainly at the lowering-graph and address-assertion level. Those assertions do not directly prove runtime serialization, per-instance resource management, or cross-invocation events.

A test could deliberately make state writeback slower than graph-output writeback and immediately start the next invocation. An independent teaching simulator makes that delay easy to inject and separates two facts that otherwise look alike: both outputs are correct, and state is ready for the next call.

Regression Matrix and Experiment Plan

The structural checks in the commits cover some output relationships. Repeated-call behavior, instance isolation, and performance comparisons remain proposed experiments.

An independent experiment could begin with the tiny teaching recurrence h_next = 2*h + x, avoiding complex gate functions. If two calls distinguish persistence from reinitialization, the state pipeline can be tested directly. Once that pipeline works, substitute a reference recurrent cell to separate state-management errors from arithmetic errors.

When the same value seems to be written twice, first draw who owns each destination. In a stateful system, apparent duplication may be the visible trace of two different promises.

Back to series contents · Previous · Next

• ← Previous Post
• Next Post →