Series contents · Resources and Scheduling · 阅读中文版
“We’ve passed this tensor’s last user. We can reclaim it.”
“That user only changed its shape. The operation that actually reads the data is still waiting in another queue.”
If memory allocators had an award for excessive helpfulness, premature reuse would be a strong contender. It follows IR order diligently: once direct users have been handled, it hands the space to the next writer. There is just one problem. The user has disappeared; the reader has not.
Three Different Moments of Completion
The first moment is when the compiler finishes visiting an IR operation. The second is when a device command has been issued. The third is when the device actually finishes reading. In asynchronous computation, these moments can be far apart.
Views add another level of indirection. A reshape or a layout view that performs no transfer can return a new value while still referring to the same storage. It needs no device command and therefore offers no completion event to wait on. Yet its downstream consumers will read the original data.
Consider this redesigned chain:
1 | Read A → Extract region B → View C → View D → Compute U |
A check limited to B’s direct users may skip C because C produces no runtime dependency. The allocator then decides B is no longer used and lets E overwrite B’s address. By the time U actually reads, the bytes belong to E.
Zero-copy does not mean zero lifetime. Eliminating a transfer usually means more values share storage, placing an additional alias-propagation obligation on liveness analysis.
Why a Small Fix Has a Large Meaning
The verified change replaces a direct-user check at local-storage reuse points with a search for runtime consumers through view chains. An existing helper first resolves group aliases and uses a visited set to avoid repeated traversal. When it finds a user that produces a device dependency, it checks the ordering between that user and the next write. When a user produces no runtime dependency, it continues through that user’s results.
The change applies both to ordinary allocation owners and to every member of a shared allocation. The latter matters especially: one allocation may have several slices, concatenation aliases, or output members. Proving that the owner itself has no remaining users does not prove that the actual readers of every alias have finished.
The patch is small because the traversal mechanism already existed. What was missing was calling it at the correct release decision. The broader lesson is simple: even a thorough safety check cannot protect a system if resource-release paths bypass it.
Use Happens-Before, Not Textual Order
If U and E occupy the same strictly ordered queue, with U before E, queue semantics may be enough to prove reuse safe. If they belong to different engines, a cross-engine dependency must establish completion order.
A public teaching predicate could be:
1 | safe_to_reuse(storage, next_writer): |
The predicate deliberately says completion(reader), not “the compiler encountered reader first.” Substituting an IR-index comparison for a completion relationship still allows asynchronous queues to defeat the check.
At the other extreme, inserting a global barrier before every reuse is not free. It establishes strong order, but may stop unrelated tasks together. A better strategy is to reuse where existing relationships suffice and retain separate storage where they do not, then assess whether additional fine-grained synchronization would be worthwhile.
Where Should Traversal Stop?
Once traversal reaches a real runtime reader, its completion relationship can be checked. Not every downstream result must continue to be treated as an alias of the original storage. Actual computation usually produces new storage; whether later users still read the old address depends on the operation’s specific alias semantics.
This reveals a modeling boundary: “produces no runtime dependency” cannot universally mean “pure view.” Some operations may represent host control, return boundaries, unknown dialects, or side effects not yet modeled. The reviewed helper treats unfamiliar users conservatively rather than allowing reuse by default.
The visited set does not mean “a cycle proves safety,” either. It prevents duplicate traversal. Correctness still depends on valid IR, alias resolution, and operation classification. A teaching implementation should distinguish “already visited; do not repeat” from “semantics verified.”
Why the Regression Case Takes Two Detours
The relevant test does more than produce a tensor and use it once. It builds two data branches, sends each through a view chain, computes on them separately, and finally combines the results. Its key checks require the branches to keep distinct storage while each branch’s own views share its address.
That structure exposes the old check’s blind spot: direct users seem finished, while real consumers hide behind several views. Simplifying the test to remove the views might allow the old implementation to pass, defeating its regression value.
An address-pattern test establishes that the compiler did not choose that dangerous reuse plan. It does not establish that every timing relationship has been observed to be safe on real asynchronous hardware. That also depends on instruction dependencies and device completion semantics, which require another layer of validation.
Does More Memory Mean a Performance Regression?
Fixing premature release may increase peak memory use. This is not necessarily a regression; the old estimate may have understated the real requirement. Incorrect reuse is not a valid optimization baseline.
If the new peak hits a capacity limit, three directions are worth considering: reschedule readers to finish earlier; materialize some views to obtain independent lifetimes; or insert sufficiently precise synchronization to permit safe reuse. These trade scheduling freedom, transfer cost, and parallelism differently. Static byte counts alone cannot choose among them.
There is also an often-overlooked compile-time issue. Repeatedly traversing long view chains at every release point can duplicate work. Measure chain length, branch count, and alias-set size before deciding whether to cache actual-consumer sets. Such caches must be invalidated correctly when IR or scheduling changes. Otherwise, they merely preserve stale answers more efficiently.
What About Partially Overlapping Aliases?
Suppose an allocation has views of its first and second halves. Readers of the second half have finished, but a reader of the first half is still running. A conservative allocator managing whole allocations retains the entire region. An allocator managing subranges may be able to reuse the second half safely.
Both can be correct; they differ in the granularity of their proof. If the system records only that values share an owner, without precise offsets and covered ranges, tensor shape alone cannot establish that two views do not overlap. Strided views, transposes, and noncontiguous slices make intuition even less reliable.
The fix discussed here strengthens safety checks using the existing alias set by checking consumers of all allocation members. It does not establish that arbitrary byte ranges can be reclaimed precisely. Keeping this limitation in a public explanation prevents a correctness fix from being mistaken for a complete alias-analysis system.
When the peak rises, first draw a three-level teaching diagram: storage, aliases, actual consumers. Mark the reader that prevents release. If the difficulty is a long view chain, improve analysis efficiency. If it is a real cross-engine execution tail, reconsider scheduling. If a tiny live fragment pins a large allocation, consider subrange allocation. All three look like high memory use, but call for different remedies.
A Minimal Regression and Experiment Set
| Case | Property that must hold | Failure signal |
|---|---|---|
| One-level and multilevel views | Final readers determine storage lifetime | More views cause earlier incorrect reuse |
| Two cross-engine branches | No overwrite before completion order is established | Two live branches receive the same address |
| Serial reads and writes on one queue | Explicit ordering can support reuse | Excess conservatism creates an unnecessary peak |
| Multiple alias members | Readers of every member are included | The owner is safe but another alias remains live |
| Unknown users or return boundaries | Retain conservatively or model explicitly | An unrecognized operation is treated as finished |
| Branch joins and long chains | Traversal terminates and results are stable | Repeated visits cause excessive compile time |
A separate teaching simulator could assign different delays to each queue and enumerate interleavings. Each read records its expected data version; each write increments the storage version. A read observing the wrong version supplies a concrete counterexample to premature reuse. This would explain more than inspecting peak addresses alone, but it remains a proposed experiment.
When a tensor appears ready for immediate reclamation, the useful follow-up is not just how many users remain. Ask who still reads those bytes, through which alias chain, and under what completion relationship.
Back to series contents · Previous · Next
If you like this blog or find it useful for you, you are welcome to comment on it. You are also welcome to share this blog, so that more people can participate in it. All the images used in the blog are my original works or AI works, if you want to take it,don't hesitate. Thank you !