Putting Activation Inside Convolution: Preserve the Quantization Boundary Before Removing the Node

Series contents · Operators and Layouts · 阅读中文版

“These two nodes are adjacent. Fusing them should make things faster, right?”

“Possibly. First, explain what that intermediate tensor used to be.”

The most tempting picture of fusion is one fewer circle in a graph. What disappears may include more than a name: real storage, a quantization step, a data type, and a value read by other nodes. Turn on a postprocessing switch while forgetting those intermediate semantics, and the graph becomes shorter while the answer may move further away.

Check Fusion Preconditions

Our teaching computation is u=Conv(x,w,b), followed by y=GELU(u). Suppose the target has a convolution postprocessing path that can apply a lookup table or piecewise-linear approximation after convolution produces a result. Fusion aims to make y the convolution node’s final output, reducing scheduling and intermediate movement for a separate activation stage. Generated artifacts and measurements must establish whether those costs actually disappear.

The first condition concerns u’s users. If a neighboring branch computes z=Abs(u), changing the convolution output directly to activated y makes z see an entirely different value. Historical rules required the producer to have a single user and rejected a producer that already had a fused activation. This may be conservative, but it is easy to justify. Duplicating the convolution to retain the other branch might enable fusion, but could repeat substantial computation and should not be the default without a cost model.

Grouping is another explicit restriction: all the inspected convolution and transposed-convolution fusion rules required group=1. Whether grouped convolution is first legally decomposed, and whether the resulting nodes can then fuse, is a separate path to verify. These fusion commits do not establish it.

The second condition concerns the activation itself. A family of table-based graph nodes may express many nonlinear functions, but that does not mean one convolution postprocessing path supports all of them. Historical transposed-convolution fusion first accepted GELU and later added Sigmoid; ordinary convolution accepted a restricted pair of functions. Rejecting other functions marks a capability boundary that should not disappear under a broad “nonlinear fusion” headline.

One commit title appeared to concern only removal of an old filename, yet the change also updated matching conditions, code-generation validation, and test expectations to support another activation. This is a useful reminder: renaming files can accompany semantic expansion, so review cannot stop at checking that files moved. A test that previously expected an independent activation changed to expect fusion; the assertion change supplies the evidence.

The third condition concerns shapes. Elementwise activation preserves element correspondence, but graph optimization may produce types with equal element counts and different apparent ranks. Inspected fusion rules did not all check shapes identically: some required equal shapes, while others checked equal element counts. The general public conclusion should be “prove one-to-one element correspondence and layout compatibility,” not that either local check is sufficient in every setting.

Preserve Intermediate Quantization

Intermediate quantization is especially easy to lose. Suppose convolution output u uses scale su, while activation output y uses scale sy. The LUT’s segments or lookup input depend on u’s encoding coordinates. If the fused node adopts y’s final type and later derives LUT input interpretation from that type, it may incorrectly use sy to interpret the intermediate convolution result.

For a teaching example, let su=0.25 and sy=0.0625, with both zero points equal to 0. The same integer 8 then means 2 in one encoding and 0.5 in the other. Using the wrong scale systematically changes the function location accessed by the table. Retaining a description of the pre-activation type distinguishes the producer’s internal boundary from the node’s final output. The historical rewrite preserved this information along with the LUT, function kind, and table-header data.

A fusion plan can be expressed as:

1
2
3
4
5
6

fused = ComputeWithActivation(
    inputs = original_compute_inputs,
    table = activation_table,
    intermediate_type = type_of_u,
    output_type = type_of_y,
    activation_kind = supported_kind)

This pseudocode does not reproduce the original interface; it expresses the contract. Attribute names are secondary. Deleting a node must not erase the memory of its intermediate numerical domain.

Code Generation and Regression Evidence

Successful graph rewriting is only the first half. Code generation must obtain the allocated LUT address, build postprocessing parameters, select the correct function encoding, associate descriptors, and load LUT parameters before issuing the compute instruction. Historical changes connected address handling, descriptors, parameter memory, and kernel emission. Seeing a postprocessing enable bit and declaring fusion complete could overlook a parameter register that remains uninitialized.

The attribute set must also be complete. A LUT operand without an intermediate type or table header is an incomplete fused state. Fusion attributes left behind without a LUT should be rejected too. The historical code added diagnostics for these combinations and restricted supported intermediate storage types. This prevents malformed hand-written IR or states produced by other passes from bypassing rewrite checks and reaching emission.

Why test ordinary convolution as well? Adding an optional fusion operand is an easy way to damage the existing path. Without an activation, no useless LUT parameters should be loaded and postprocessing should remain disabled. Historical tests inspected both fused and unfused instructions and checked that fusion removed the corresponding standalone vector lookup operation. This is closer to actual execution behavior than merely counting one fewer graph node.

Still, absence of an independent lookup instruction proves only a change in generated structure, not acceptable numerical error. Piecewise-linear approximations, fixed-point parameter precision, saturation, and rounding require dedicated numerical tests. Sample more densely around changes in the activation curve, near saturation, and around quantization boundaries, rather than relying only on uniformly random samples over a broad interval.

A useful comparison keeps two teaching implementations: convolution followed by an independent activation, and the fused implementation. They should use the same intermediate quantization convention. If the reference applies activation directly to a high-precision real convolution result while the fused path models quantization followed by activation, a difference may come from inconsistent reference definitions. Optimization correctness usually means preserving the original graph’s semantics, not silently replacing it with another function that looks more precise.

Costs, Semantics, and Safe Rewriting

The performance bill can first be decomposed qualitatively. Potential savings include intermediate reads and writes, task launches, and synchronization. Added or retained costs include LUT parameter loading, table accesses, and limits that postprocessing throughput places on the compute pipeline. An intermediate already resident in on-chip cache has a very different cost from one written to larger memory. Fusion may also change parallel scheduling. A reduction in node count cannot be converted directly into a speedup percentage.

Suggested measurements should be grouped by output size, channel count, and activation type. Record whether intermediate storage actually disappears, whether LUT loads are repeated, total latency, and arithmetic-unit utilization. Parameters and launches may dominate small operations; computation or bandwidth may dominate large ones.

One fusion restriction can hide behind a function name. Two representations called GELU may use different approximations, input ranges, or segment parameters. Matching the function name identifies a candidate; execution still needs to retain the generated table and its header meaning. Replacing the activation’s LUT with a default table that “has the same function name” may change its error contract. Carrying the existing LUT and related information into the fused node was an important part of preserving the historical approximation path.

Write pre- and post-fusion semantics as two data chains. The original graph produces u in a particular quantized encoding, uses that encoding to access the activation approximation, and obtains y. The fused graph should retain the same u encoding boundary internally, even if u no longer becomes a separate memory tensor. A boundary that is no longer materialized still exists. If target postprocessing directly receives a wider accumulator, we must establish how it reproduces the original scaling and rounding, or explicitly state the numerical differences that are permitted.

Deletion order in the rewrite also deserves review. Creating the fused node and replacing activation results before deleting the original producer must leave no overlooked users or references to old nodes, and must not insert the new node before its dependencies. A single-user restriction simplifies this proof. Future support for multiple results or cross-region fusion may outgrow the original short pattern: dominance, region boundaries, and side effects would need consideration.

Source locations and diagnostics are part of artifact quality too. After fusion, an error should lead users back to both convolution and activation rather than just a newly anonymous node. Historical rules combined source locations, preserving useful clues. A general recommendation is to use provenance for explaining errors without indiscriminately copying every source attribute as an execution attribute. Some attributes describe the old output and may no longer apply. Explicit rules for preserving, converting, and deleting each kind are more reliable.

Why be cautious about “copy all attributes, then override a few”? An attribute may cache the original output element type, quantization range, or memory estimate and become stale when the result type changes. The inspected diff explicitly preserved the intermediate type, but that does not guarantee automatic correctness for arbitrary attributes added in the future. Tests should include distinct pre- and post-activation encodings and verify that generated parameters come from the correct side. Attribute evolution is part of long-term fusion maintenance.

Finally, structural negative tests should correspond to individual causes. A multiple-user case should violate only the use-count condition; an unsupported-function case should change only the function kind; a shape-mismatch case should change only correspondence. A negative case containing several invalid conditions proves only that something blocked fusion, not which safeguard did so. Small teaching tests that break one invariant at a time make future changes from “retain” to “fuse,” or back, much easier to explain.

Fusion is worth celebrating when user relationships and intermediate types are preserved, parameters are complete, the ordinary path remains sound, and generated artifacts and numerical tests support equivalence. Only then has the removed node earned a dignified retirement.

Series contents · Previous · Next

• ← Previous Post
• Next Post →