Grouped Convolution: Once You Slice the Cake, Slice the Labels Too

Series contents · Operators and Layouts · 阅读中文版

“Ordinary convolution already works. Surely grouped convolution just needs one more parameter?”

That sentence is about as dangerous as “this is only a one-line change.” An integer in an operator definition can become an entire graph in the backend: split the input, split the weights, split the bias, and concatenate the outputs. Worse, the integer weights have quantization metadata beside them that explains what those integers mean. Slice the cake correctly but hand out the wrong labels, and it will still taste wrong.

Grouping and Legality

First, narrow the problem down. Consider a teaching input X:[1,6,5,7], interpreted as NCHW, with 10 output channels and two groups. Each group has 3 input channels and 5 output channels. The logical weight shape is [10,3,KH,KW]. We want to reuse an ordinary convolution path that already works reliably, rather than build another implementation from machine instructions upward.

The mathematical meaning of grouping can be written directly:

1
2
3
4
5

for each group g:
    Xg = X[:, g*3:(g+1)*3, :, :]
    Wg = W[g*5:(g+1)*5, :, :, :]
    Yg = Conv(Xg, Wg, bias_g, group=1)
Y = Concat(Y0, Y1, channel_axis)

Every slice here has a step of 1, and the concatenation order is not arbitrary. The eighth output channel is the third output channel of the second group. Reverse the Concat order, and the shape, element count, and even value range may still look fine while the meaning is completely wrong.

The more mature the reusable backend support, the more attractive decomposition becomes. The historical implementation used a dedicated lowering with a higher match priority for supported grouped cases, then sent the generated ordinary convolutions through the existing path. Ordinary convolution itself remained with the original rule. This avoids making one enormous lowering responsible for the ordinary path, the grouped path, and every failure diagnostic.

The catch is that “we can decompose it” does not mean “all group counts are supported.” The inspected implementation accepted a finite set of group counts and required static four-dimensional inputs, weights, and outputs. The group count in our example is a mathematical illustration, not a device capability list. A real compiler must express target restrictions explicitly; channel divisibility alone is no promise that the hardware can execute the result.

Legality checks should ideally finish before new nodes are created. Both input and output channel counts must be divisible by the group count. The weight’s output channel count must describe the full output, while its input channel count must describe a single group’s input. Output spatial dimensions also depend on convolution stride, dilation, and boundary conditions, so decomposition must inherit those attributes. Copying only the shape and forgetting the attributes is like copying a recipe’s title and guessing the cooking temperature.

Bias and Quantization Follow the Channels

Bias deserves its own interrogation. “The bias is a vector” describes a common case, not the whole contract. The changes distinguished no bias, a single-element bias, and one bias per output channel. A single element can be reused; per-channel bias must be sliced along the axis that actually carries channels. A 10-element bias might have shape [10] or [1,10,1,1]. Matching the element count is necessary, and identifying the channel-bearing axis is necessary too. For more general multidimensional broadcast biases, “find an axis with the right length” is not a sound generalization.

Weight quantization is the part most easily underestimated. Suppose each output channel has its own scale:

1
2

s = [s0,s1,s2,s3,s4, s5,s6,s7,s8,s9]
q_weight[g] corresponds to real_weight = s[channel] * (q - z[channel])

The second group’s weights must take s5...s9 with them, rather than start again at s0. Otherwise, the integer multiply-accumulate can be perfectly correct while dequantization shifts the entire result. The same applies to per-channel zero points and other channel parameters. A per-tensor parameter has one value and can be reused unchanged; a multivalue array requires checks on both its length and its quantization axis. The historical code explicitly checked the per-channel quantization axis and derived new metadata records for each group.

An entertaining failure mode is that an incorrect split almost disappears when every scale is equal. A test with identical scales across channels can verify dataflow but does little to prove that the labels moved with their channels. A teaching test should give different groups clearly different but reasonable scales, then choose inputs that avoid saturation so that the mistake becomes observable. Do not make every parameter identical merely to make the test easy to pass.

New weights have new names, so an external quantization mapping needs additional entries as well. The historical changes added merge-and-save logic: retain entries unaffected by the transformation, then add encodings for the generated weights. This hides a compiler engineering problem: graph rewriting and updating external metadata are not inherently atomic. If the graph has already been replaced and saving metadata fails, how should the failure propagate, and how do we prevent an incomplete artifact? The inspected code contains error diagnostics, but that does not establish a complete transaction. A more robust design would build the full plan and derived metadata first, validate them, and then commit.

The Cost of Decomposition

Decomposition also comes with a performance bill. Suppose the logical output contains E elements of b bytes each. If Concat really moves data, then, ignoring alignment padding and cache effects, it may add approximately E·b bytes of reads and E·b bytes of writes, or about 2E·b in total. If the backend lets each group write directly into disjoint regions of the final output, some movement may disappear. Slices, too, may be views or copies; the number of nodes in a high-level graph does not settle the question. More small convolutions also mean more launches and less parallel work within each task. A larger group count is not automatically faster.

Here is a measurement plan that has not been run: hold the total input and output channel counts constant and vary only the group count. Record the number of nodes after lowering, actual bytes moved, task launches, and end-to-end time. Separate small spatial dimensions from large ones. Launch overhead can dominate the former, whereas data movement and arithmetic throughput may matter more for the latter. Without those measurements, we can say “the implementation reused ordinary convolution support,” but not “it significantly improved performance.”

Regression, Alternatives, and Convergence

Regression tests should follow two tracks. One checks graph structure: slice counts, channel boundaries for each group, the group attribute on ordinary convolutions, the output concatenation axis, and complete removal of the original grouped node. The other checks semantics: distinct constants and quantization parameters in different groups should demonstrate that groups do not interfere. Historical tests also checked for quantization entries and included rejection cases for unsupported group counts and nondivisible channels.

Another historical test expansion prepared depthwise-convolution models with different boundary conditions and strides, using IR assertions to confirm that they took the dedicated depthwise path. Depthwise convolution is a special case of grouping, but a compiler need not expand it into many ordinary convolutions. A dedicated implementation may have different weight layouts and performance properties. The mathematical relationship can be unified; the engineering path needs evidence.

Code review can also become a conservation check before and after grouping. The union of input channel slices should cover the original input channel interval, and adjacent slices should not overlap. The output channel counts of all groups should sum to the original output channel count. Each weight chunk’s intended input group should match the input slice it actually consumes. The first relation prevents omissions, the second guards against an incorrect total, and the third prevents crossed connections. These relations can be checked at compile time, without waiting for a complicated model’s final output to provide an ambiguous clue.

To test cross-group leakage, set all inputs in group zero to zero and activate only the other group. A convolution without cross-group connections should leave group zero’s output at zero except for the bias contribution. Disable the bias separately, or give it an easily recognized value, to distinguish a wrongly sliced bias from a wrongly sliced input. This produces a clearer conclusion than randomizing every input at once. If a later normalization mixes channels, observe the result after concatenation and before that next kind of operation.

“What if we treat all groups as one ordinary convolution and fill the cross-group weights with zeros?” That is an interesting alternative. Mathematically, it embeds grouped connections in a block-diagonal structure. In engineering terms, however, those sparse zeros may be processed as part of a dense matrix. Assuming equal input and output channel counts per group, increasing the number of groups increases the useless cross-group multiply-accumulates in the zero-filled version. Unless the backend recognizes and skips those zero blocks, a simpler graph may buy larger weights and more computation.

There is also an apparently administrative question: what should the new nodes be called? Names often support diagnostic tracing, quantization metadata association, and artifact inspection. Adding a group index to a name is intuitive, but collisions still need attention when original nodes share a name, a pass runs repeatedly, or derived weights already exist. A more robust design could separate identity from display names and connect metadata through stable internal references. The historical implementation used derived names; that does not prove that every collision was handled. It is a useful failure case to add.

Finally, rule priority belongs in regression testing. Running the dedicated grouping rule first, then handing the generated ordinary convolutions back to existing lowering, resolves complex semantics in stages. But a broader rule could consume the node first, or a generated node could retain its old group count, bypassing expected checks or triggering decomposition again. Tests should establish a fixed point: running the optimization again must neither add more group nodes nor change their slice boundaries. This checks convergence of the transformation, rather than whether one invocation happened to produce the right count.

When a colleague asks again whether this can be done by “just adding a group parameter,” ask for answers to four concrete questions: which channels belong together, how the weights are sliced, which data the quantization parameters follow, and where the output lands. Answer each one, and decomposition becomes a design instead of a lucky round of string replacement.

Series contents · Next

• ← Previous Post
• Next Post →