Speed Up Grouped Operations in R by Reducing Split-and-Combine Overhead

Why grouped operations are harder to speed up

Vectorization matters in R because an R-level loop repeats interpreter work on every iteration: name lookups, argument matching and function setup. A vectorized operation such as `x * 2` pays that overhead once, then processes the elements in compiled code. In a benchmark on 100,000 random numbers, a `for` loop and vector multiplication used roughly the same memory—819 KB and 781 KB—but the loop took 2.22 ms versus 59.02 μs, nearly 40 times longer. `lapply()` and `vapply()` still run a loop; they just hide its syntax. The practical question is how often R has to call back into R, and how much work each call does. Grouped calculations make that harder: an expression such as `x - mean(x)` must be evaluated separately for each group. You can’t remove that loop when the expression may contain arbitrary R code. You can strip away the overhead around it. That’s the problem {poorman} set out to solve. Its earlier grouped verbs used split-apply-combine: make one data frame per group, run the operation on each, then bind the pieces together. That approach is easy to understand, but it slows sharply once group counts reach the hundreds. The author has since rewritten those verbs and draws on the experience in this article, using base R techniques that can also apply in other code. The earlier account is [“Building a base dplyr with primitives: grouped operations, pipes and more”](https://nathaneastwood.github.io/2020/02/27/building-a-base-dplyr-with-primitives-grouped-operations-pipes-and-more/).

Keep the group loop, cut its overhead

To centre a value within each group, a conventional implementation splits the entire data frame, modifies each piece, binds the pieces back together and restores the original row order. A leaner version still loops over groups, but splits only the column it needs, performs the calculation, and combines the results with one `unlist()`. On 100,000 rows, the leaner approach took 3.76 ms with 10 groups, 6.23 ms with 1,000, and 26.27 ms with 10,000. The split-and-bind version took 29.62 ms, 86.71 ms and 522.85 ms, respectively. Memory use also fell: from 26.1 MB to 7.57 MB with 10 groups, and from 55.61 MB to 6.18 MB with 10,000. The gap widens as the group count rises. That’s the point: the group calculation remains, but the machinery around it doesn’t have to be expensive. The article develops six techniques: number groups with `match()`, construct factors directly from those IDs, split columns instead of data frames, combine vector results with one `unlist()`, restore row order with a permutation, and use a specialized grouped operation when one exists. The timing examples use 100,000 rows and typically 1,000 groups; a small sales table illustrates the mechanics.

Assign group IDs in one pass

A straightforward way to find the rows for each group compares every row against every distinct group. With `n` rows and `G` groups, that means `n × G` comparisons, along with repeated logical-vector allocations. In the article’s benchmark, this took 162 ms and allocated 764.87 MB. `match(g, unique(g))` assigned IDs in 653 μs with 2.16 MB allocated. `match()` gives each value the position of its first occurrence in `unique()`, so IDs follow the groups’ order in the data. Both functions use C code and hash tables inside R. The work scales roughly with the number of values examined, rather than requiring a scan of every row for every group. Multiple grouping columns need IDs for combinations of values. `interaction()` can build labels by joining values with a period, but that can collapse distinct combinations: `("a.b", "c")` and `("a", "b.c")` both become `a.b.c`. It can also create levels for combinations absent from the data, consuming memory. The alternative is to number each column with `match()`, combine those numbers arithmetically, then call `match()` again to compact the result into consecutive group IDs. Subtracting one before the multiplication makes the arithmetic use doubles, avoiding integer overflow in R.

Split less, combine once

Once rows have consecutive group IDs, they can serve directly as factor codes. Calling `factor(ids)` would sort and match those IDs again. Building the factor with `structure()` and supplying its `levels` and `class` attributes avoids that extra work. For 100,000 rows and 1,000 groups, splitting with `factor()` took 5.63 ms and allocated 4.2 MB; constructing the factor directly took 1.28 ms and 1.2 MB. This shortcut depends on the IDs being gap-free integers from 1 through the number of groups, which `match()` provides. Splitting a data frame invokes `[.data.frame` once per group, with its general indexing and row-name checks. Splitting columns individually avoids much of that overhead: in the article’s benchmark, column splitting took 2.04 ms and allocated 2.02 MB, compared with 13.01 ms and 3.9 MB for splitting the data frame. There’s another choice: index each column using precomputed row positions, or call `split.default()` on the column. Indexing is quicker when groups are large and few; it makes an R function call for every group, so it gets costly as groups become small and numerous. In tests across 100,000 rows, the two methods were about even at 3,000 groups, or roughly 33 rows per group. {poorman} uses indexing when groups average at least 32 rows and `split.default()` below that threshold. Combining group results has a similar opportunity. `do.call(rbind, pieces)` invokes the general machinery of `rbind.data.frame()` for every piece. If each group produces a plain vector, one `unlist()` can join them. With 1,000 one-row results, the benchmark reported 7.11 ms and 267.74 KB for `rbind`, versus 7.38 μs and 7.86 KB for `unlist()`. The speedup is striking, though `unlist()` drops attributes, so factors and dates need extra handling.

Restore order, or skip the loop

After splitting and combining, results are arranged by group. A grouped `mutate()` must return them in their original row order. The row positions collected during splitting provide the mapping: write the sequence of group-ordered positions back into those row slots, then use the resulting permutation to reorder any output column. Compute that mapping once, and reuse it across columns of any type. Sometimes the group loop can disappear altogether. For group means, base R’s `rowsum()` performs the sums in C; dividing by group counts and indexing the results centres each value. On 100,000 rows, this method took 1.64 ms with 10 or 1,000 groups, and 6.59 ms with 10,000. The general grouped implementation took 3.81, 5.55 and 22.65 ms, respectively. {poorman} can use this approach for verbs with a known operation, but not for `mutate()`, where the expression can be arbitrary. Filling missing values offers another case where vectorized operations can replace per-group iteration. For a vector, multiply each position by a non-missing indicator, then apply `cummax()`; the result points to the most recent non-missing value. Within groups, mark each group’s first position as valid even if it contains `NA`. That prevents a value from carrying over across a group boundary.

Why the redesign matters

The biggest gain in {poorman} 0.3.0 is how its grouped operations scale as the number of groups rises. In the benchmark, every tested verb was faster than in version 0.2.8. With 10,000 groups, the reported speedups range from 21 times to about 3,800 times. The memory figures tell the same story: for example, mutate() drops from 18.72 GB to 11.63 MB.

That result comes from avoiding repeated searches for group membership. Version 0.2.8 repeatedly compared rows against groups, both when creating grouped data and inside grouped verbs. The new approach records each group’s row indices once, then reuses them. Most work now happens across all rows at once; only the steps that need group-specific handling still loop over groups.

The benchmark used 100,000 rows, with 10, 1,000, or 10,000 groups, and compared {poorman} 0.2.8 with 0.3.0 in separate R sessions. The measurements cover group_by(), summarise(), mutate(), filter(), slice_head(), distinct(), and fill(). The exact results will depend on the workload, but the pattern is hard to miss: the older version’s time and memory use balloon as group counts climb, while the new version stays far lower in these tests.

How to try it

Version 0.3.0 was expected to reach CRAN soon. Once it does, install it with:

install.packages("poorman")

Until then, the article gives this GitHub install command:

remotes::install_github("nathaneastwood/poorman")

If you’re using {poorman}, report bugs or differences from {dplyr} on GitHub.

The broader takeaway is practical: these gains came from base R techniques and from arranging the work so each group is identified once. For anyone building grouped data tools, that’s a useful reminder that performance can hinge less on eliminating every loop than on making sure expensive work isn’t repeated inside one.

To comment for the author, visit Random R Ramblings.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and other topics. Find or post an R/data-science job.

Want to share your content on R-bloggers? Submit your blog, or use R-Posts if you don’t have one.

Continue reading: {poorman}: Tips and Tricks for Faster Grouped Operations in base R