Best Practices: Managing npm cache and memory footprint during npm ci in large CI/CD pipelines #206751
🏷️ Discussion TypeQuestion BodyHi everyone 👋 We are currently optimizing our CI/CD pipelines (using GitHub Actions and GitLab CI) for a large-scale monorepo, and we've run into a couple of nuanced behaviors with npm ci that I’d love to get feedback on from the community. Memory Consumption & Garbage Collection: During npm ci, especially in memory-constrained CI runner environments (e.g., 2GB-4GB RAM containers), we occasionally observe high memory spikes or out-of-memory (OOM) kills when extracting large transitive dependency trees. What is the recommended way to regulate Node’s memory ceiling specifically for the npm CLI without affecting the downstream build steps? Is passing --max-old-space-size via NODE_OPTIONS the standard approach, or does npm CLI have native thread/concurrency flags we should tweak? Cache Isolation vs. Persistence: We cache ~/.npm across pipeline runs to speed up installations. However, over time, the cache directory inflates significantly unless explicitly pruned. Is running npm cache clean --force after every CI run considered an anti-pattern when persistent caching is active, or is there a lighter, recommended strategy to prune unreferenced tarballs without invalidating the entire global cache? Would love to hear how other teams handle npm cache lifecycle and memory tuning at scale! |
Replies: 2 comments 1 reply
|
Thanks for the detailed questions. There are two separate issues here, so I'd handle them independently. 1. Memory & concurrency
If the issue is related to npm's network concurrency, you can also limit the number of sockets used during installation: npm ci --maxsockets=5This can help reduce concurrent network activity and the associated memory pressure, especially in resource-constrained or isolated Docker runners. One important caveat: make sure the container actually has enough memory available. Setting 2. npm cache hygieneI wouldn't recommend running: npm cache clean --forceon every CI run. If you're using GitHub Actions caching, a better approach is to key the cache against the lockfile: key: npm-cache-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}That way, changes to For occasional maintenance, you can use: npm cache verifyThis checks the cache and removes invalid or unnecessary entries without throwing away the entire cache. So my recommendation would be:
|
Thanks for the detailed questions. There are two separate issues here, so I'd handle them independently.
1. Memory & concurrency
NODE_OPTIONS="--max-old-space-size=4096"is a reasonable way to increase the V8 heap limit when Node is hitting OOM errors, but it doesn't necessarily prevent memory spikes by itself.If the issue is related to npm's network concurrency, you can also limit the number of sockets used during installation:
This can help reduce concurrent network activity and the associated memory pressure, especially in resource-constrained or isolated Docker runners.
One important caveat: make sure the container actually has enough memory available. Setting
-…