Auditing NPM Modules in Microservices

Where I work at Beamery, we follow Microservices Architecture and have dozens of git repositories. With time, these repos become bloated with old NPM modules that are either outdated or not used anymore at all. I wanted to find a way to quickly audit and clean this mess.

A note from the future: this post predates npm audit, so npm's native tooling now covers the security side of dependency auditing; finding outdated and unused modules across many repos is still a manual affair.

I came across the great npm-check. Its README lists a dozen features; the ones that mattered for us were:

• Tells you what's out of date.
• Kindly informs you if a dependency is not used in your code.
• Interactive Update for less typing and fewer typos via -u.
• Supports public and private @scoped/packages and private registries.

The README also dates itself: at the time it advertised support for the then-new npm@3 alongside npm@2, and for alternate registries like Sinopia, which has long since been archived (Verdaccio is its successor).

To automate running npm-check across all of our repos and generate one coherent report, I created a simple bash-it plugin. First, a small helper both functions below share, which checks whether a command exists:

# Helper function to check if a certain command existscommand_exists () {    type "$1" &> /dev/null ;}

The reporting function, at its core, does the following:

# Generate an NPM report using the npm-check module to inspect the state of our npm modules# The function will check if npm-check is installed and install it otherwise# The report will be generated in the root directory and will be called npm-report.txt
generate_npm_report() {    if command_exists npm-check ; then        find . -maxdepth 1 -type d \( ! -name . \) -exec bash -c "cd '{}' && printf 'Examining NPM modules for '{}'\n' && echo '{}' >> ../npm-report.txt && npm-check >> ../npm-report.txt" \;    else        printf 'npm-check module was not found. Installing now:\n';        npm install -g npm-check        find . -maxdepth 1 -type d \( ! -name . \) -exec bash -c "cd '{}' && printf 'Examining NPM modules for '{}'\n' && echo '{}' >> ../npm-report.txt && npm-check >> ../npm-report.txt" \;    fi}

This script is executed at the root folder that contains all of your repositories, will execute the npm-check command, and aggregate the results in the npm-report.txt at the root directory where you executed your script. The script also checks first if the npm-check command exists or if we need to install it via an npm install -g npm-check.

To go step by step inside the main function:

• find . -maxdepth 1 -type d \( ! -name . \): Will find all the directories within one level down of the current folder
• printf 'Examining NPM modules for '{}'\n' will just print out a message indicating which folder we are currently examining
• echo '{}' >> ../npm-report.txt will print out the folder name examined in the output file npm-report.txt
• npm-check >> ../npm-report.txt This will execute the npm-check command and pipe out the result into the output file

Auditing NPM Modules

After knowing the various modules used, I cleaned my file, pasted the results in an Excel sheet, sorted the cells, and created a subTotal on the count. This generated a list of all my NPM modules and their respective count.

Cleaning out unused NPM modules

The previous function gives us an idea of the modules and frequency used. However, we might have a bunch of unused modules that were left over from old code and will just increase the size of our containers with no actual use.

We can easily clean out those modules by taking advantage of the npm-clean and plug that into a similar wrapper as the function above, reusing the same command_exists helper:

# Clean unused NPM modules from each repository# The function will check if npm-clean is installed and install it otherwise
clean_npm_modules() {
    if command_exists npm-clean ; then        find . -maxdepth 1 -type d \( ! -name . \) -exec bash -c "cd '{}' && printf 'Cleaning NPM modules for '{}'\n' && echo '{}' >> ../npm-clean-report.txt && npm-clean >> ../npm-clean-report.txt" \;    else        printf 'npm-clean module was not found. Installing now:\n';        npm install -g npm-clean        find . -maxdepth 1 -type d \( ! -name . \) -exec bash -c "cd '{}' && printf 'Cleaning NPM modules for '{}'\n' && echo '{}' >> ../npm-clean-report.txt && npm-clean >> ../npm-clean-report.txt" \;    fi}

I hope this helps you in cleaning out your repositories as well.