Missing packages in registries

Adding all dependencies to Bytesafe as part of an install

Issues with missing dependencies in a registry after npm install are typically related to the npm client using another registry or a local cache.

Depending on the npm version used, this can occur even when --registry URL has been explicitly specified.

In such cases, as the request never passed through Bytesafe, no package versions will be added to the specified registry.

Two common explanations:

  1. Dependencies are already cached locally in the environment by npm cache
  2. Project has an existing package-lock.json file (or other lockfile) specifying another registry URL

Both npm cache and package-lock.json are key parts of npm, but when adding packages to a registry they can potentially take away from the desired behavior.

Let’s look at how we can fix issues with missing packages by forcing installs to use Bytesafe registry as the package source.

Force requests to bypass the local npm cache

The npm client has its own local cache that is preferred over external fetch of packages.

The cache contents will be checked and used before attempting to retrieve packages from any external source.

For users that have previously added the same package versions to their local environment, the cache will contain the requested versions and requests will be served by the cache and not any external source.

In most cases this makes perfect sense since the cache improves installation times. But when installing to new registry, this is not desired behavior.

Users can choose to clear the cache with npm cache clean, use --prefer-online to force updates even for cached data or to instead bypass the regular cache for a request with --cache.

# Clean the npm cache before installing dependencies

$ npm cache clean --force

...

# Force the npm cli to look for updates even for cached data

$ npm --registry https://{WORKSPACE}.bytesafe.dev/r/{REGISTRY}/ install --prefer-online

...

# Specify another temporary empty cache directory with --cache

$ npm --registry https://{WORKSPACE}.bytesafe.dev/r/{REGISTRY}/ install --cache /tmp/

...

For more information on npm cache refer to the official npm documentation .

Managing lockfiles

The purpose of the lockfiles is to be able to reproduce a specific node_module state when installing dependencies, including what source was used for each package version.

In the case of adding a projects dependencies to a new registries, this is partly undesirable. Existing lockfiles (created using another registry) may supersede the input provided by the user.

To add dependencies when using a new registry, users should delete and regenerate any lockfiles.

Common lockfiles are:

  • package-lock.json
  • npm-shrinkwrap.json
  • yarn.lock

The lockfiles contains registry url information as part of its resolved information for each package dependency.

...

"dependencies": {

"some-pkg": {

"version": "7.3.0",

"resolved": "https://registry.npmjs.org/some-pkg/-/some-pkg-7.3.0.tgz",

...

So even when specifying --registry URL, an existing lockfile may supersede this, and use the resolved path as the package source. The exact behavior depends on the version of npm client.

This is highlighted in the npm docs with: “If the package has a package-lock, or an npm shrinkwrap file, or a yarn lock file, the installation of dependencies will be driven by that."

Remove old lockfiles and generate new

Users should delete any existing lockfiles and generate new ones when using a new registry.

# Go to project directory

$ cd your/project/path/

# Delete existing package-lock.json

$ rm package-lock.json

# Generate new lockfile with npm install

$ npm --registry https://{WORKSPACE}.bytesafe.dev/r/{REGISTRY}/ install

...

Commit new lockfile to repository

Remember to commit new and updated lockfiles to the project repository. Making sure collaborators will get the same dependencies from the registry when using npm install or npm ci in the future.