We need to scan for vulnerabilities in our container images. Vulnerabilities are publicly documented with a CVE - Common Vulnerabilities and Exposures - number, assigned by CNAs (CVE Numbering Authorities). The National Vulnerabilities Database (NVD) lists these vulnerabilities for visibility and usage.

Traditionally, we scan for vulnerabilities in software by listing out software installed on a host (via the root filesystem + software installed by a package manager like npm or brew + manual / adhoc installs via wget or curl (not a great idea)). Software can be updated or patched directly on the host.

Nowadays, SSHing and patching/updating is not a good practice - we have containers, and we can treat our containers as immutable. Containers that download their own software are hard to manage uniformly. You can keep a container immutable by having the filesystem read-only, and storing writes on some kind of tmpfs. Immutable containers have many advantages:

• You only have to scan/rescan one image instead of potentially thousands of containers, since all containers spawned from that image will be identical.
• You can build vulnerability scanners into your CI/CD pipeline, close to your image builds - this is a form of “shift left”. Then you can scan locally, on build, and also once the image hits your registry.
• You can use admission controllers and policies to reject images that fail a scan (remember to think deeply about how you identify an image! By tag? By digest?).
• You can rescan images every 24 hours as vulnerability databases update (remember, lots of vulnerabilities are found in software that is years, or even decades, old!).
• Scanner can detect things beyond vulnerabilities, like:
  • setuid bit set
  • Images running as root
  • Secrets injected improperly

So immutable containers + scanners in CI/CD are deeply helpful. What are some pitfalls?

• Vulnerability sources can be out of date, or perhaps encompass vulnerabilities that aren’t applicable to your distribution (false positives).
• Some vulnerabilities are listed as “won’t fix” - should you ignore them in your scanner?
• Some packages allow installation with only a subset of subpackages - like, installing just docs from the bind Ubuntu package shouldn’t get you in trouble for broader bind vulnerabilities.
• Nothing will 100% save you from 0-days or non-public nation-state issues.