Services
Training
About
Blog
Contact

CRI-O holistic security audit engagement

6th June, 2022
David Korczynski,
Security Research & Security Engineering
Adam Korczynski,
Security Engineering & Security Automation

Ada Logics Ltd. recently performed a holistic security audit of CRI-O. CRI-O is an implementation of the Kubernetes Container Runtime Interface and is used as a core component in Kubernetes clusters for orchestrating containers on each node. This makes it an important software package as issues in CRI-O can have far reaching consequences as it’s a key part of many Kubernetes Clusters.

The key security finding from the engagement is a high-severity denial of service issue. It’s a node-level denial of service attack for anyone with the ability to create a pod on a Kubernetes cluster. In essence, if one can create a pod on a node in a given Kubernetes cluster then one can cause a denial of service attack by way of memory exhaustion. This is for nodes that rely on the CRI-O runtime, and, interestingly, the vulnerability is also existing in another popular container runtime, Containerd.

We would like to thank the CRI-O maintainers for the collaboration. This engagement was funded by the CNCF and facilitated by OSTIF. In addition to ou work Chainguard participated in the audit to conduct a software security supply chain audit. We would like to thank all of the collaborators.

In this blog post we will give an overview of the engagement and full details can be found in the report on the CRI-O repository here.

Summary of audit and results

The goal of the engagement was to perform a broad analysis of the security posture of CRI-O, divided into the tasks:

  1. Threat model formalisation of CRI-O
  2. Security auditing of the code
  3. Fuzzing integration of CRI-O, including continuous fuzz integration by way of OSS-Fuzz
  4. Review of documentation and testing

The primary security finding of the work is a single high-severity issue. A few minor issues were found as well, however, our view from completing this audit is that CRI-O is a well-written project that has a high level of security assurance.

Cluster denial of service by way of deployments

The high severity finding is a denial of service attack on a given cluster by way of resource exhaustion of nodes. The attack is performed by way of pod creation, which means any user that can create a pod can cause denial of service on the given node that is used for pod creation. The CVE for the this vulnerability is CVE-2022-1708 and Github advisory can be found here: https://github.com/cri-o/cri-o/security/advisories/GHSA-fcm2-6c3h-pg6j

Interestingly, the denial of service attack also occurred in other container runtime interface implementations, most notably Containerd. Specifically, the exact same attack that exhausts memory in CRI-O can be used to exhaust memory of Containerd. The CVE for the containerd issue is CVE-2022-31030 and the Github security advisory for containerd can be found here: https://github.com/containerd/containerd/security/advisories/GHSA-5ffw-gxpp-mxpf.

In the Kubernetes world, for both CRI-O and Containerd this issue can cause a denial of service of nodes by way of deployments without any user interaction, effectivelly enabling a Cluster DOS if the cluster uses any of these container runtimes interface implementations. Patching is highly recommended.

Continuous fuzzing integration of CRI-O

We integrated an extensive fuzzing suite targeting the CRI-O infrastructure. The main challenge of this was to set up infrastructure to make fuzzing of CRI-O work, which is difficult because CRI-O is an interconnected system and relies on many components, e.g. binaries on the system and is mainly communicated with by way of Kubelet which constrains the data sent t CRI-O, and also uses a fairly complex testing framework that involves many many mocks.

In summary, we implemented 14 fuzzers targeting the CRI-O code, as well as the containers/image repository and containers/storage, and integrated the project into OSS-Fuzz. The fuzzers are available at https://github.com/cncf/cncf-fuzzing/tree/main/projects/cri-o and the OSS-Fuzz integration is available at https://github.com/google/oss-fuzz/tree/master/projects/cri-o.

The primary focus of the fuzzing was to target the gRPC handlers. This is mainly done by fuzz_server which is a fairly large fuzzer consisting of 900 lines of code. This fuzzer initiates a gRPC server and sends sequences of random messages to the server. In this way, the fuzzer has a significant reach throughout the code of CRI-O.

References