Istio service mesh security audit
30th January, 2023
In this blog post we will detail a security audit of Istio that we recently performed at Ada Logics. The security audit of Istio was categorised as a holistic audit, wherein Ada Logics would (1) Perform a manual audit of the code; (2) Extend Istio’s continuous fuzzing suite; (3) Perform a SLSA (https://slsa.dev) software supply chain assessment and (4) develop a threat model.
Ada Logics previously contributed to Istios security posture during a fuzzing engagement wherein we wrote 50+ fuzzers and found a high-severity CVE that had existed in the Istio code for 1 year.
In our recent security audit, we again had the opportunity to improve Istios security and impact millions of users that interact daily with services managed by Istio. Our overall conclusion with regards to Istios security posture is that Istio is a well-maintained software project with sustainable security practices in place.
We would like to thank OSTIF for facilitating this audit and CNCF for the commissioning.
Istio background
Istio is a platform-agnostic service mesh that allows users to easily deploy a range of routing, security, stability and observability features to a micro-service architecture containing services implemented in any language. Istio was originally developed by Google, IBM and Lyft and is today used by hundreds of organizations to manage clusters of any size with some users using Istio in multitenant clusters with each cluster having up to 5,000 nodes and running up to 100,000 pods. Istio applied to join the Cloud Native Computing Foundation in April 2022 and was accepted at the Incubating maturity level in September 2022.
Notable issues
We identified a total of 11 issues in Istio ranging from informational to high. A high-severity issue is worth highlighting since the triaging stage of it revealed an impact on both Google’s managed Istio offering as well as a vulnerability in Golang. This issue is “10: H2c handlers are uncapped”, ADA-IST-10, in the report. The root cause of the issue was that Istio handled HTTP requests on an h2c connection without setting a limit on their size. This is a security issue, in that Golangs h2c library reads the first request on an h2c connection entirely into memory:
https://pkg.go.dev/golang.org/x/net/http2/h2c#NewHandler
"The first request on an h2c connection is read entirely into memory before the Handler is called. To limit the memory consumed by this request, wrap the result of NewHandler in an http.MaxBytesHandler."
This could allow a malicious actor to send a large request and cause denial of service by way of system memory exhaustion. Fortunately, the vast majority of Istio users were found to not be affected by this, as Istio per default only accepts HTTP/2 with prior knowledge. As such, Istio did not issue a CVE for this issue. Further investigation found that Google's managed Istio offering was affected by this issue, and the issue therefore did have impact in production use-cases.
During analysis of the issue, Istio maintainer John Howard was evaluating the suggested mitigation in the Golang documentation for h2c.NewHandler() - wrapping the result of NewHandler in an http.MaxBytesHandler - and found that the body was not fully consumed which was found to be an http request smuggling vulnerability in Golang that was fixed upstream and published as CVE-2022-41721.
Fuzzing
Ada Logics looked at Istios fuzzing suite to identify areas where this could potentially be improved. Most notably we found that coverage could be improved in security-related parts of Istio which are the most exposed and critical parts of Istios architecture. We wrote 6 fuzzers that we committed upstream and added to Istios OSS-Fuzz integration so that they run continuously. At the end of the audit, the fuzzers had each been running between 90-150 hours. 4 of the fuzzers target security-related components, whereas the remaining 2 target the bootstrap and the kubeinject packages respectively.
SLSA
As part of the audit, we did an assessment of Istios SLSA compliance which found that Istio performs well in the build and source categories but does not make any provenance available to consumers.
You can participate!
We hope that the community, including security researchers, users and maintainers will receive the publicly-shared report as a step to making their own observations and contributions to Istios continued security posture. Istio maintains an inclusive, highly-supportive community with public meetings, a slack channel and the possibility to discuss issues and concerns under “Issues” on Istios github repository.
References