Fuzzing 100+ open source projects with OSS-Fuzz - lessons learned.
31st August, 2021
In this blog post we will describe our efforts of integrating continuous fuzzing into more than 100 open source projects by way of OSS-Fuzz that has resulted in more than 2000 issues being reported, 1300 of which are verified and fixed. This is an effort of work that was split out over the duration of a year and the goal of the blog post is to give insights into the end-to-end process of integrating projects into OSS-Fuzz as well as highlight some of the interesting results that we encountered.
The work described in this post was carried out by David Korczynski and Adam Korczynski from Ada Logics with a lot of support from maintainers of each open source project discussed in this post and from the entire OSS-Fuzz team.
TLDR: OSS-Fuzz is a service run by Google for fuzzing important open source projects. We integrated 115 projects into OSS-Fuzz and 2104 bugs have been reported in these projects, divided into 1545 generic bugs and 559 security-relevant bugs, of which 1330 bugs are verified and fixed.
A complete list of projects integrated and data for each project are available here.
Continuous fuzzing and OSS-Fuzz
In recent years the concept of continuous vulnerability analysis has gained significant attraction. The idea is to implement vulnerability analysis techniques into a given project and then run these techniques in a continuous manner. Fuzzing is one of the techniques that has been well-suited for continuous analysis, since a fuzzer as a concept works by continuously exploring the code of a given target by building up inputs that trigger unique code paths in the target code. Running a fuzzer continuously allows the fuzzer to incrementally build up the corpus of inputs, which is important for the fuzzer to explore new code.
OSS-Fuzz is a free service that provides infrastructure and computing resources for doing continuous fuzzing of important open source projects. From a developers point of view, if you write fuzzers for your open source project then OSS-Fuzz provides the infrastructure for running these fuzzers, analysing crashes, sending out bug reports and much more. There are already many important open source projects of varying nature integrated into OSS-Fuzz, such as curl, libxml, NGINX and the Envoy proxy.
OSS-Fuzz has been around for a few years now and the first project was integrated into OSS-Fuzz in late 2016. Since then the number of projects fuzzed by OSS-Fuzz has steadily increased as the following Figure shows. The Y-axis shows the number of projects in OSS-Fuzz and the X-axis shows the date.
The motivation for OSS-Fuzz is simply that critical modern software products rely on open source libraries and these libraries need to be secure for the software products to be secure. Chromium is a prime example of this in that the “third_party” folder of the Chromium Source code https://chromium.googlesource.com/chromium/src.git/+/refs/heads/master/third_party/ contains more than 240 open source projects. These projects must be analysed for security issues, and OSS-Fuzz provides one way of doing this. In addition to this, OSS-Fuzz sponsors the projects by offering rewards for project integrations, which can be used to justify the time invested in the integrations.
Results from integrating 100 projects
We started contributing to OSS-Fuzz in April 2020. We did it largely because we are doing a lot of fuzzing for our clients and wanted to contribute some of our skills and efforts to open source projects. However, after having made a few contributions we started enjoying it a lot and began to devote more time to it. Now, more than a year later we have integrated more than 100 open source projects into OSS-Fuzz. In this way it was not planned from the get-go for us to contribute this much, but the gratification of working with open source security and contributing our skills to many different projects made us continue contributing.
The projects that we have integrated have a wide range in terms of complexity. On one end we have smaller projects like simple json parsers, and on the larger end we have integrated large projects that are central projects across the software landscape. Projects we integrated include:
The difficulty as well as the process of integrating each project can vary a significant amount, and we describe this in more detail in the next section.
To give an estimate of the bugs found by our efforts we looked at Monorail and did various searches on all the projects where we are in charge of the initial integration. In this blog post we discuss the public data available as there are bugs under embarco due to the OSS-Fuzz disclosure deadline.
Our efforts and results can be summarised as follows:
- Integrated 115 projects into OSS-Fuzz
- 2104 bugs have been identified divided into 1545 generic bugs and 559 security-relevant bugs.
- 1330 bugs have been verified and fixed
- 774 bugs are either not fixed, declared WontFix or are false positives.
It is important to note that there is an error-margin in terms of true bugs amongst the 2104 total bugs, as some of these bugs will be due to issues in the fuzzers or infrastructure. In this sense the bugs were indeed correct in that the bugs were present, however, some bugs may not have been in the code of the project being analysed itself. We estimate a 15-20% error margin.
Throughout the process we integrated projects written in C, C++, Python, Go and Rust and the types of bugs we found across the projects are a reflection of the language the project was written in. Typically, for managed languages the bugs are within the umbrella term of uncaught exceptions and denial of service bugs, whereas in native languages the bugs are mostly split between assert violations, NULL-dereferences, heap-out-of-bounds, stack-out-of-bounds, stack overflows, integer arithmetic, memory leaks, out-of-memory and timeout bugs.
For a complete overview of the projects we integrated and the bugs reported, please visit the following spreadsheet.
Interesting project stories
There are some bugs and projects that stand out and we will go through a few of these now. There are many more interesting and similar stories to the ones below but we will not go into details with more to avoid having the blogpost excessively long.
Go-ethereum is likely the project that had the biggest moment in our efforts and also contains the bug that likely received the most attention. We integrated Go-ethereum into OSS-Fuzz (we only did the integration as Go-ethereum already had a large fuzzing suite) in October and shortly after a bug was found by the fuzzers in Golang itself. The bug could consequently take down major parts of the Ethereum network and cause a massive denial of service. The bug was assigned CVE-2020-28362 and is detailed on the Ethereum blog on Geth security release Additionally you can read about it on our blog in a post on: the importance of continuiuty in fuzzing.
Sudoers is another project that stands out. Earlier this year Qualys reported a privilege escalation issue in the Sudo tool and following the public release of the bugs we decided to integrate Sudo into OSS-Fuzz. In less than 48 hours after the bug release Sudo was integrated into OSS-Fuzz with continuous fuzzing analysing the project and as you can see in the spreadsheet of bugs Sudo has already fixed 26 generic bugs as well as 9 security relevant bugs.
Sudo developers took over the work and created an extensive integration that you can read about on their blog here. It is noteworthy that the current fuzzing set up is able to detect the bug discovered by Qualys.
Pidgin is another interesting open source project. In June 2021, large bounties were offered for exploits against the project corresponding to four times what the maintainer of the project were sponsored in 2020 (see the following tweet). We integrated the project in 2 days after the tweet here.
We have also added several important Cloud Native projects like Istio, Prometheus, ContainerD and Vitess. These are more difficult to fuzz in comparison to standalone libraries, e.g. parsers, but so far we have seen some interesting results. For example, we integrated Istio into OSS-Fuzz December 2020, and we have now contributed more than 10 fuzzers and slightly more than 20 issues have been reported by OSS-Fuzz. Several of these projects have now written blog posts about their fuzzing integration, e.g. Teleport and Linkerd.
Wasm3 is another successful project. We integrated it in April 2021 and since then more than 100 issues has been reported by OSS-Fuzz with most of these being verified and fixed now. In this case the Wasm3 authors themselves were responsible for writing the fuzzers and had the first fuzzer written dating back to 2019.
Fluent-bit log processor is another project that stands. Fluent-bit is a project written in the C language and we integrated it into OSS-Fuzz around May last year, and since then the fuzzing has found more than 150 bugs in the project (granted there is an error margin as described above), many (if not all) which are fixed now.
The OSS-Fuzz process
During our engagement the, perhaps, largest insight we have gotten is about the end-to-end process of integrating an open source project into OSS-Fuzz from a security researchers perspective. In this section we will try to outline this process and describe some common problems that have to be overcome in each step of the way.
The complete process of integrating a project into OSS-Fuzz varies a great deal from person to person. It depends on your background knowledge as well as whether you’re a maintainer of a project or an external engineer looking to integrate a project. In the following we describe the process as it is for a security engineer that is well-seasoned in fuzzing but has no knowledge about the project that is to be fuzzed.
Step 1: Find a relevant open source project
The first step is to find an open source project that is relevant. This includes a project that has security relevance for the broader community. To do this, the best way is often to select important open source projects and (assuming these projects are already integrated into OSS-Fuzz) then look at the dependencies on this project. Another important element of a given project is that it is actively maintained. At the end of the day, for a project to be integrated into OSS-Fuzz you must ensure that the maintainers of the project are willing to engage. The project will never get integrated if there is no participation or engagement from the maintainers. The OSS-Fuzz maintainers are of great assistance here and keep a list of relevant and desired open source projects in the OSS-Fuzz repository itself in issue 402. Another great resource is looking at the newly released security score index which has data available in the public OSSF criticality score
Step 2: Understand if the project has potential for it to be fuzzed
Some applications are more prone to fuzzing than others, and some can be extremely difficult to integrate. It is important early in the process to extract a definitive conclusion on whether the project is suitable for being fuzzed.
From a technical perspective, we are interested in understanding if the code can be fuzzed. For example, signs that make a project harder to fuzz can be that the code often fails by killing the process or if the code is not meant to be used as a library. Some of the signs of a project that is friendly towards fuzzing are self-contained libraries with a clear API. In this step, the purpose is to understand what the scope of the integration is and determine whether the effort may be better elsewhere.
From a logistics perspective we are interested in understanding if the project is active and if the maintainers are happy to have the project fuzzed. This is a must for the project to be integrated into OSS-Fuzz, as otherwise OSS-Fuzz will devote a significant amount of computing resources to producing results for a project that will not use the results.
Step 3: Make the target compilable with fuzzer instrumentation
The third step is to convert the build process so it can be compiled with the appropriate compilers and sanitizers. This step differs significantly depending on what language the target is written in. Assuming we are dealing with a C/C++ project this step involves converting the project such that it can compile with Clang and also be instrumented with sanitizers.
This step is non-trivial, and for larger projects this step can easily be the one that requires the most effort. Some build systems are very complex, and in addition to this, converting a build system is often less fun than writing the actual fuzzers, which adds to the burden of the task. However, doing this step properly can save great deals of resources in later maintenance, and also assist a lot in making it easy to write fuzzers at a later stage. There are many different build systems used by open source projects but you will come a long way knowing about Make, CMake, ninja and bazel.
Step 4: implement a fuzzer for the given project
The fourth step is to implement the actual fuzzer. This is where the traditional “fuzzing” action is, however, in the majority of initial integrations we have found this to be the smallest part of the engagement. Once step 1,2 and 3 have been completed, the chances are you already know the source code of the project well, and in consequence knows some APIs of the target that are well-suited for being fuzzed.
Step 5: Integrate the project into the OSS-Fuzz infrastructure.
The fifth step is to create the proper set up for OSS-Fuzz. The artifacts needed here is a Dockerfile for setting up the environment for your project as well as a build file that will build the fuzzers. Most often, this will be a natural progression from the earlier steps, and thus this has a fairly low effort. However, an important step in this process is to ensure the maintainers are onboard with the integration, which includes collecting email addresses for the maintainers of the project that will be used for receiving error reports.
Step 6: merge your code into OSS-Fuzz Github repository.
The final step in the process is to merge your setup in step 5 into the OSS-Fuzz Github repository. This involves a pull request with the build file, Dockerfile as well as the project’s yaml file. Sometimes, if the maintainers of the project prefer to have the fuzzers outside the upstream repository, your pull request will also include the fuzzers for your target project. Once you have completed your pull request, the OSS-Fuzz team at Google will review your submission and then you are good to go!
Next steps: collect integration reward and continue work
Once you have completed an integration OSS-Fuzz offers an integration reward of 500-1000 USD. This is a nice reward for your work, and following the non-trivial effort of integrating your project it is of great pleasure to receive a monetary award.
The next step is then to continue working on the project or progress to integrating another project. An initial integration of a project is the first step in a long process of integrating continuous fuzzing into a project. The steps towards integrating a project with ideal continuous fuzzing are (1) integrate more fuzzers of the project so the code coverage of your fuzzers extend to the entire target and (2) fix the bugs that your fuzzers have found.
In terms of results, we have found the optimal set up is to have the maintainers of the project engage in the work and in particular the two tasks mentioned above. The distinguishing factor is simply that the developers are the ones that have the most knowledge about the target code which is instrumental in setting up fuzzing infrastructure for a project. Indeed, in the projects that we have integrated, the ones that are most successful in terms of bugs found are the projects where the maintainers take over the integration and start contributing fuzzers themselves. As such, encouraging the maintainers to engage with the fuzzing is an important step and perhaps more impactful at times in comparison to writing fuzzers. Naturally, the maintainers are often busy with a lot of tasks so it is crucial to work at the pace of the maintainers rather than assume they can devote much attention to the project.
Conclusions and looking forward
We spent significant effort over the last year to integrate continuous fuzzing into more than 100 open source projects. It has been a great pleasure to do this and the engagement with the open source community is a great work environment. The tangible result is that more than 2000 bugs have been reported and many of these have been fixed already.
Our efforts and results can be summarised as follows:
- Integrated 115 projects into OSS-Fuzz
- 2104 bugs have been identified divided into 1545 generic bugs and 559 security-relevant bugs.
- 1330 bugs have been verified and fixed.
- 774 bugs are either not fixed, declared WontFix or are false positives
For a complete overview of the projects we integrated and the bugs reported, please visit the following spreadsheet.
Going forward our plan is to continue contributing to OSS-Fuzz. However, over the next year we plan to focus more on depth rather than breadth, and thus focus more on fewer projects of higher importance rather than a large set of projects.
We hope this blog post will help developers and security researchers understand the process involved in integrating projects into OSS-Fuzz and ultimately contribute towards securing open source software.
We also hope this blog post sheds light on the impact of securing open source software and encourages more technology companies to provide monetary support for this type of work. The financial rewards offered by Google were crucial to justify allocating the time to perform the integrations. We hope to see more support for this type of work in the future.
Finally we would also like to thank the entire OSS-Fuzz team for patiently reviewing our work. Throughout the entire process we have been very impressed with their devotion to securing open source software.