"The Codenomicon tools are amazing. Using them is like being attacked by the most relentless adversary who uses every possible method to find flaws in your code
We fixed subtle crash bugs in Samba that had been in the code for over ten years. We would never have found those bugs without the Codenomicon tools.
If you're serious about implementing protocols correctly, you need the Codenomicon tools."
-- Jeremy Allison,
Co Creator of Samba
Fuzzing Performance
Introduction |
Features and Benefits |
Test Results |
Whitepaper
Test Results
Theoretical Maximum Speeds
As a part of our study, we measured the theoretical speeds for the HTTP and TLS Server test tools using a dual-core PC with 2 GB of memory. The theoretical speeds were as follows:
- HTTP Server: 1000+ test cases per second per test suite
- TLS Server: 200+ test cases per second per test suite
On the faster servers, the theoretical speed per test suite did not vary significantly with the number of test suites running, but the speed increased linearly as more and more test suites were launched in parallel.
Fuzzing Performance Results for HTTP
The HTTP protocol was chosen for the study because it is a simple request-response protocol over TCP/IP. A total of 170 HTTP server test tools were started before the 96 GB of RAM available was consumed. In total, during test execution 340 test generation processes and 235 multi-threaded Apache processes were running at the same time. The best average test case speed was 16.000 test cases per second with 40 parallel suites, each running in 5-threaded mode. This results in 200 parallel fuzzing sessions.
Fuzzing Performance Results for TLS
The SSL/TLS protocol is much more complicated than the HTTP protocol. In total of 125 SSL/TLS server test tools were started before all of the 96 GB of RAM available was consumed. The best fuzzing performance for the SSL/TLS tool was 2400 test cases per second with 40 parallel test suites each running in 5-threaded mode. This results in 200 parallel fuzzing sessions.
Performance of Multi-threaded vs Multi-process Fuzzing
In our study, we did not observe a significant difference in performance between the multi-process and the multi-threaded execution. In the figure below, the curve that climbs faster depicts Defensics instances running in 5-threaded mode, whereas the other curve shows how the fuzzing performance of single-threaded Defensics instances scales up as more test generating processes are started.
From our tests, we found that each Defensics test case generator seems to consume approximately 60-70% of one CPU core. The most significant benefit of multi-threaded fuzzing is that the execution threads can use the same protocol model, and thus consume less memory. On the other hand, multi-threaded operations can create some CPU overhead, because they share the same process resources.
Conclusions
The theoretical test execution speed only depends on the number of CPUs available. In real test execution, the execution speed is approximately 40% slower than the theoretical speed due to the limitations of the network and operating system, such as the amount TCP sockets. Another bottleneck arises from the logging options: as extensive documentation is generated for each test case being executed. With zero logging, the test speed increase was approximately 20%. Test execution can be further optimized by using a fast harddisk such as SSD.
Recommended Hardware
Our study was conducted on a 32-core Dell PowerEdge R910 server. Dell recently released a new PowerEdge R815 server with 48-cores (4 x 12-core AMD CPUs), which is able to run approximately 80 test generator processes in parallel. You should not need more than 64 GB of memory to run the 80 Defensics test instances in single threaded mode and 16 GB of memory should suffice when running them in 5-threaded mode. The SSD harddrives allow for sufficient logging, even with this type of load. 1Gb NIC might not be enough to process all packets, so either use several 1Gb NICs, or try a 10Gb NIC.
We look forward to trying out other hardware configurations. For example, on the basis of our study, the 48-core Dell PowerEdge R815 could potentially improve performance by approximately 20-50% compared to the Dell PowerEdge R910 that was used in the research.
Acknowledgements and Future Work
Codenomicon would like to acknowledge the excellent support from the Dell team that helped us complete these tests.
Codenomicon will add more test results here as they are available. Please do not hesitate to submit your own speed records to us.
