"The Codenomicon tools are amazing. Using them is like being attacked by the most relentless adversary who uses every possible method to find flaws in your code

We fixed subtle crash bugs in Samba that had been in the code for over ten years. We would never have found those bugs without the Codenomicon tools.

If you're serious about implementing protocols correctly, you need the Codenomicon tools."

-- Jeremy Allison,
   Co Creator of Samba

Fuzzing Performance   pdf

Fuzzing Performance


Increasingly complex technologies and faster product release cycles create a need for faster and more effective testing. Testers need to run more tests in much shorter time frames. To achieve this, testing platforms need to be able to run faster sequential tests and to run more test cases in parallel.

"One of the most important aspects of fuzzing is how fast you can execute test cases", says Dr. Charlie Miller, principal analyst from Independent Security Evaluators. "The faster you can execute test cases, the more test cases you can run and the more vulnerabilities you will find."

Codenomicon Defensics is a flexible and scalable testing solution, which can generate and execute tens of thousands of dynamic test cases per second. Each one of these test cases consists of full communication transactions, ranging from simple protocol requests to complex message sequences. The test cases are model-based, enabling easy creation of new tests. The Defensics platform then allows the use of those test cases for all types of functional testing, such as conformance, performance and robustness testing.

With over 200 protocols supported off-the-shelf, and easy test creation frameworks, the Defensics testing platform provides unparallel capability extend your functional tests to any communication domain. As a software-based solution, Defensics testing platform is not restricted by the hardware constraints of any specific testing appliance. Moreover, it allows automation of the tests with the help of scripts.

Overview of our Case Study

At Codenomicon, we conducted a study to see how scalable the Defensics testing platform is. We used a number of hardware setups to test how many parallel test suites it can execute simultaneously and how many test cases all the suites can run together per second against single system under test (SUT) and a single network interface. In the study, we looked at different test scenarios and changed the test setups to see how resource constraints and bottlenecks, such as test target behavior and the amount of CPU, memory and network connections available, affect the performance of the used test configurations. The purpose of the study was to find the configuration for the maximum throughput and to help testers to achieve better test performance, when using fuzzing tools in load, stress and denial of service testing. In other words, improving test coverage by executing more test cases faster.

Download the full whitepaper from here: http://www.codenomicon.com/resources/whitepapers/2010-parallel-execution.shtml

Defensics for Performance

Defensics automates robustness testing and fuzzing. It utilizes protocol specifications and real traffic samples to create protocol models, which it then uses to generate test cases. The platform can be used to generate test cases containing both valid and invalid traffic, thus it can be used for all types of functional protocol testing, namely:

  1. Robustness and fuzz testing
  2. Load and performance
  3. Features and conformance

Because Defensics utilizes protocol models to generate attacks, it can be used to test both client-side and server-side implementations. Fully model-based test suites, which are based on protocol specifications, are extremely flexible testing tools, because the message sequences and the messages can easily be edited by the end-user.

Defensics provides model-based test suites over 200 protocols. All other protocols can be tested with the Codenomicon Traffic Capture Fuzzer, which can be used to replay and test any communication protocols captured by commercial and open source network analyzers. The Codenomicon Defensics testing platform is used by hundreds of companies and organizations around the world for various purposes ranging from fast prototyping projects to more complex conformance testing solutions, which test complex specifications with hundreds of carefully built use cases.

There are two factors that make Defensics a most suitable solution for high-speed robustness testing. Firstly, the Defensics platform is software-based, and secondly, it runs each test tool and user interface as a separate process. Thus, it can run multiple test suites simultaneously and its performance can be improved by simply increasing hardware resources. A third powerful feature is that the Defensics platform can create multiple execution threads from one test generation process, enabling more efficient use of hardware resources.

Performance Metrics for Fuzzing

To test how suitable the Defensics fuzzing platform is for high-speed robustness testing, we test how many test cases it can generate and execute per second. The load generated in performance tests is the result of two factors: the amount of sessions the test suite is running sequentially and in parallel.

Performance Metrics for Fuzzing

The key metrics for fuzzing performance testing are:

  • Amount of sequential test cases executed per second by a test suite
  • Number of test suites running in parallel
  • Amount of test cases executed per second by all parallel test suites

The speed of one single test case is impacted by the complexity of the message sequences it includes, and the complexity of all the protocol messages in the protocol transaction.