FAQs (Frequently Asked Questions)
Codenomicon Test Tool General Questions
This section of the FAQ contains answers for general questions on Codenomicon test tools and the Codenomicon robustness testing methodology. Licensing, support and versioning issues are also covered.
Introduction
Q: What are Codenomicon test tools intended for?
Q: What is the meaning of the tests?
Q: Can I be sure that my product is bug-free?
Robustness testing
Q: What is robustness testing?
Q: In what phase during the development process should robustness testing be employed?
Q: What is functional / negative / PROTOS testing?
Q: Do I need to possess the source code for the tested system?
Q: Do Codenomicon test tools include all PROTOS tests?
Licensing/Pricing/Availability
Q: For what operating systems are Codenomicon test tools available?
Q: Do you offer an evaluation version?
Q: What is the difference between the evaluation version and the licensed version of a test tool?
Q: How I can obtain Codenomicon test tools?
Development/Upgrades
Q: How can I help develop Codenomicon test tools?
Q: Do I have to run all of the test cases again after a new test tool version has been released?
Support/Maintenance
Q: Are printed manuals available for the test tools?
Q: Can I contact Codenomicon by phone?
Q: Do you offer any training courses for using the test tool(s)?
Q: Can I send you PGP-encrypted and/or -signed e-mail?
Q: How can I get a test tool for another (similar) protocol interface?
Q: Where can I find more information about test tools and testing?
Q: How can I continue a test tool maintenance contract?
What are Codenomicon test tools intended for?
Codenomicon test tools are intended to discover security-related flaws in protocol implementations. They test the security and robustness of any implementations of the particular protocol for which the tool has been designed. This is done by sending a large amount of automatically generated, intentionally malformed protocol messages and message sequences to the tested implementation and observing how it reacts to these messages. The robustness of the implementation is measured by its capability to survive processing these malformed messages. Any crashes, denial-of-service conditions, slowdowns, and all other exceptional behaviour are clear indicators of security-related bugs in the implementation. Our tools are intended for weeding out these bugs proactively.
Robustness is the ability of software to tolerate exceptional input and stressful environment conditions. A piece of software which is not robust fails when facing such circumstances. A malicious intruder can easily take advantage of robustness shortcomings in order to compromise the system running the software. In fact, a large portion of information security vulnerabilities reported in the public are caused by robustness weaknesses.
Robustness problems can be exploited by causing denial-of-service conditions by feeding the vulnerable component with maliciously formatted input. Often, a buffer overflow type of robustness flaw can be exploited to run externally supplied code in the vulnerable component. Rooting out robustness problems will result in more reliable and fault-tolerant software. Robustness equals error-free operation, which leads to better customer acceptance and reduced support costs. Robust software is more secure.
It is important to note that Codenomicon test tools never test protocol conformance or other such functional aspects in any way. They are strictly intended for negative black-box software testing.
To summarize, our goal is to:
- Offer efficient, yet easy-to-use tools for vendors and buyers to verify the dependability of software components.
- Raise the baseline of software robustness beyond trivial vulnerabilities that continue to infest software.
- Work in a close, trustworthy relationship with software vendors so that any problems that are found will be eliminated as efficiently as possible.
What is the meaning of the tests?
The security assessment of a software component is based on a robustness analysis of the component. Robustness is the ability of software to tolerate exceptional input and stressful environment conditions. A piece of software which is not robust fails when faced with such circumstances. A malicious intruder can easily take advantage of robustness shortcomings and compromise the system running the software. In fact, a large portion of information security vulnerabilities reported to the public are caused by robustness weaknesses. All robustness problems can be exploited by causing a denial-of-service condition by feeding the vulnerable component with maliciously formatted input. Often, a buffer overflow robustness flaw can be exploited to run externally supplied code inside the vulnerable component.
In addition to increased information security, software robustness promotes software quality in general. A more robust piece of software is more stable. This increases user satisfaction and promotes higher uptime. Robustness analysis provides tools for assessing the quality of software as a complementary method with traditional process-based quality systems and code audits.
In light of the issues specified above, robustness testing tools can be used e.g. for the following purposes:
- Setting a baseline for new implementations of the protocol
- The test tool aims to set a baseline that eliminates a specific subset of errors that may infest a new implementation of the protocol. The test tool may be used early on in the development process for the elimination of the most trivial pitfalls.
- Acceptance testing
- When evaluating a commercial off-the-shelf product that implements the protocol in question, the results of running the test tool may be used as criteria for accepting the product.
- Product evaluation
- The test tool may be used as a supplementary metric for evaluating several competing products during the process of making a purchase decision.
- Regression testing
- If the test tool is used during the development process, it may be adopted as a part of the regression test material in order to ensure that the errors covered by the test tool do not re-emerge during code maintenance.
Can I be sure that my product is bug-free?
In short, you cannot. No software product, apart from only the shortest and most trivial programs, is effectively bug-free. However, resources can be directed towards making the software as bug-free as possible. There are several avenues available for achieving this. Firstly, training your developers and programmers on common programming errors cannot be stressed enough. An abundance of books, Internet resources and courses exist on secure programming. There really should be no excuse in modern software development for sloppy coding practices, lack of knowledge about programming security or the inability to avoid common programming errors.
Since fixing bugs after the software has been released will always prove to be much more expensive, as many bugs as possible should be caught early on during development. Even the best and the most-informed programmers make mistakes, and automatic tools should be used to get rid of the remaining bugs. Tools exist for such things as source code audits, memory allocation debugging etc.
Robustness testing has proven to be an invaluable method in reducing the amount of security-critical software vulnerabilities proactively and cost-effectively. For this reason, robustness testing is recommended as one of several complementary methods for improving software quality.
What is robustness testing?
The robustness testing method was originally devised in the PROTOS project, a joint applied research project between the University of Oulu and VTT Electronics[PROTOS]. Robustness testing is based on the systematic creation of a very large number of protocol messages (tens of thousands) containing exceptional elements that simulate malicious attacks. The method provides a proactive, low-cost way to assess software dependability and overall quality.
The security assessment of a software component is based on a robustness analysis of the component. Robustness is the ability of software to tolerate exceptional input and stressful environment conditions. A piece of software which is not robust fails when faced with such circumstances. A malicious intruder can easily take advantage of robustness shortcomings and compromise the system running the software. In fact, a large portion of information security vulnerabilities reported to the public are caused by robustness weaknesses. All robustness problems can be exploited by causing a denial-of-service condition by feeding the vulnerable component with maliciously formatted input. Often, a buffer overflow robustness flaw can be exploited to run externally supplied code inside the vulnerable component.
In addition to increased information security, software robustness promotes software quality in general. A more robust piece of software is more stable. This increases user satisfaction and promotes higher uptime. Robustness analysis provides tools for assessing the quality of software as a complementary method with traditional process-based quality systems and code audits.
Robustness weaknesses are introduced during programming (implementation) of the vulnerable software component. These kinds of errors easily slip through ordinary code auditing and testing since robustness problems generally do not manifest themselves during normal operations. They become visible only when someone or something presents the implementation with carefully constructed "malicious input", or corrupted data. The Codenomicon security assessment method is based on the systematic creation of a large amount of exceptional input data that is fed into the tested component. The number of input data units causing problems provides a quantitative figure about the robustness of the component as well as its quality and information security.
In what phase during the development process should robustness testing be employed?
Codenomicon test tools are ideally used early on during the development of a protocol implementation. This ensures that the implementation will be free of a majority of common pitfalls. Later on, robustness testing can be employed for regression testing. In addition to this, test tools can be used to find vulnerabilities in already-released products before any malicious third parties find them.
How is robustness testing related to other testing disciplines (white-box testing, conformance testing)?
Robustness testing is a black-box testing method intended for testing how well an implementation can withstand anomalous conditions and outside protocol-based threats.
First of all, let us look at black-box vs. white-box testing. In black-box testing no details of the implementation need to be available for the tester. The test tool only needs to be able to connect to the tested protocol interface. Specifically, no source code for the tested implementation is required for black-box testing. Although black-box testing is possible even without access to the tested implementation, usually some method of restarting the implementation after a crash is required. Nevertheless, black-box testing differs from white-box testing in the fact that black-box testing does not require access to source code, the capability to run a debugger on the tested software, or any information about the internal state or health of the implementation.
Second, let us look at how robustness testing differs from conformance testing. Whereas conformance testing attempts to determine whether an implementation behaves according to specification, robustness testing deliberately looks at how the implementation behaves when it is fed unexpected input. In conformance testing all test cases usually are normal protocol messages or sequences, for which a well-defined outcome can be predicted based upon the protocol specification. In robustness testing all test cases typically are protocol messages or message sequences that do not conform to the specification (packets that contain invalid fields, messages arriving out-of-order, etc.). A conformant implementation responds to well-formed inputs in the way specified in the specification, but a robust one is able to withstand (if not process) also any number of invalid inputs.
What is functional / negative / PROTOS testing?
Functional testing attempts to test software by observing its behavior. This is different to structural testing, which looks at how the system has been put together. Codenomicon test tools can never be used for structural testing, but in a sense they can be said to do some flavor of functional testing. In the strictest sense Codenomicon robustness tests are not "functional tests", since they do not test any particular features or functionalities, but rather the capability to withstand anomalous environment conditions and malicious input.
Negative testing is usually defined as reverse conformance testing, i.e. how an implementation behaves in the face of improper inputs. In a sense robustness testing is close to negative testing. However, since robustness testing is not simply conformance testing, Codenomicon tools cover a superset of those things that negative testing usually looks for.
PROTOS testing is used sometimes as a synonym for robustness testing. The PROTOS term comes from the Oulu University Secure Programming Group (OUSPG) research project with the same name. Codenomicon robustness testing has evolved from the original work made by PROTOS/OUSPG. Please refer to the FAQ entry on robustness testing for more details.
Do I need to possess the source code for the tested system?
In short, you don't. Codenomicon test tools use black-box testing methods. The only requirement for testing is the availability of the tested implementation. This makes the test tools usable for integrators and evaluators who want to assess the quality and robustness of acquired or evaluated systems.
Do Codenomicon test tools include all PROTOS tests?
Usually, if a test tool for a particular protocol is offered both by the PROTOS project and Codenomicon, the Codenomicon test tool will contain some kind of superset of the PROTOS tests. Codenomicon and PROTOS tools are both created with the Codenomicon Mini-Simulation Toolkit, which already ensures some similarity to the resulting test materials. However, Codenomicon tools are intended as commercial-grade testing applications, and as such they have been developed to cover additional functionalities and features that are missing from the more proof-of-concept PROTOS tools. However, PROTOS and Codenomicon usually concentrate on producing tools for different software protocols, so this may not even be an issue.
For what operating systems are Codenomicon test tools available?
The following platforms are officially supported:
- Microsoft Windows 2000 Professional with Service Pack 2
- Microsoft Windows XP Professional
- Fedora Core Linux
Most test tools should also work without problems on most platforms that have a suitable Java Runtime Environment available. Note that if the tool uses raw socket interface code (IP/Ethernet sockets) or similar platform-dependant custom code, it is only likely to work on officially supported platforms. We cannot provide any guarantees for platforms that are not officially supported.
Do you offer an evaluation version?
Certainly. Contact sales@codenomicon.com for details.
What is the difference between the evaluation version and the licensed version of a test tool?
An evaluation version of a test tool is intended strictly for evaluation purposes. This means that actual testing of different implementations and acting on the found bugs is not allowed. The customer can run the test tool, but not act on the findings. Additionally, a time limit is usually imposed on the evaluation version, after which the tool will cease to function. Sometimes the amount of available test cases may also be restricted in an evaluation version.
How I can obtain Codenomicon test tools?
Contact sales@codenomicon.com for licensing options and pricing. You can also will out our web page form at http://www.codenomicon.com/.
How can I help develop Codenomicon test tools?
We encourage and welcome feedback from all of our customers. We are especially interested to hear suggestions for improving our test materials. Moreover, any requests for new protocols or extensions to existing protocol test tools are welcomed with great interest. We like to develop high-quality products that will prove useful to our customers, and to this end we hold all customer feedback in exceptionally high regard.
Another usage aspect that we are especially interested in is the efficiency of our test materials. Although we understand and respect the fact that our customers rarely want to reveal the specifics of the bugs they have discovered with our tools, we would like to receive more input on which test groups and test cases have been proved useful, so that we can further bore in on the specific protocol messages, fields or structures that they exercise. This will aid in the development of further versions of our test tools. If a full listing of specific test cases or groups cannot be made available, we welcome even rough quantitative details on the amount of especially efficient test cases (e.g. a percentage figure). Naturally, no details on the tested implementation need to be made available. On the contrary, we hold our clients' confidentiality to the highest regard and will never publish any details under any circumstances about our customers' identities or discovered vulnerabilities.
Do I have to run all of the test cases again after a new test tool version has been released?
This is definitely recommended. Sometimes a new release version contains fundamental changes to the test materials, which means that a large number of test cases may have been added or modified. Due to this we always recommend that testers rerun all tests after the test materials have been changed, if possible. An alternative is to rerun only those test groups that have changed, but sometimes it may be difficult to deduce what groups have changed and how. All updated test tools should contain information on this in the Change history documentation section, but we cannot ensure that this will always be the case. Although a complete re-execution of all tests can be time-consuming, it provides an optimal result for test coverage and ensures that the tested implementation has been tortured as comprehensively as possible.
Why are there less test cases in certain test groups in a new version of the test tool? Why are there less test cases overall?
Sometimes when a new version of a test tool is published, test groups are edited to become more efficient. Test cases with new invalid contents can be added, but test cases that have been deemed redundant can also be removed. Usually these removed cases contain only trivial or no variations in the embedded anomalous data. Due to this pruning process, the amount of test cases can become reduced. Note that this does not mean that the test efficiency is diminished. On the contrary, we aim to improve input space coverage and overall test effectiveness in all test tool updates.
Are printed manuals available for the test tools?
The comprehensive HTML documentation that accompanies each test tool is currently the primary means of learning to use our software. The upside of using pre-packaged HTML is that it can be both viewed online and printed easily by using any standard web browser. Every user should read through all of the HTML documentation thoroughly and thoughtfully. It is easily accessed via the index.html page in the testtool/shared/info directory after test tool installation.
The main documentation structure consists of approx. only 15 pages, so it should be quite easy to print out. The individual test case visualizations and the test suite/group documentation should also be easy to print, but the total number of these pages is usually quite large (i.e. equal to or larger than the amount of test cases in the tool). Whenever a test case triggers a failure in the implementation, it might be a good idea to print out the visualization of that particular case and attach it to the bug report that gets sent to the developers (if such a mechanism is used in your particular testing process), or used otherwise to learn what kind of invalid data managed to crash the implementation.
Can I contact Codenomicon by phone?
Although a telephone contact is available, email is the preferred method for contacting Codenomicon in all issues relating to tech support, sales information and general correspondence. The use of email is preferred not only due to our geographical location, but also due to the fact that email allows us to handle customer contacts more efficiently, enabling faster response times and the maintenance of audit trails for all communications. Using email also provides for easier and more effective encryption capabilities, which many of our customers value highly.
Codenomicon contact information can be found at http://www.codenomicon.com. Codenomicon public PGP encryption key for secure email communications can be found at http://www.codenomicon.com/codenomicon-key.asc.
Do you offer any training courses for using the test tool(s)?
Yes, we do. We offer a comprehensive practical course for all of our test tool products. The course includes detailed information on test tool installation, usage and practical testing issues, including an introduction into the robustness testing methodology. We encourage all of our customers to receive at least a one-day training for best results. Contact sales@codenomicon.com for details and pricing.
Can I send you PGP-encrypted and/or -signed e-mail?
Yes, you definitely can. We respect our clients' confidential communications needs to the utmost, and we can accept most kinds of secure communications on request. Our PGP public key can be found at http://www.codenomicon.com/codenomicon-key.asc.
How can I get a test tool for another (similar) protocol interface?
This is usually possible in a reasonable timeframe. Contact sales@codenomicon.com for more details.
Where can I find more information about test tools and testing?
Your best resource regarding Codenomicon test tools is the Codenomicon website at http://www.codenomicon.com/. There you will find information on all of the currently-available test tools, along with general information about robustness testing and Codenomicon technology. Additionally, you can always contact us directly for more personal communications. See our contact information page at http://www.codenomicon.com/company/contact.html for details.
How can I continue a test tool maintenance contract?
Contact sales@codenomicon.com for details.
Follow us on:
