Fuzzing Windows Applications and Network Protocols

Fuzzing is a technique for detecting software flaws by intentionally sending invalid input to a target of evaluation, generally involving a high degree of automation. For software engineers it is crucial to identify and eliminate such flaws since they might be exploitable by a remote attacker. As a consequence an attacker could compromise the application as well as the operating system the software is running on, gain unauthorized access and steal or modify confidential data. Fuzzing provides an instrument to find bugs fast with relatively low costs. While the basic idea behind fuzzing is simple, creating thorough, precise and performant fuzzers is a real challenge. With time and computing power being limiting factors, the success of fuzzing depends on the level of detail when modeling the protocol as well as the effectiveness of the data mutations performed. The goal of this paper is to present methods to effectively fuzz a network (i.e. server) application using an example protocol and a custom fuzzer.
By choosing the standards based Extensible Messaging and Presence Protocol (XMPP) we not only selected a contemporary protocol that has a gaining popularity in real-time communication applications but that also incorporates many advanced concepts found in other common networking protocols. The scope of the XMPP protocol including all its extensions is vast and therefore presents a major challenge in terms of protocol modeling and target coverage.
As our tool of choice we selected the Peach Fuzzing Platform for fuzzer modeling. Peach is a free, powerful framework including many automation features. Partly due to the documentation, which in some areas is very scarce, Peach has quite a steep learning curve. By documenting our experience, successes and mistakes we aim to reduce the initial effort required to become acquainted with Peach and its varied features. Further- more, by following an incremental approach to build-up expertise moving from simple to more challenging problems and techniques proved a viable project guideline.
In addition to general fuzzing considerations such as the circumstances under which fuzzing makes sense and the limitations of fuzzing we also present methods to approach fuzzing extensive XML-based protocols and tuning Peach to achieve higher performance and point out strategies to a high level of coverage and precision.