Friday, 10 June 2011

Requirements for Security Code Reviewers


A few days ago I read a statement of someone mentioning the lack of knowledge and experience of application security reviewers or consultants. According to the author application security testers should have a proven track record in security research and published security advisories.

I do not agree at all with that author. I believe the above to be an example of groupthink and looking at the statement I have no problem placing the author in some academic group. No problem with that, but are these really requirements for application security reviewers?

Maybe I was guilty of the same prejudice or groupthink when a few years ago my company Astyran was asked to create a short document to enable a client to hire the correct people as secure code reviewers.

Note that this was a client in the financial industry with very high security and compliancy demands, including the need to review applications for design errors and malicious code planted by the developers.

I present that document as is in the next few pages. Some details are changed to protect the privacy of the client.

Let me know what you think are the real desired capabilities for secure code reviewers. Now shoot!



This document presents our view on the desired capabilities for source code reviewers and the selection criteria to detect and rank those capabilities in a formal Request for Quotation.

Based on our experience, we feel that the following table enables making a pragmatic choice (or ranking) between source code reviewers, based on the selection criteria certifications, experience and project experience.

Criteria Weight Requirement Rationale


A generic security related certificate, not a secure programming certificate.

Reviewing for security vulnerabilities is far broader than just program language related.




Formal education at bachelors or masters level in IT

Application security is more than just source code, but also concerns IT management issues as well as technical topics on server, network, and database level as well as software engineering principles.

Security Field


Number of years in the generic security field See above

Application Penetration Tester


Years of experience as a application level penetration tester Knowing how to break an application is important and implies knowledge of a wide range of exploitable vulnerabilities.

Presentation, Publications


Related to source code review or application security. This acts as an indicator of subject matter expertise
Project Experience      


Size of review or development projects participated in Exposure to code written by other people becomes natural


Nature of review or development projects Experience with large diversity in types of projects helps adapting to the logic of the applications to review


Experience in financial industry Secondary order: critical applications are not solely related to finance.

Please refer to the remainder of this post for more information on reason and background for the above recommended selection criteria.



More and more companies and governments realize that taking care of network and host security is not enough to protect vital or critical information flows enabling business or government processes. Criminals in the field exploit on a daily basis application level vulnerabilities and weaknesses that once where discussed only in small groups of practitioners or university research groups.

Regulatory bodies (such as the Monetary Authority of Singapore) and industry groups (such as the Payment Card Industry) have noticed this trend and impose the requirement to perform a source code review on critical or internet facing applications.


Objectives of a Code Review

Apart from the compliancy demands, common goals or requirements of a secure code review include:

  • To discover all vulnerabilities;
  • To review the application architecture and design for security flaws;
  • To review authentication, authorization or non-repudiation related controls;
  • To review cryptographic controls;
  • To detect potential malicious code;
  • To detect deviation from secure development practices;
  • To ascertain that users cannot bypass security checks when exceptions happen.

Manual or Automated Code Review

Although the requirement for source code review is documented, most regulatory bodies or industry groups are very light on details on exactly how this code review must be done.

A code review can be performed completely automated (using static or dynamic code analysers), completely manually or anything in between.

An overview of the differences between the approaches and the pro’s and con’s thereof is out of the scope of this document, but since this document is about requirements for code reviewers (not for the tools they might use), it might be useful to remind the reader why exactly one needs a code reviewer and why using just a tool might not be sufficient.

On the other hand, tools do have certain advantages and might be the best choice under certain circumstances.


Strengths and Weaknesses of Tools versus Humans

Studies show that humans are fallible at a much greater percentage over automation at redundant, boring and confusing tasks. The same holds for code reviewers: they are prone to skipping boring, uninteresting tasks as well as those they deem unnecessary because of their own experience.

The above means that automated tests of the same type would not be skipped. Some kind of automation is a necessity for a comprehensive test. Tools are very good at deriving results from a specific metric and the delivery of results in any form necessary (e.g. graphs, different file formats...).

Tools are however not good at interpreting or incorporating automatically generated results into better decisions. Seasoned code reviewers use automation to ensure consistency, transparency, and scale to the process of aiding managers in their efforts to make better decisions.

Recent research (2008) by the National Institute of Standards and Technology in the United States (NIST) concluded that:

  • Tools can capture:
    • Standard errors such as Cross-Site-Scripting (XSS) or SQL and command injection;
    • Certain (but not all) classes of authentication errors and information disclosure
  • Humans are required for
    • Authorization problems
    • Logical flaws
    • Complicated weaknesses

Based on our experience we might add that tools also have difficulties with newer frameworks and security issues that arise from the interaction of different client- or server side components.

Tools do not know anything of the context of the application and need human help to better find and classify certain issues: tools do not know what information should be protected, what must be logged or how a developer planted a backdoor. Not all security vulnerabilities are software defects.

Source code review can be automated up to a point, for finding "low hanging fruit", but currently even the best automated tools barely scratch the surface where it comes to finding security issues in source code.

Typical mistakes, such as using unfiltered user input for building SQL statements (SQL Injection) can be detected by automated scanners. However, when SQL statement templates are read from an XML configuration file, things become more complicated and the auditor will not only have to evaluate the source code but also the contents of the XML configuration file.

Note that for any modern web application authentication, authorization and logging are usually centrally configured. Forgetting to take the configuration into account (often skipped by static analysers) might be a very costly mistake to make.

Things become even more complicated when evaluating proper usage of cryptography: all too often secure algorithms are used in insecure ways. E.g. a 32-bit session key can be encrypted using a 1024 bit RSA key, but in the end, the effective session key would still be only 32-bits, which is considered hopelessly insufficient for modern hardware.


Outcome of a Code Review

The chosen review method (manual, automated or a combination) should be based on sound application risk assessment (internal or external users, direct influence on bottom line, hosted or in-house...) and business priorities and goals. Depending on the outcome of this process different choices or depths of reviews might be in order.

But also important to take into account is what will happen with the results.

The results might be classified and prioritized and put in a formal Change Request to the development team or service provider. Note that any Change Request might cost time and money, so care must be taken to only make security and business relevant requests.

Or maybe the result of the review will not be used to modify or change the application (e.g. because it is a legacy application) but to research what is wrong with it and how to adequately protect it until replacement.

Another objective might be to use results for different applications as a comparison between outsourced development providers: be cost-effective by choosing the best fit provider based on price in combination with the ability to write secure code.

All the above might have an influence on the review method.


Communication is Key

A code reviewer must be able to put the results of tools into context and talk at security, IT, audit and security management. But at the same time, the reviewer must be able to talk to developers and development teams about why a change is necessary. Many times we found developers referring to security and audit as a group that impedes progress.

This asks for strong communication skills as well as a deep understanding how development teams and QA teams work. Any requests for remediation might have a team miss their release deadline, resulting in penalization (e.g. missed bonuses) or getting negative comments on their performance review.


Requirements for Code Reviewers

From the earlier examples a number of requirements for good source code reviewers can be derived, as requested based on certifications, experience and project experience.



Certifications of secure programming knowledge are generally of little value for evaluating the capabilities of a code reviewer. The certifications are obtained after answering a sufficient number of questions correctly during one test.

What the certifications prove is that the test person knows which issues are important for secure coding and, possibly, recognizes those issues in a staged test environment. Using the obligatory car analogy, this can be compared to knowing which pedal in a car serves which purpose, but it does not guarantee that the person is capable of winning Formula 1 Grand Prix races or is capable of driving a car through traffic without accidents.

We would recommend checking for a security related certification and not for a specific secure programming certification.



Foremost is a thorough knowledge of application security as most serious security risks are programming language independent. Languages are in fact just a tool to implement a technique. A good code reviewer will have several years of experience in the application security domain: (web) application penetration tests, war games, and analysis of published security vulnerabilities...

In order to be able to process large quantities of source code in a short time frame, a code reviewer will have many years of experience in hands-on programming. That experience allows understanding the workings of source code simply by browsing the code, but also helps in detecting unusual code constructs. The definition of "unusual" cannot be given in a precise way, but comes only after years of experience. In practice, wherever an "unusual" piece of code is found, there is a high risk that the code implements a flawed logic (such as backdoors or plain logical errors that could be exploited)

Extensive experience with multiple programming languages provides a solid foundation allowing a code reviewer to learn enough of a new language in a short time for performing a successful code audit. In practice, many large scale projects use a mixture of different languages (Java, JavaScript, PHP, SQL ...) Checklists for language specific issues are readily available on the Internet.

Application security encompasses a lot more than just the code. Formal education in Information Technology might proof that the code reviewer is able to take networking, server, database and good programming practices and software engineering principles into account.


Project Experience

Important questions regarding project experience in source code review projects are:

  • Size of projects participated in (in large projects, exposure to code written by other people is natural, which helps reviewing code written by other developers)
  • Nature of projects (experience with large diversity in types of projects helps adapting to the logic of the applications to review)

Experience in the financial industry might be of secondary importance: any typical bank or insurance company has multiple critical applications that are not necessarily related to finance. Software in other fields often has to meet strict criteria (control of chemical reactors, control of assembly lines ...) and developers with experience in such environments are used to handle hard requirements.

No comments:

Post a Comment