Source code analysis tools: Evaluation criteria


Source code analysis tools: Evaluation criteria

Support for the programming languages you use. Some companies support mobile devices, while others concentrate on enterprise languages like Java, .Net, C, C++ and even Cobol.

Good bug-finding performance, using a proof of concept assessment. Hint: Use an older build of code you had issues with and see how well the product catches bugs you had to find manually. Look for both thoroughness and accuracy. Fewer false positives means less manual work.

Internal knowledge bases that provide descriptions of vulnerabilities and remediation information. Test for easy access and cross-referencing to discovered findings.

Tight integration with your development platforms. Long-term, you’ll likely want developers to incorporate security analysis into their daily routines.

A robust finding-suppression mechanism to prevent false positives from reoccurring once you’ve verified them as a non-issue.

Ability to easily define additional rules so the tool can enforce internal coding policies.

A centralized reporting component if you have a large team of developers and managers who want access to findings, trending and overview reporting

Tagged : / / / / / / / / / / / / / / / / / /

The basic steps to adopt the continuous inspection pattern


The following outlines the basic steps to adopt the continuous inspection pattern:

1. Tool selection. Evaluate and select one or more tools that can perform static analysis on your code base. Evaluation criteria include: a. the programming and scripting language(s) used in your software projects versus the ones supported by the tool; b. whether the tool provides an API for developing customized verifications; c. integration with your IDE; d. integration with your continuous integration server; and a e. set of built-in verifications, which should include items that are deemed more relevant for your software project.

2. Verification selection. Most tools come with a variety of built-in verifications and metrics. You need to evaluate what built-in verifications in the selected tool(s) are applicable to your software project. Criteria may include: a. Modifiability requirements. If you are experiencing problems with maintenance and evolution of existing projects, you should enable verifications that pinpoint duplicated code, cyclic dependencies, overly complex and/or large classes and methods, and others that will help you to improve modularity and modifiability of the code. b. Security requirements. There are verifications that can spot security-related issues, such as SQL injection vulnerability, hard-coded credentials, and overridden security methods. Continuous Inspection – 7 c. Team skills. d. Quality of existing code base. It is likely that a more strict selection of verifications can be enabled for newer projects. For older code, a strict set of verifications may generate so many violations that they might tend to be ignored. In any case, it is desirable that the same selection of verifications is applied to all projects that are under the same governance system and quality scrutiny. e. Development is/is not outsourced. The development contractor and the contracting organizations may want to negotiate what types of violations will not be accepted in delivered code. f. Development team is/is not collocated. g. Greenfield development. An existing code base will often exhibit several violations and the cost to fix them may be too high. Greenfield development opens space to enable all verifications considered important and enforce them from the get-go—a zero-warning policy is more easily adopted in such scenarios. h. Feasibility to fix violations. Verifications that generate violations in the existing code that realistically will not be fixed in the foreseeable future due to lack of resources or other reasons should not be enabled.

3. Tool setup. The selected tools with the selected verifications enabled need to be configured on a continuous integration server. Ideally the tools should run as often as possible, say every 30 minutes. But in practice, if the code base is sizeable the static analysis can easily take tens of minutes. In that case, you may want to run the checks once every night.

4. Accountability for violations. It is important that developers are accountable for violations and that fixing them become part of the development process. As mentioned before, a zero-warning policy should be in place, at least for the most critical categories of violations. Checkpoints for fixing violations should be defined. For example, at the end of each sprint or at each code release, all violations should be fixed.

5. Customized verifications. The continuous inspection can go one step further and create customized verifications using the API available in the selected tools [Mer13]. This is a non-trivial step since it typically requires expressing an architecture decision or programming rule in syntactic terms of the target programming language. Albeit challenging to create, customized verifications represent an enormous enhancement to the different aspects of code health inspection. Built-in verifications are by-design generic. On the other hand, customized verifications can be specific to a software project or organization and hence more powerful. They can deal with module, layer, and method names that are specific to a project; they can accommodate known exceptions to a given rule; they are aware of home-made libraries and wrappers.

6. Enforcement at commit time. As an ultimate barrier to avoid code violations to enter the codebase, you can configure the automated verifications to be executed upon each source code commit operation [Mer13]. Of course, you need a versioning system that provides a mechanism to run user-defined scripts that can have access to the files in a code commit operation. This kind of solution will deny the code commit operation and give the user a clear error message pointing out the specific reason. Thus, even if a developer ignores the violations displayed on the IDE and on the continuous integration dashboard and reports, these Continuous Inspection – 8 violations won’t make it into the codebase because of the enforcement at commit time. Clearly, only violations that are critical and/or applicable to all source files in the code repository should be enabled to run on this solution.

Tagged : / / / / / / / / / / / / / /