Software Configuration Management

Managing software development is tough enough as it is. Managers take responsibility for the results their teams deliver. They’re on the hook for hitting schedules, delivering quality, and meeting customer requirements. Worst of all, if they’re saddled with slacker engineers, they still have to get the work done, and done right.

Tools are needed to help managers stay on top of their projects and find problems when there’s still time to fix them, before the schedule pressures explode. Project management tools are valuable, but their schedules only reflect the quality of the information that went into making them. Garbage in, garbage out.

The SCM system is the source of the information that drives smart decisions. Here are four sources of real data that can come out of any quality SCM system. I’ll illustrate with AccuRev because, well, because it’s the best.

1. Bug Arrival Rate
2. Churn rate
3. Task Estimate accuracy
4. Complexity Creep

1. Bug Arrival Rate

A key graph in any project manager’s arsenal compares bugs reported over time against bugs closed over time. Here we use AccuRev’s AccuWork issue tracking tool to store all issues, whether reported by QA, sent in from customer service’s CRM, or entered by the powers that be as customer requirements. Since our iterations are four weeks long, we use the week as a reporting time period. Some AccuRev customers use days.

In AccuWork, queries are easy to create, and we get a report from AccuWork in XML covering all the issues opened or verified [foot note: In our terminology, developers “close” issues, and QA “verifies” them. Other companies have developers “submit” issues and QA “close” them] against a particular iteration.

The XML slides right into Microsoft’s excellent Excel 2003. I use PivotTables and Excel graphs for all my reports. Others might leverage their superior Perl skills.

Here, we add a “week” field based on the AccuRev time field.

2.Churn Rate

A key metric for quality is how often different parts of the code base have to be modified to address issues. Here, the AccuRev code repository has all the information we need. I use the AccuRev Hist command

accurev hist –a –s “release candidate stream” –fx –t “2006/1/8 – 2006/1/1”

to get an XML file of all the “Promotes” (sort of like “check-ins”) to a release candidate stream within a time range. Some use a separate “hist” command for each release period. Personally, I take a big time range and then use Microsoft’s Excel 2003’s XML file support to slice and dice into tasty chucks. I tried it once with Excel 2007. Once.

Since we component-ize our code base based on directories, the “path” field is particularly interesting, as directories can be as significant as individual files.

Loading the XML file into Excel, my macro deletes some unneeded columns and creates a Pivot Table like the following:

Here I’ve filtered the data to not include the binary files, and used the Excel Pivot table’s advanced feature to show the top ten in descending order. AccuRev.c takes a beating every release but it isn’t significant. Server/diff.c and Client/stat.c seem to be at the center of a lot of changes that release, and are probably candidates for some code reviews.

3. Task Estimate Accuracy

Along with dates of issue opens and issue closes, we have chosen to include a user field for the original estimate of the length of time it will take to complete the issue. That gives us something to compare to when we “close” an issue and send it to QA, and an opportunity to correlate against the length of time before an issue gets verified.

As a best-practice, we try to normalize issues to a couple of days in length. Longer and you haven’t really figured out what needs to be done. Shorter and you’re liable to strangle in your own spit, a management practice I can’t recommend having seen the results.

We get this data out of AccuWork again, and can slice it by group, release, or even individual developer. This is a quick way to quantify the subjective feel we all have for who is slow, who’s fast, and who’s clueless. Here’s a scatter plot for a group. Note that they are generally on the high side of their estimates, and that they haven’t done a very good job normalizing tasks to 2-3 days:

4. Complexity Creep

As software gets modified, and especially when managers become conscious of the amount of time taken to fix a bug or implement a feature, there is a tendency for developers to make the code base less maintainable. There are some metrics for code complexity that can benefit the manager when trying to assess when it is time to refactor. A frequently used metric is cyclomatic code complexity.

While we don’t do any code complexity measurements here, some of our larger customers do. This is easy to collect using a trigger on changes being sent to a release candidate (or QA candidate) stream, where the trigger calls the analysis tool and puts the results into a tab-separated file for later plotting from Excel.

A simple (Perl) implementation of the server side “server_post_promote” trigger creating a file “complexity.txt” looks like:

open COMPOUT, “>>complexity.txt” or die “Can’t open complexity.txt”;
if ($stream eq “acmeProj_QA”) {
$complexity = complexity($stream, $version, $workspaceDir, $filePath);
print COMPOUT “$user\t $date\t $transaction\t $stream\t $version\t $filePath\t $complexity\t $issueNum\n”;
}
close COMPOUT;

Together, these reports provide a good mix of data around schedule, estimation and code quality, giving our beleaguered project manager a pretty nice set of armor.

Please post your favorite management reports, even if you don’t (yet!) use AccuRev.