I recently consulted with a company that had a requirement to version control their test results because generating the results took
hours and, in some cases, days to produce. Having the ability to compare results between specific source configurations over time was critical. However, the engineers found it cumbersome (and ultimately unnecessary) to include the vast amounts of test output within the actual release configuration.

Lets consider a typical development scenario that includes both nightly unit testing and scheduled system testing. This first picture is a screenshot of an AccuRev stream structure showing the streams involved in the workflow. Nightly testing of new development is captured by creating snapshots from Integration. Each snapshot then serves as a new root of development for committing the subsequent test results. A Nightly_Testing stream is created from the snapshot to store the test results. After promoting the test results, a new Nightly_Results snapshot is created to capture both the source code and test results. A similar pattern is (re)used for managing test results from SystemTest.

This second picture is the same picture but annotated to describe how to version control the transient test results. One way to absolutely prevent test results from being promoted higher up is to set incl/excl rules for the results directory. In this example, the SystemTest stream sets an incldo \.\test\results to exclude any relative files or subdirectories for the entire stream hierarchy (due to rule inheritance). However, each NightlyTesting stream sets incl \.\test\results so the results can be stored locally. Setting these incl/excl rules is an easy way to control the visibility and location of code throughout the stream structure.

Storing transient test results in isolated sub-stream hierarchies provides the ability to do stream-diff comparisons of test results. Furthermore, the golden Release snapshots are not cluttered with interstitial test results. In fact, if you need to backtrace the test results, you have them… afterall, they are now safely organized in CM.

/happy testing/ – dave