Software Configuration Management

Let’s face it, process-centric SCM is not just about the process, code and issue tracking, it’s also about document management. Simply storing documents in a shared network space is no longer sufficient; documents related to a product release need to be versioned and tracked. We’re talking about a wide range of documents here — UML diagrams, meeting notes, functional specifications, wireframe prototypes, product plans, slide decks, planning spreadsheets, and, of course, end-user documentation from release notes to complete online help systems.

AccuRev’s stream-based, process-centric approach makes it easy to manage your product-related documents right alongside your code and issues. Add a couple of folders to your workspace — call them planning, doc, or whatever you like. Add your planning documents to the folders, and promote everything. Now your product documents are part of the stream.

• When you create a child stream for the next release, those planning documents are included automatically because of AccuRev’s built-in inheritance. Just update the documents for the new release and promote. Product managers and others can create their own workspaces where they update those documents as needed.
• Want to track general-purpose planning documents that aren’t related to a specific release? Create your child stream under the root, and store all those documents there. Create child streams and workspaces based on document evolution, not product releases.
• Want to free your product managers and technical writers from using the AccuRev client interface? Now there’s the AccuBridge for Windows Explorer (for AccuRev V4.5.4 and newer), which provides AccuRev functionality from a right-click menu in Windows Explorer, and icon decorations to indicate AccuRev status.

Documents represent knowledge; they are part of the company’s history. They provide records of decisions and the context in which those decisions were made. They show the reasons behind the implementation decisions, the backtracking, the roads not taken, and the plans for the path ahead. When these important documents are not versioned and tracked, information is lost.

One of the most common and problematic challenges that exists in today’s software development environments is how best to support a Globally Distributed Development organization. In ye olden days, you had the entire team co-located in the proverbial cube farm under a single monolithic roof. If Brad wanted you to review the code he just wrote, he would literally turn around in his chair, ask you to come in and look over his shoulder.

Times have definitely changed. Now, your team might be headquartered in Boston, separate R&D sites in California and London, with some specialized groups in Bangalore and Shanghai. But that’s not necessarily the hard part. Where it gets complicated is when all these developers are trying to work on the same source code at the same time. Mastership issues questions, latency, mismatched process across sites; communication problems, lack of project visibility; these things all lead to significant decrease in productivity, not to mention the chaos for those trying to manage the effort.

Enter AccuRev. Uniquely architected to support remote and geographically distributed development (GDD), there are several key built-in capabilities that make the challenges of the past disappear. Consider the following graphic:

• AccuRev’s Stream Browser presents a dynamic visual representation of the software development process that is both fundamentally tied to the source code itself as well as being flexible and enforceable. At a single glance, *anyone* working on the project anywhere knows exactly what the process in place is. (See example AccuRev annotated screen shot here from the Alaska Airlines success story).
• Geography has zero impact on your position in the process; a developer in the UK can happily use a Workspace on the same parent stream as a developer in the United States. For low bandwidth locations, AccuReplica can provide LAN-quality access without introducing the traditional mastership and latency problems of other replication solutions.
• The private nature of the Workspace means that these remote developers can “share” code while still determining when to deliver their changes publicly.

Here’s the scenario: Developer ibergman works out of London, while jtalbott works in Boston. However, they are both part of a virtual team working on ComponentC. With AccuRev, the normal boundaries and limitations of time and space - not to mention being constrained by an inadequate SCM tool - no longer apply. Okay, I took some verbal liberties with the “time and space” bit, but it’s actually not too far from the truth.

In London at noon, ibergman wraps up a section of code she’s been working on and performs a Keep, which in AccuRev is a private check-in. The change is versioned yet stays within the confines of the Workspace, not yet ready for public consumption. But ibergman wants some validation, and asks jtalbott to review her code. Using instant messaging (IM), she pings him and catches him as he’s having his first sip of coffee, probably a colombian supremo. In other tools, how would someone review a private change that was just committed on the other side of the ocean? Would they even have private commits in the first place? In AccuRev, the moment ibergman performed that Keep in London, a visual identifier is available to anyone viewing the StreamBrowser, such as to jtalbott in Boston. So jtalbott clicks on the icon, and now has immediate access to operations like View, to see the file, and Diff, to compare against any previous revision of this file:

Did I mention that jtalbott’s access to these operations is instantaneous, as soon as ibergman performs the Keep? He can even take her version and send it into his own workspace if he finds it interesting enough to want to do additional development on.

So the previously mentioned problems of mastership, latency, visibility, communication, and most importantly Process, have all gone away. No more waiting on a 24-hour turnaround to get that Shanghai code copied into the branch. No more working in the dark not knowing exactly where your piece of code fits into the puzzle. Each team can regain the responsibility of merging in their efforts into common integration points, using a well-defined process implemented with streams.

It’s a remarkably simple and elegant solution to a complex and challenging problem. Of course, it’s still not going to solve the amusing problem of both the London and Boston developers feeling like they are superior to each other, but at least now they can actually review each other’s code real-time to help figure that one out

Read my earlier post: Fortune 500 Software Company Answers Your Questions on Using AccuRev

Up until fairly recently I had never worked anywhere that was able to do parallel development without it becoming a nightmare. Individual contributor or manager, we just seemed unable to find that secret magical incantation that would lead to a truly successful parallel development cycle.

Every time the team had a different plan, and the project’s champion would trumpet why this time ‘things would be different’. But in the end, we always failed in one of two areas (sometimes both):

• The quality of the software suffered dramatically

• We lost control of the schedule and we were forced to ship late

I actually worked on one project where the resulting release was so bad that a few weeks after release we had to pull the product off the shelves and work frantically to repair the damage.

Then a few years ago my team was looking for a new software configuration management tool to use. We started out wanting ClearCase, but after much deliberation, decided on Accurev. At first, we didn’t realize the power of the tool now in our control. But after a release or two, we started to realize what we had and adjusted our process accordingly and began to achieve very successful, high quality, and on time releases done in parallel with one another. We had found the holy grail of project management!

We started to understand not just what the best practices were with Accurev, but to understand what we were getting by following them. We made other changes to our processes that helped as well, but I give a lot of the credit to the tool-set for helping us in the right direction.

We started out performing a round-table series on our past projects, looking for common themes, and then choosing the top themes and seeing how we could use Accurev to solve them, and here are some of the things we found:

Problem: Often parallel projects require extensive code changes to the same files, leading to a need for developers to communicate what is being done to which files and when. Often this communication is not handled well.

Solution: Accurev’s work in progress and annotate commands, as well as the version browser allow developers to quickly view who else is making changes to the files that they care about, and what is being done to them. Now developers can pro-actively gather the information they need instead of depending on someone else remembering to tell them.

Problem: Often projects have late phase merge points which are both complicated and error prone. In addition since merging is generally done in the mainline, the entire release can be held when any two projects try to merge together.

Solution: By allowing stream definitions to be dynamic, Accurev allows developers to create common merge streams ad hoc at no cost, and to join the projects without effecting any other users. It also provides excellent trivial merge capabilities that save the users much of the error prone work, calling out only the non-obvious decision points which greatly simplifies the merge task.

Problem: Often bug fixes get made to one ongoing project, but developers forget to make the same fixes to other active lines of development.

Solution: Using change packages users can quickly make changes, move them to all appropriate locations, and quickly verify that the tasks were completed successfully. Using the stream inheritance feature minimizes the number of lines where a developer must make a fix for his coworkers to take advantage of it. This greatly lightens the load on developers and cuts down on the chance that something will be forgotten.

Problem: Ensuring 100% reproducibility of a build from pristine sources is a huge challenge. Changes to the build system environment, tools and code changes can all effect your ability to make a minor fix to released software.

Solution: By keeping your complete build system (environment, tools, sources) all in Accurev, and by using a snapshot you are able to guarantee 100% that what you try to build today is the same as what your built the day the snapshot was taken. Unlike a label, a snapshot is forever! This enables you to reduce the regression test matrix and focus on only the changes that developers make to the released software.

While I am sure that other SCM tools can offer similar results if you know where you are starting and know where you want to go. But what I marvel at is that using Accurev really lead us to the cup so that we could drink from it and enjoy the benefits of successful parallel development.

Have you seen the AccuRev 2-minute demo?

You’ve started using Continuous Integration (CI), but it’s not quite meeting your needs. You’ve started out simply enough, getting a build to happen on Linux or some other common environment. You’ve gotten comfortable enough to start using it in more and more places.

You hear the whispers of potential. Maybe they come from coworkers, maybe you hear them in your head, maybe you’re hearing the antipattern. Yeah he’s talking to you, and he’s very tempting. Go ahead, add more tests, he says. Automated documentation, yeah that will work. Run some performance and benchmarking. Run some reports it can take it he says. Worse yet, the antipattern is sitting in the cube next to you. The antipattern doesn’t like to check in his code until it is ‘just right’, and then it is a big chunk of messy merged globby spaghetti code I wouldn’t eat and your product shouldn’t either.

The antipattern is tempting. Even the “good book” of CI talks about being careful of the antipattern, and then goes on to say go ahead and add more to your CI process. Very very tempting.

Beware of the temptation! Don’t fall into the hands of the antipattern. It only results in disappointment. There are people out there just waiting to show that CI isn’t successful. They are waiting for the first real failed build, delayed delivery, or time spent working on a tool instead of the product to kill the usefulness.

I should know, because I’ve given into temptation. We had a good CI setup going, our build processes were running smoother, broken builds were being caught before they became critical. And then I added benchmarking. Worked great. We were getting additional reports on leaks and bottlenecks. And we had time to attack them and slowly make them less and less of a problem. We were being lean. But then I heard the whisper. Run reports it said. Build some doc it said. So I tried. I wanted automated reports, one less thing to do manually right? But I wanted them with every CI build. Now the builds were taking 1/2 hour to an hour. Not so bad. Until code started breaking. You see, at the same time that I was tempted so were others. And it was easy. Large chunks of code were being added, destabilizing builds. Tests were failing over and over until people stopped listening to them. There was no buffer between changes and the rest of the developers. Yeah, we were able to clean up the mess using AccuRev, but the CI damage had been done. Benchmarking fell by the wayside while I tried to get reporting done. And the build took days to stabilize even while we scraped the code clean of the problem areas. And this was when the antipattern was able to leverage our misstep. Leaks got into the code. Not enough to destabilze, but enough that product performance could suffer. Instead of catching it early we caught it late, and the cost was that much higher when we fixed the problems.

That was over 2 years ago, but I’m here to repent. I had forsaken the usefulness and message of CI. The temptation was to strong for me then, but I and our product have recovered. Non-critical automation is no longer a dream to be shared with CI, but as a post operation run more infrequently. Stability and Performance of our product our paramount and CI is helping us keep them in line. I have exorcised the antipattern and remain vigilant.

It’s Friday… so I dug up one of my all-time favorite web gems: The Unix Family Version Tree. Ever wonder when Unix started? Or the relationship between BSD and System V? Or how closely related Mac OSX is (or is not!) to Linux? Click the chart to see the full-size version.

Here are some notable shortcuts: SystemV, Linux, BSD, HPUX, Solaris, MacOSX, GNU/Hurd, Plan 9, Atari Unix, iPhone OS, Windows OS.

Even for those new to Unix (out on the leaves of the tree!), this time-line has a wealth of interesting information showing the history and relationships of most unix varietals. It will also help explain the usability challenges of switching between divergent OS’ from navigating file-systems to loading drivers to managing hardware.

While this reference isn’t exactly about AccuRev or its software development process automation per-se, it’s a great example of how divergent software can grow over time and how tracking changes between active mainline and previous releases becomes really tricky — unless you have good tools (like AccuRev!).

/happy friday/ - dave