Stephan Dahl

Stephan Dahl, July 23., 2009

Software, being the most fluid medium of construction known, is constantly changing. And while the medium is fluid, the cost of change in terms of wasted effort and time is every bit as crippling to a software project as to any other construction effort.

This article deals with tools and techniques to manage and describe change, and control its impact on the project at hand.

Tracking Change

First and foremost, to control change we must be able to observe it - if we can't see and measure changes as they occur, we have little chance of keeping changes under control.

To realize this goal, it is necessary that all project artifacts are under revision control.

Keeping all project artifacts under control is both easier and harder than it sounds - easier, because every artifact in a software project is a file on a computer, and harder because many of those files are traditionally kept in many different repositories, some of which are transient in nature and many of which are not connected or synchronized with each other - emails for important decisions, USB sticks for test samples, multiple copies of spreadsheets (some attached to emails, some stored on shared drives) for plans. The list goes on.

In order to consider all artifacts under revision control, it is necessary to have a single repository for use within the project. The repository must meet a number of stringent requirements:

• It must be accessible to all project members at all times. For any but the smallest project, this means that the repository must be reachable from the internet. Of course, this in turn requires that the access controls to the repository are strictly enforced.
• It should have configurable access control lists - it should be possible to define read and write authorizations to different parts of the repository based on project member role. Some project artifacts (like bids, budgets, salaries, performance evaluations, etc) are confidential even within the project - and most project artifacts (like source, design and plans) should not be accessible to external contractors doing specific, limited work. Confidential information (where not even the librarian or systems administrator may read the files) can also be stord in a repository; For such hopefully rare artifacts, files can be encrypted prior to checkin.
• It should have easy reporting facilities - in particular, tools for showing the difference between versions of artifacts. This is more complicated than it sounds - "versions" can be specified both in terms of dates and labeled revisions, and "artifacts" can be specified in terms of files and configuration items. Typical questions that a proper repository can answer are "what changes occurred in version 1.8.7 of foo.c?" and "what changes occurred to any files in the requirements folder between january and august this year?"
  As an aside, to answer questions like this in a truly useful way, the artifacts themselves must be amenable to inspection by the repository - it is desirable to use file formats where the repository can extract a 2-line change in a 50-page document and present it in a meaningful way in such a report.
• The reporting facilities must be easy to use (at least for a trained librarian). It must be possible to define the queries ("questions", above) on the fly and expect immediate answers.
• The reporting facilities must be scriptable, to allow running canned queries at pre-defined times.
• Since more than one person will be working on the files in the repository, there will eventually be situations where more than one author needs to edit the same file at the same time. The repository must therefore have some means of conflict resolution - either a solid merge facility (which again requires that the repository understands all the file formats used) or an unobtrusive locking mechanism, where an author can reserve a particular artifact for editing. (footnote: "unobtrusive locking mechanism" is almost a contradiction in terms, especially when dealing with code with many small files and powerful refactoring tools. It is entirely possible that different areas of the repository should have different conflict resolution strategies).
• The repository must be immutable - once a file is checked in, it must never be altered, and it must be accessible through a well-defined label (eg, "revision 17 of file 'foo.c'"). It must of course be possible to undo bad changes - but this results in a new version of the artifact, which may happen to be identical to an earlier version of the same file. All versions of a file must be available, permanently; It should never be possible (or necessary) to "clean out old versions" to "save diskspace" or for any other reason. If the repository uses diskspace at a rate where saving diskpace becomes relevant to the cost of the project, then either the repository chosen is inadequate for the task, or the storage used is too expensive.
• It is highly desirable that common queries (like "latest version of", "version x of", "the difference between version x and version y of") are all expressible as Uniform Resource Locators ("URLs"), so that the result of such queries can be referenced directly as hyperlinks from other documents or as text messages.

Once the repository is in place, it is necessary that the project actually

uses

the repository - this goes for every project participant. To this end, fostering a culture of "if it isn't in the repository, then it doesn't exist" can be helpful. If everyone as a matter of course goes to the repository for documentation, then it has succeeded - but it only takes a few exemptions where critical documents are stored "elsewhere" for the utility of the repository to collapse.

Configuration Management

Now that all project artifacts are under revision control, we realize that there are different categories of files that we may like to group into logical units - these units are the Configuration Items of the project.

Configuration items share similar characteristics, such as

• General subject matter
• Author/reader groups - ie similar access controls
• Baselining schedules

A typical minimal set of Configuration Items are:

• Requirements - all the documents that constitute the formal requirements to the system, both functional and non-functional.
• Plan - all the documents that together documents that the project manager has allocated adequate resources for the project, such as staffing plans (adequate people), budget (adequate funds) and development schedule (adequate time). This configuration item will also include work progress documentation as the project proceeds, such as revised budgets, updated schedules and milestone plans, minutes of meetings documenting that the stakeholders have been adequately informed etc.
• Design - all the documents that demonstrate that the development team has understood the task at hand and has a clear idea of how to execute it. For larger projects, this item is typically split into Architecture and (Detailed) Design.
• Source - the actual product delivered by the project. The source of course includes all the build and deployment scripts needed to build the executable code, so there is no real need for a separate configuration item for "executables" or "gold masters" or whatever.
• Unmanaged - any files that are under revision control, but which do not need to be baselined. Typical examples are temporary work files that were accidentally included in the revision control system. The configuration item is present to allow the librarian to make a deliberate decision to exclude a file from configuration management.

A configuration item consists of a well-defined subset of the files that are stored the revision control system. Since all files in the revision control system must eventually be included in at least one configuration item, the configuration management tool must be able to report which files are not yet under configuration management, to serve as a to-do list for the librarian.

Once all files are under configuration management, it becomes possible to define Configuration Management Baselines. A Baseline is a key concept in managing change in general, since Baselines are the fixed points from which changes are measured.

A Baseline is defined in terms of a Configuration Item by specifying a particular revision of each of its constituent files. Typically, this is done by selecting the revision of all files in the configuration item as of a particular time, but it is entirely possible (and frequently desirable) to select some file versions based on time, and other files based on revision number - for example, if the baseline need to include a late-arriving but critical change.

A Baseline, once defined, is considered Open - while the constituent files (being under revision control) can't change, the Baseline itself can. However, once the Baseline is approved by all concerned parties, it is Locked, and then it is a true fixed point - a Locked Baseline can be used to represent final requirements, or an approved plan, or a reviewed design, or a shippable product. It is useful and desirable to be able to access the locked baseline as a complete package; usually this is done by a copy and export script.

Software projects being what they are, there are of course constant changes to the constituent files; Change control is never about preventing change, but always about observing change, and change management must never become a hindrance to the activities of the project participants. To this end, a configuration management system must be able to show the difference between the latest locked baseline and the current state of the documents under revision control. Likewise, it should be able to list the differences between any two baselines. As with the revision control reports, it should be possible to pose such queries ad hoc and expect a prompt answer.

Baselines should be labeled in a consistent manner; A three-level scheme with Major, Minor and Fix version increment counters usually works well for most purposes. The Major version is usually incremented at the end of a new project, when the first shippable version of the product is built. The Minor version is usually incremented when a new feature is included in a product, and Fix version is usually incremented when a critical bug is fixed. Lower counters are reset to zero when a higher counter is incremented. A product that has not yet shipped is usually assigned a major version of zero, with incrementing minor versions for each planned development timebox.

Note that labeled baselines as described can accommodate development branches. If a new release (eg, "2.0.0") of a product branches off the previous production version 1.4.13, then a bugfix (version 1.4.14) can be built and shipped and include files or file revisions that are not yet included in the current working set for release 2.0.0. Of course, the fix should eventually be merged back into the new version, as is usual.

A locked baseline can be reflected in the underlying revision control system, for example by tagging files with (Configuration Item)-(Baseline label). This makes it conveniently accessible to project participants (who usually view files through the revision control system) that a particular file is included in "Source-1-4-13". Note that an unlocked baseline (being mutable) is never reflected in the revision control system.

Baselining Procedure

With the configuration items in place, we can now specify a Baselining Procedure:

The project librarian frequently (at least weekly) reviews the list of unconfigured files, and (possibly with the help of the file creator) assigns them to a configuration item.

In preparation for Project check points, the librarian prepares a Baseline of the relevant configuration items and a Baseline Report for each.

There is typically a project checkpoint once the requirements (usecases, interface models, performance requirements etc) are gathered and approved by the client. For this checkpoint, the Initial Requirements Baseline is created and locked, and it is from this baseline that the inevitable project change requests are measured.

The completed systems architeture (architectural overview, system context, logical data model etc) is sometimes subject to external review. The review material is a natural baseline, as is the material after the review findings are incorporated. If such a review is included in the project, it is advisable to have separate configuration items for Architecture and Design.

The project plan (budget, allocations and schedule) is also baselined once the client approves the plan.

The design and source configuration items are usually baselined together. An open baseline is created for the next release of the product (this may be a timebox in a project, or it may be a fix release). File changes for the release are then updated in the baseline, and once all changes are complete and test approved, the baseline can be reviewed and locked.

Project changes are usually gathered into releases, and each release has its own set of checkpoints; At least one for the updated requirements and one for the updated design and source files.

It is usually the project librarian who performs the day-to-day work with assigning files to configuration items and file revisions to baselines. However, the critical work of approving and locking baselines are done by a formal Configuration Control Board, which consist of the product owner or client representative, the Project Manager, and the librarian. For some configuration items, the technical lead or IT architect must also be included.

The control board meetings have a fairly fixed agenda, and for minor fix releases they can be completed in a few minutes on a conference call. For large releases, more time is of course needed.

1. The librarian presents the baseline report(s) for the configuration item(s) being changed. These reports reflect the complete set of files that constitute the configuration item, and highlights the changes that have happened since the last locked baseline. In addition, the librarian confirms that there are currently no unmanaged files.
2. For an initial baseline, the project manager and client representative confirms that the baseline is complete and approved. The actual production and approval of the requirements and design documents involved in an initial requirements and design baseline would have occurred at a separate meeting, possibly even a multi-day workshop, involving many stakeholders - the control board meeting is a short formality by comparison.
3. For a change baseline, the librarian shows that all project changes scheduled to be included in the baseline are reflected in one or more changed files, and that all file changes are related to at least one of the approved project changes. For this purpose, it is desirable that the baseline report shows the commit comments for all the file changes included in the baseline. The project manager confirms that all project changes have passed test, and the client representative formally approves the new baseline.

The control board meetings may never be skipped; The approved minutes of a control board meeting are formal evidence of approval of changes.

Configuration Management Requirements

Observe that while a revision control system is necessary in order to perform configuration management, it is not sufficient - in particular, managing open baselines is not normally possible in a revision control system. A separate tool is normally needed to perform configuration management; This is not normally a problem, since the configuration management system user group is usually limited to the project librarian.

Since the number of files being managed in a typical project can easily run into the tens of thousands, some sort of user-friendly automation must be available to create and edit configuration items and baselines, select and assign files and revisions using configurable filters, and to generate the necessary reports.

Implicit in the above section is that the project has an assigned Librarian; This is not a full-time role for any except the largest projects, and would usually be assigned to a project secretary. For very small projects, the role is assigned to either the project manager or the lead developer.

Change Management

Building on established procedures for Configuration Management, we can now define procedures for Change Management.

Changes usually start with a Feature Request for a production application, a Project Change Request for a project in progress, or a Bug Report when a user or project participant encounters an error. For the purpose of Change Management, these are all equivalent; The processes to follow are always the same. The distinction in labeling is still useful, since prioritization and budgeting would differ between the different types of changes. For the remainder of this section, all the change types will be called Change Requests.

The first step when a Change Request is received is to perform a rough validation (is the bug real, is the feature requested relevant and possible?). Some Change Requests can be rejected early this way.

Once a Change Request is accepted as valid, its cost must be roughly estimated. Small bugs (eg, "typo in a screen label") can be estimated reasonably precisely, large changes (eg "application must be available in multiple languages") can only be guesstimated. If a large, imprecise guesstimate is provided, then a more precise estimate on how much a detailed analysis would cost must also be provided.

Once the Change Request is estimated, it must be assigned to a release. This is done by a Change Management Board, consisting of the product owner or client representative, in cooperation with a project manager. For a critical bug, a new release may be scheduled, small feature requests can be assigned a planned release where there is still space in the plan, and larger or low-priority changes can be assigned to "future release". During an active project, these assignments should happen on weekly meetings between the client representative and the project manager. The project manager should have authority to make these assignments on an immediate basis and just report his ations during the formal meeting, in case a critical problem arises. The formal meeting also reviews the funding for the change - bug fixes are usually funded by the project budget, changes in requirements are funded by the client, while feature requests from external groups may include funding for the work.

Change Request that has received approval for development are then assigned to the relevant project participants; Large changes can be subdivided and assigned to different people, and the subtasks may be marked with dependencies - eg, a task may be assigned to a business area expert for usecase updates, a lead developer for design work, two developers to write and review code, and a tester to write new testcases - naturally, some ordering between these tasks are in order.

If the estimate for the Change Request was large and uncertain, then it must be possible to approve work on a detailed estimate only (without naming a target release, but still assigning funding for the estimate preparation work); Once the detailed estimate ("bid") is prepared, the Change Management Board can assign the actual work to a release.

Once the project participants are working on a particular Change Request, they must be diligent in tagging their checked in work with the ID of the Change Request. This makes it possible to identify all changes across the project artifacts that are due to a particular change.

Once the changes are complete and the (new or changed) related tests all pass, the change may be included in a locked baseline as described above. The configuration mangement board can mark the work complete, and the Change Request submitter can mark the work accepted.

It must be possible to create project changes to include file changes already done by the project team on their own initiative (eg, code refactorings, document cleanups, requirements clarifications etc). Well-behaved team members create the change requests before they commit their changes to revision control, so they have a label to use. Such changes should generally pass without comment (and without funding) through the Change Management Board, under the assumption that there is a good reason for them, and that they can be contained within the already allocated time. Including and following this procedure prevents the Change Management processes from being a hindrance to the day-to-day work of the project.

Change Management for stable applications differ slightly from project Change Management - in particular, a stable application may not have a project group at hand to assign changes to. In these cases, approving the change and assigning funding also implies securing the resources needed to execute the work (both in terms of people and work environments).

Change Management Requirements

Change Requests are usually tracked through a separate system; There exist a large variety of change-tracking packages in the market, and most would (if properly configured) meet the requirements of the process described above.

It is desirable that the same system is used for all types of Change Requests, and that the same system is used during the project phase as well as during the subsequent stable lifetime of the product: This makes it possible to extract stability metrics in a structured manner throughout the lifetime of the application.