How to measure application lifecycle management?

Last year we’ve introduced Application Lifecycle Management in the company I work for. We already had a lot of best practices in place for the applications in our product portfolio. We have been using subversion as source control system and cruisecontrol.net as automated integration server, but we had no automated process for releasing a new version of an application and deploying it with the customer. The main goals of introducing ALM for our applications are:

• to deliver software of higher quality to our customers
• to automate the whole build & deploy process
• to standardize on 1 process for all of our applications

Once we have set this up for an application (I won’t go into detail now how we set this up) we would like to know if it truly delivers software that is of higher quality then we had without the automated process. We would also like to know if we can capture KPI’s or measurements that can indicate when we’re falling back in old habits that would lower the quality of a software release. These measurements should be able to be retrieved without additional manual actions, they should be measurements that can be extracted from the automated ALM process.

I’ve made a small mockup of how these measurements could look like on a dashboard. Each measurement has a value and 3 states (green, yellow, red) where green is ‘good’, yellow is ‘could be better’ and red is ‘not good’

I’ve included 8 measurements for a given product. The thresholds for the measurements haven’t been verified with the actual situation yet and will probably change.

NCover percentage

If the application makes use of unit tests. The coverage of how much code is tested is a good indicator of the quality of the application. I believe you don’t always need 100% coverage and you need to find the sweet spot between the risk that you take not testing code and the cost it takes to maintain the testing code. This sweet spot may be different for each part of an application. Threshold: 80+ is good

Support lifecycle policy compliancy

Our applications need to be compliant with newly released OS, database, Office and browser versions within a given period. This indicator tells us whether the application is compliant with the company policy or not. Threshold: 0 is good

Commits per day / team member

The more frequent someone commits a piece of code the easier it will be to merge that code and if a defect is detected the easier it will be to find it and correct it. If we assume that the defect will be detected before we release and we don’t commit that often then this indicator won’t say much about the quality itself, but will about the speed of development. Threshold: 2+ is good

Average duration of a failed build / month (or release cycle)


We won’t measure the amount of failed builds because that’s what it’s for, detect if we have code integration defects. But a failed build should be fixed as quick as possible. That’s why we’ll measure the average of the failed build durations for a release cycle / iteration. Threshold: less than 30 minutes is good

Software distributions to production emulation / month

Before an application can be deployed at a customer it has to be deployed first in our production emulation. This deployment process is equal to the deployment process in production. If something fails here, it will certainly fail in production. This is the last step where we can detect defects before we release to production. A deployment to the production emulation is like saying that you have a release worthy product. The best scenario is that you have an equal amount of distributions to your production emulation as to production. If you have done enough testing in the earlier stages of the process, the chances of defects should be very low in this step. Threshold: 1-3 is good

Software distributions to production / month

Our customers want new releases with a fixed frequency of about once or twice a month. Our aim at this moment is to release 1 version each month. This will allow the team to pack a good amount of new features in the release and to test it thoroughly without letting the customer wait too long for requested changes. Threshold: 1-2 is good

Amount of helpdesk calls / month / customer

Although this indicator only partly applies to the quality of the process, it is still very useful. By partly I mean that not only the ALM process will be guilty for helpdesk calls, but also how the features are implemented in the product. Threshold: less than 10 is good

The last informative measurement is the current status of the defined builds on the buildserver (trunk, development branches, nightly builds) of the application.

If we really want to detect if we’re building a qualitative application, we should measure a lot more, but these measurements should already remind us of the constant focus we need to have on building and deploying qualitative applications.

What other measurements can we make that will help govern the ALM process?