Being able to measure success for a software development group is a difficult thing. But not being able to show the success of your development group is a dangerous thing. Your management team will want to be able to measure quality, show improvement, and benchmark against industry standards. All the popular maturity models (if you put stock into them) emphasize the ability to measure, react, and improve your quality and processes. My department is no different. We try to remain as flexible and lightweight as possible, and add formality where improvement is necessary. Here are some of the metrics that we collect to measure our success and find ares of improvement.

Code Coverage

There will always be bugs. No matter how hard you try, there will always be bugs in your software. That does not mean that you cannot try to prevent them. One of the easiest ways to do that is to write automated unit tests to validate your code is doing what is expected. You can write your unit tests lots of different ways, and Steve Cornett’s Code Coverage Analysis paper gives lots of different ways to break down code coverage. A great place to start is to aim for 50% coverage of all of your lines of code. And, as Tony Obermeit says in his Code Coverage Lessons paper, your ultimate goal of course is to always hit 100% coverage. You will need to pick a code coverage tool to help measure your success. In my department, developing in a Visual Studio and C# environment lends itself to the nCover open source solution. This solution works well with our CruiseControl environment. Test Driven Development methodologies and mocking tools can help you get closer and closer to covering as much of your code as possible with automated tests.

Defect Tracking

I use the words defects and bugs interchangeably. This I am sure is something that some people would disagree with, but I think that it is close enough. If it is not working as expected, then it is a defect, and it is a bug. Regardless, defects are identified in the development process, system and user acceptance testing process, and in the production environment. The objective is to minimize the number of bugs that are found in the live environments. To do that, you need to encourage the identification and mitigation of bugs in earlier and earlier stages. This sounds fundamental, but becomes difficult to implement. There are lots of methods that you can do to identify, solve, remove, and prevent bugs. You must have a way to measure that these methods are improving your success rate. And that means measuring the number of defects found in each environment – development, system test, user acceptance test, and production. The easiest set of numbers to get in a corporate environment is production defects. There is always a problem management system or help desk that tracks these kinds of things. But as a software development organization, you need to implement bug tracking throughout the entire lifecycle of your software. You can trend the numbers, and make sure you are finding more bugs before UAT, particularly in development. Tools like Bugzilla, an open source bug tracking tool, can help you track, trend, and manage defects in your software throughout its lifecycle.

Support Ticket Management

Software is not a static entity. It is always changing. Just think of all the patches, updates, service packs, and bug fixes that Microsoft releases on its suite of software. In a corporate environment, it is no different. Software management does not end once it is released. Teams of developers will be constantly updating desktop, web based, and console applications based on new requirements and requests from their clients. Problem Management software can be used to help track and trend all of these requests bydata points such as severity (Urgent, High, Medium, Low), priority (Urgent, High, Medium, Low), assessment (Customer Assistance, Required Modification, Elective Enhancement, Break Fix), difficulty (High, Medium, Low), etc. You can measure success agains more complex metrics such as the number of tickets created, number of open tickets, time to resolution, etc. All of these metrics will help you determine how fluid, stable, usable, sustainable, and maintainable your software is. Do not ignore your software or its users once it is released to production.

Analytics

Web Analytics tools can tell you how many users you have had on your site, how long they visited, where they came from, where they went, how they found your site, did they reach your goal pages, did they convert, and did they return. There are free web based tools like Google Analytics, and over the shelf packages like WebTrends and CoreMetrics that can help you measure site activity. Do not ignore these metrics to help you define current activity, make improvements to your site, track your new results, and continue to improve. They directly measure your clients’ interaction with yoru software, and can identify trends that with simple changes can vastly improve your software and development processes.

Conclusion

So… these are some of the ways that we track the success of our software. There are a host of other methods to measure software, such as function points, lines of code, complexity, interfaces, velocity, etc. What ways do you measure your software? how do you define success? What are your plans for the future?

Leave me a comment and let me know what you think.