Category Archives: Software

400 Level Testing – 20 Advanced Topics

Now that we have walked through the basics in my posts on Testing 101, 200 Level, and 300 Level testing, the number of bugs in your production environment should be falling. To continue to eliminate the nasty bugs, it is time to dig deep, get creative, and use skills from other fields. Here are some out-of-the-box ideas to continue the momentum and focus on quality and stability of your software products.

  1. Shift Left – this is the general principle to find bugs as soon as possible in the software development process – write automation tests to find bugs in QA, write unit tests to find bugs in development, use TDD to find bugs before you write code, architect testability into your platform before you even create the solution.  TechBeacon has a great article about Shift Left.
  2. Crowdsource Testing – sometimes you need to get a fresh, new perspective on testing.  Using the general public can help you get there.  I have used a  company called uTest, and they were great.  They have all kinds of purchase options.  There are lots of other crowdsource testing companies to choose from.
  3. Fuzzing – I have done this before and never knew it was called fuzzing.  Build automation tests that randomly generate characters (ISO-8859-1 and UTF-8) and submit forms, to test positive and negative cases you never thought of.  Run it thousands of times, and you are bound to find some errors.  Read F-Secure to learn more about fuzzing.
  4. Innovation – When most people think of innovation, they think of a bolt of lightning, striking randomly. While this may be true some of the time, that isn’t always the case. There are methods to developing and trying new ideas, and that kind of innovation needs a culture to support it. This great article from The Medium outlines some of the methodologies to use to find innovation.
  5. Design Thinking – Over the last few years, I have been working with Rutgers on their Advisory Board for their Design Thinking certificate program. I stay on top of the subject, and tweet a lot of great articles to read on the topic. When I came across this article on DT and Software Quality, I knew I needed to share it.
  6. Bug Bash / Bug Hunt / Bug Day – Make finding bugs a social event. Invite your team, or other teams, to find bugs in your product. Set aside an afternoon, order some pizza, plan some prizes, and find some bugs! Read more about how to run a Bug Hunt.
  7. Quality Hackathon – another way to generate ideas about new approaches to testing and quality, you could hold a hackathon, one that focuses specifically on testing. Are there new ways to automate? New tools to try? New approaches to testing? Challenge your team, set some parameters, pick some judges, give away some prizes, and make it fun!
  8. Apply Gamification to Testing – In World of Warcraft, there are monsters to kill, skills to learn, reputation to gain, equipment to discover.  You can do the same thing with Quality and Gamification – each person on the team has a QE Score, and gets points for bugs found, classes taken, Tech Talks conducted, junior teammates mentored.   Define a reward system, how to grow, and publicly praise the growth – different levels, badges, and achievements to earn and the steps to get there.  Read more about gamification on
  9. Machine Learning and Testing – Can we teach a machine to find bugs in software that humans write? Or identify what sections of our software systems will most likely have the most bugs? Or what sections of code are the most complex? Maybe we should give it a shot
  10. Measure your Test Maturity – According to CIO Magazine, “The Capability Maturity Model Integration (CMMI) is a process and behavioral model that helps organizations streamline process improvement and encourage productive, efficient behaviors that decrease risks in software, product and service development.” Initially developed by the Software Engineering Institute at Carnegie Mellon University, in conjunction with the DoD and U.S. Government, CMMI is currently administered by the CMMI Institute as a process improvement tool for projects, divisions or organizations. CMM defines 5 levels of maturity – (1) Initial, (2) Repeatable, (3) Defined, (4) Managed, and (5) Optimizing. If thought of in a different way, this maturity model could be specifically applied to software quality and testing, and used to measure current state and set goals for improvement.
  11. Think Different – While this may be the slogan for a famous ad campaign for a large tech company named after Newton’s fruit, it is also a technique to help people solve problems by taking a different mental approach – Creative, Analytical, Critical. Concrete, Abstract, Divergent, Convergent, Sequential, and Holistic thinking.  Read more about software testing and different thinking types on
  12. Complexity = Errors – Exploit the correlation between complexity and errors. A rule of thumb is that one component should do one thing, and do it well. Make complex classes smaller. Break monolithic services into micro services. Break database tables into multiple, smaller tables. The same is also true for tests. Make unit tests smaller. Test smaller sections of the user journey. And aim your spotlight of testing onto complex areas of your application – UI, services, or data, integration points, service signatures, common failure points.
  13. Wear different Thinking Caps – It’s a common enough expression… “put on your thinking cap.” The 1985 book Six Thinking Hats by Edward de Bono promotes multi-dimensional thinking by mapping different colored hats to different cognitive styles – blue for an overview, white for facts, red for emotion, yellow for optimism, black for risks, green for creativity. Put on different thinking caps, and innovate in whole new ways.
  14. World Quality Report – “The World Quality Report is the only global report analyzing software testing and quality engineering trends. It presents an analysis of developments in agile and DevOps, artificial intelligence, automation, test environments, data, security and budgets, showing once again the importance of quality, and of the measures that are put in place to maintain it.” – sogeti (part of capgemini)
  15. Online Resources – Things change, evolve, improve. It is our obligation as technologists to stay on top of those changes. There are a lot of resources that can help you do that. Training, conferences, certifications, and industry trends are some ways to stay on top of those changes. Sometimes it is just staying on tho of the news as it happens. While doing research for my blog posts, some of the best resources for me were Ministry of Testing, Software Testing Help, StickyMinds, InfoWorld, and Software Testing Magazine.
  16. Training – Continuing your education as a technologist and quality engineer is critical. There are lots of online video training resources available online. Pick your favorite and keep learning. Two of my favorite are Udemy (and their class on Software Testing and Innovation): and Lynda (and their class on Exploratory Testing).
  17. Conferences – Learn about new techniques, hear from industry experts, plan for future innovation, look at products from vendors, continue your education. Check out a list of conferences from TechBeacon and .
  18. Certifications – Certifications are more than just a test and a score. They are managed by a company with resources, continuing education, advanced certifications, and a community of professionals you can tap into. Check out a list of certifications for Software Testing professionals.
  19. Industry Trends – Remain on top of industry trends allows you to adapt to current challenges, leverage new tools and techniques, and benefit your skills and your company. Read more about quality assurance trends in 2019.
  20. Beyond The Bugs – What I have focused on in these last four posts is how to find bugs in your code. Ones that would functionally prevent users from using your site. But there are lots of non-functional ways to test your site. Security testing, penetration testing, performance testing, reliability testing, efficiency testing, maintainability testing, portability testing.. and plenty more I am sure. Keep on expanding your knowledge, wide and deep.

In the list above, I just scratch the surface on each one of these ideas. Spend time researching each of them, dig deeper into the ideas, or come up with your own. Have an idea, technique, or best practice I haven’t covered? Leave me some comments and let me know.

300 Level Testing – Test Case Design Techniques

Leveraging a process is the basis to defining and executing a test strategy.  This allows the development team to focus on repeatability, stability, speed, and results.  While researching the landscape of test design techniques, I came across three very good articles that outline a clearly defined set of test design strategies (articles from Invensis, Art of Testing, and Test Automation Resources).  These articles outline techniques based on static code and compiled dynamic code; manual and automated testing; black box, white box, and experience based testing.  Below is a breakdown of each of the test design techniques.  Some of these have been discussed in my articles on Testing 101 and 200 Level Testing.  Below is a summary of each, but follow up with each of the original articles to get more details, as well as examples.

Static Test Design Techniques


  • Walk through – a formal step by step review of all the features and documentation by the authors to better understand the software.
  • Informal review – as stated, these are more informal discussions to gather information without the documentation or code.
  • Technical review – more of a peer review of the application.
  • Audit – a formal review comparing code to documentation by an external source.
  • Inspection – a formal review by trained moderators, documenting defects in code and documentation through a detailed process.
  • Management review – a review of the project documents – project plan, budget, metrics, objectives and results, etc.

With Help of Tools

  • Analysis of coding standards (using compiler) – comparing the code against a set of rules, conventions, and standards defined within a tool or document.
  • Analysis of code metrics – analysis of things like cyclomatic numbers, complexity, nesting, lines of code, code coverage, etc.
  • Analysis of code structure – an analysis of the application by following the flow of data or paths through the code.  Also analyzes the structure of the data and the code itself.

Dynamic Test Design Techniques

Specification-based or Black-Box techniques

  • Boundary Value Analysis – test all field input values at the boundaries – highest, lowest, etc.
  • Decision Table Testing – Also called Classification Tree Method, build a decision tree for the logic of the application, and write tests for each of them.
  • State Transition Diagrams – test each of the states of the application, particularly workflow steps.
  • Equivalence Partitioning – reduce your number of tests by determining ones that test the same thing, and return the same results.
  • Use Case Testing – define scenarios based on business functionality or user functionality.
  • Combinatorial Testing – Randomly selected values, all possible values, each choice in at least one test, all-pairs or pair-wise or n-wise combinations, etc.

Structure-based or White-Box techniques

  • Statement Coverage or Line Coverage – similar t code metrics, analyzing the amount of code that has been exercised by tests.
  • Condition Coverage or Predicate Coverage – all conditions (i.e. true or false) are tested.
  • Decision Coverage or Branch Coverage – all conditions in each decision table are tested.
  • Multiple Condition Coverage – all values in all conditions are tested.

Experience-based techniques

  • Exploratory Testing – similar to an informal review, this testing is based on a general understanding of the application, product, domain, company, etc. and the experience and intuition of the tester.
  • Error Guessing or Fault Attack – leveraging prior experience and expertise, guess where the cracks are in the application, and focusing the testing there.

How to Choose the Right Technique

Once you have a general understanding of test design techniques, choosing the right approach is the most critical next step.  Here are some of the decision points to pick the right one:

  • Application Typebased on requirements for the domain as well as  mobile vs. web applications.
  • Regulatory standardsmust follow conventional rules based on IT, countries, government agencies, etc.
  • Customer’s requirements– based on relationships or contracts with customers.
  • Risk Level and TypeThis includes business risk, legal risk, compliance risk, brand risk, etc.
  • Objectives – Focus on the objectives of your testing.
  • Test Expertise – knowledge of the application, availability of documentation, familiarity with the techniques, etc.
  • Time and budget – What will provide the biggest value that fits your schedule.
  • SDLCWaterfall, Agile, Scrum, Kanban, Extreme… each affects which technique will fit.
  • Defect History – What kind of bugs have you found already for this app, in other apps, across the domain, etc.

200 Level Testing – 15 More Things To Do When There Are Still Bugs

All developers know that they need to test their code.  What new developers don’t understand is that the longer a bug exists in their code throughout the software developer lifecycle, the more expensive it costs to fix it.  So what do you do?  You start with the basic techniques I discussed in my last post. But that is not the end, only the beginning of the quality journey. Your next step is to bake Quality into your process, whether it is waterfall, agile, kanban, lean, etc.  You think about quality in every stage.  See some examples below of how to include testing throughout each phase of your software development process.

  1. Architectural Design – Plan your solution out before you write your code.  This includes how you will test your application.  Sometimes this means writing your functions or methods differently, or creating some test harness code.  Plan this out beforehand.
  2. Code Standards, Conventions, and Style Guides – Set the standards with the team, making code easier to read, modify, test, and predict.  Do you want to follow certain naming conventions?  Are you a fan of Hungarian Style?  Particular about indenting? Does bracket placement matter?  How about spacing?  Document it all, and stick to it.
  3. Use your User Personas – I bet at one time or another, you or yours  UX department has defined a series of personas to help define and prioritize features.  You should do the same for your code – use the journey of your personas to help you define what is critical for testing.
  4. Unit test your front end code, too – You have probably written lots of unit tests against your service layer and database code.  And, you probably have functional tests to exercise all that front end code. But don’t forget to unit test that front end code, too.  Libraries to facilitate this are mocha, karma, jasmine, jest, enzyme, selenium webdriver, cypress, puppeteer, protractor, and many more.
  5. Improve your Definition of Ready -Improving your code quality starts with purpose – what is the objective of your effort?  Ensure you have documented enough of your expected outcomes.  Just be sure not to take it too far – this could become an agile anti-pattern.
  6. Improve Acceptance Criteria –  Two great mnemonics to improve your acceptance criteria are INVEST and SMART.  Read more about them here. Remember the thoroughness of your acceptance criteria directly impacts code quality.
  7. Peer Code Reviews – One of the best ways to identify bugs as early in the process as possible.  Two heads are better than one.  Learn from your peers and catch errors early.  And here is a great article you can read about peer code reviews.
  8. Static Code Analysis Tools – Just like your coding standards and style guide, these tools analyze your source code (usually straight from your source code repository), compare your code against a wide range of rules, and help you identify areas that need help. Some examples of popular Static Code Analysis tools are Lint, AppScan, FxCop, StyleCop, Resharper, NUnit, SonarQube, and others.
  9. Dynamic Code Analysis / Vulnerability Test Tools – These tools are run against code actively running on a server in an attempt to measure resources used, find complexity, identify errors, or uncover vulnerabilities.
  10. Test Driven Development – Following this methodology, you think about your code first, write test cases that fail, and finally write your code until your tests pass.  Then you can write more test cases, or adjust, until you are complete.
  11. Behavior Driven Development – This is a methodology to define test scenarios using natural language, following a specific pattern, just like the user story. In fact, these scenarios can become your acceptance criteria, and using a tool like Gherkin, can become tests directly by Cucumber. Read more about BDD, and the benefits of BDD and TDD.
  12. Refactor – Just as much an art as a science, refactoring your code helps you to reorganize, simplify, provide focus and improve.
  13. Test Data, Test Data, Test Data – Sometimes, the key to testing all your scenarios is setting up your test data. Setting up all your automated test scenarios? Need an existing user to test a new feature? Need to mock results from a downstream system? Running a load test? You need data. Plan this out just as diligently as you would your test scenarios.
  14. Run your tests in all environments – Can you run your tests on your local machine? In Dev, Test, and Production? Both Green and Blue environments? Can you test all your downstream systems too? Is there data to be purged, or added to the environment? Need a flag to indicate this is test data? Plan this ahead – with your infrastructure teams, your partner systems, databases, and product teams to make this happen.
  15. Plan your Rollback before you need to – Not every release is a success. Faced with a production issue, you have to make the tough decision… roll back, or forward fix? What if you used Flagr, your entire feature was behind a feature flag, and could be turned off at the flip of a switch? What if you deployed using a blue / green strategy, and could roll back by flipping environments? Or kept a library of deployed code to re-implement at a moment’s notice? Plan your strategy ahead of time.


My next blog post will cover test strategies and techniques that can help you better define your test cases, and approach your code in unique, thorough, and thoughtful ways.

Testing 101 – 15 Ways to Build Quality Into Your Code

Things certainly have changed.  I remember a day when testing meant writing up a set of scenarios to manually test, or having a checklist of features to repeatedly test against on deployment day.  Now, with modern tools, there are lots more ways to test your code.  Automation can repeatedly test those scenarios for you, and ensure you didn’t break your code when new changes are made.  You can use these automation tests in lots of different ways, along with some basic things to keep in mind to ensure those tests are as valuable as possible.

Basic Types of Automation

  1. Unit Testing with Code Coverage – Write reusable test cases that test your functions, methods, and classes.  Integrate them into your automated build pipeline, and measure how much of your code is tested.  Aim for 100%, but expect a bit less.  Some unit testing tools are jUnit, nUnit, CA LISA, Rhino Mocks, and many more.
  2. Functional / System Testing – Write tests from your customer’s perspective.  walk through your application in the browser, fill in the fields with predefined values, and submit.  Test if the data is correct, if the validations worked, if the data was submitted.  Common tools for this are Selenium, Cucumber, CA Lisa, Rhino Mocks, etc.
  3. Browser and Device Testing – The Browser Wars may be over, but we still have a fragmented landscape of browsers used, and you have to test them all – Chrome, Edge, IE, Safari, Firefox, Android, iOS, phones, tablets, desktops… To cover them all, you can automate your tests and integrate with tools like BrowserStack, SauceLabs, Selenium, etc.
  4. Beta / Acceptance Testing – You need to ensure that you have built a quality project that meets the requirements of the business and the customer.  This is usually a set of manual tests completed by your customer before and / or after launch.
  5. Install Testing – This subset of existing automation tests ensures you exercise as much of your codebase as possible in a new environment, particularly production, to ensure your server installation has completed properly.
  6. Smoke Testing – A small subset of existing automation tests, most probably used in other cases, that ensures any environment changes have not had an impact on your codebase and its functionality.  These tests usually follow the happy paths (or red routes).
  7. Regression Testing – a complete and thorough suite of tests that exercise all of your code, new features and old, happy path and all known edge cases, including past and present bugs, ensuring you are prepared for a launch to production.
  8. Performance / Load / Stress Tests – Identify break points and thresholds in your software and hardware, and improve.  Common tools include jMeter, LoadRunner, Silk Performer, and many others.
  9. Security / Penetration tests – Make your code more secure, prevent hacking and attacks.  Tools include SonarQube, OWASP ZAP, Nexpose, WireShark, Retina, Aquila, and many more.

Tips To Keep In Mind

  1. Keep your code simple – Keep your code focused on solving one problem at a time.  This makes it easier to read and understand, code correctly, debug, and maintain.
  2. Pair Programming – Two heads are better than one.  Write your code cleaner, faster, and with less bugs.  This method comes with built in code reviews.
  3. Use Design Patterns and Architecture Patterns – Follow these, and you are using proven methods that other developers will understand.  The Gang of Four is a good place to start learning design patterns, and there are many others.  MVC, MVP, MVVM, and VIPER are some common architecture patterns that you can use to organize your code at an application level.
  4. Comment and Document your Code – Do this as close to the code as possible.  This makes your code easier to read and understand.  One way I like to do this is write pseudocode first, then fill in with actual code.  Then the pseudocode becomes natural code commenting.
  5. Automated Builds and Continuous Integration – Run your code and test cases as often as possible, make sure they all work together.  Jenkins is the most common way to achieve this.  Another common tool is Bamboo.
  6. Demos – this forces you to think holistically about your code, how you will communicate your solution to others, and gather feedback from your audience.  If you follow an agile methodology, chances are this is already part of your process.


9 Tools to Help You Recover from Malware

So I was at my parents’ house this weekend, and my dad asked a very simple question… “Could you take a look at my computer?  I can’t run CCleaner.”  I thought to myself… That’s great!  He has been running CCleaner regularly!  But when I sat down to take a look, things only got worse.  Windows Update wouldn’t start, CCleaner wouldn’t start, Microsoft Security Essentials wouldn’t start properly or update, and AVG Free wouldn’t install.  That is not good.  Here is a list of tools I used to scrub his machine, and boy did i need them all, and for different reasons.

  1. MalwareBytes Anti-Malware
  2. PortableApps
  3. SpyBot Search & Destroy
  4. SuperAntiSpyware
  5. CCleaner
  6. HijackThis
  7. AVG
  8. Microsoft Windows Defender
  9. Belarc Advisor

14 Web Site Graders To Test Your Redesigned Site

When you redesign or enhance your site, you make a lot of changes.  You change the content, the design, the front end technology, the back end stack, the user flows, the information architecture, everything.  It is tough to know what you have done right, and what needs help, particularly as it compares to other sites.  These sites can help show you what you have done right, what needs help, and how you compare to other sites.  I use them… and so should you.

  • – the gold standard of online web site graders. Shows performance, SEO, mobile capability, and security.
  • – this site gathers a LOT of marketing information about your site… Monitor this information before and after your cutover.
  • – Are you W3C Compliant?  Are you writing valid HTML?  Using this throughout your development will ensure your site is as readable and indexable as possible.
  • – How long does the first view of my page take?  How about the second view?  This grader shows you both… just like the Developer Tools in Google Chrome.
  • – another technical site grader that can give you guidance where to increase performance.  Be careful trying to get 100/100, though… not everything NEEDS to be done.
  • – Evaluates your site down in four areas – Accessibility, Experience, Marketing, and Technology.  Still useful to get another view of your site.
  • – “Run a review to see how your site can improve across 70+ metrics” – Marketing, SEO, Mobile, Usability, Technology, Crawl Errors, Backlinks, Social, Local, SERP Checker, Visitors.
  • – Another great site for a large, corporate web site.  But not a lot of information about performance.  Good to monitor usage and marketing metrics.
  • – Moz is known for its SEO tools, and this is an easy dashboard of information to monitor before and after your redesign.  The free version is useful, but the Pro version is even better.  Not a lot of tech help here, though.
  • – 7 days for free, the paid version is the only one really useful.  Lots of marketing information is available, though.
  • – Very technical.  Shows you the infrastructure and software choices made by the development team.  You will be surprised.  Helpful for technology and information security teams.
  •  – Free analytics tool.  Tells you who uses your site, how much, where they are from, what browsers, what time of day… a plethora of information.  Including Page Speed.
  • – Free tool that shows you what index errors Google has encountered, things to make your site more indexable, and what your pages look like to the Google Search Crawlers.  Use this.
  • – Everything that Search Console is for Google, this site is for Bing.

So did I miss any tools that you use?  Are any of these ones you have struck off your list?  How do you measure results of your site before and after?  Leave a comment and let me know!

EDIT: Two more sites were recommended to me that help redesign projects, so I am adding them here:

8 Planning Poker Options for Remote Teams

The most common way to size user stories for an agile team is to use planning poker and fibonacci sequence numbers.  But sometimes doing this is difficult if you are not colocated.  I was a big fan of  But without much fanfare, they changed their tool to only allow 10 people into their estimation session.  Definitely hoses half of my team.  now i need a new tool to help me size my user stories.  Here are some web based tools that can help me out.

  1. The Original – Planning Poker – .  This is what I used for years.  But now, the free version is only available to 10 people on your team at a time.  To add more you must pay $25 per month.  Crazy talk.
  2. Pointing Poker – .  Simple, basic, very popular, and FREE.  That is all that needs to be said.
  3. Plan It Poker – .  I like their tag line.  “Completely free to use no matter how large your team.”  Sign me up.
  4. Planning Poker for Hangouts – .  If you or your team are fans of Google Hangouts, this is the tool for you.
  5. Scrummy – .  This one looks cool… but sounds like it was more of a technology proof of concept than a new product to be launched.
  6. FirePoker – .  Another popular estimation tool.  This one uses angular.js .  Give it a try.
  7. Planning Poker (old version) – .  This one looks to be a legacy install of the original planning poker before the change, and before the redesign.  This might be the answer to my problem.
  8. Agile Estimation for Jira – .  If you are a Jira user,and don’t mind spending money on a plug-in, this is for you.  Not how I would go… but maybe you will.

Which one do you use?  How do you estimate?  Have a tool I missed?  Leave me a comment and let me know!

Debugging Home Run – Problem Step Recorder in Windows 7


I just got an email about a really cool new tool built into Windows 7 that Microsoft used to debug their new platform. It is called Problem Step Recorder. The best thing to do is to post a snippet of the email right here. I think it says everything perfectly:

“In case you’re not aware of this, here is a little known Microsoft tool bundled with Windows 7 that can be extremely useful to illustrate a problem when testing an application. The diagnostic tool called “Problem Step Recorder” was originally produced by Microsoft during the development of Windows 7 Beta to assist their Quality Assurance team in debugging the OS. It uses a combination of screen captures with mouse tracking to record your actions and can be a great way of describing a problem to others. The program is launched from the Start menu by typing ‘psr’ or ‘psr.exe’ in the search field. You’ll get a floating applet that looks like this: When you hit the Record button, the applet tracks your mouse and keyboard input while taking screenshots that correspond to each new action. When you stop recording your session is saved to an HTML slide show that recreates your steps. It also allows you to add comments to further document the problem. I think it can be very useful as an attachment in [your bug tracking tool] for those hard to describe issues or as a “How To” document for end users.”

Which leads to other ways of doing this… you could youse WebEx or Windows Media Encoder to document any bug as a step-by-step. If you use WatiN, Selenium, or VS2010, you can also use their recorders to document any bugs you may find in a web application, hand that to the dev team, and then there is no guessing how to reproduce the bug.

Kudos to Microsoft, and to the folks who uncovered this!

Virtumonde is not your friend

I was the victim of a very annoying piece of malware. I have been avoiding the corporate install of Internet Explorer for months now, and I have been using Firefox 2 and 3 instead. I am sure I was doing something I should not have been, because for the last two weeks these strange popups have been plaguing my Firefox browsers, and my machine has been running like there was taffy on my hard drive. I tried to remove the trojan with Spybot S&D, and that did not work. It did identify a Browser Helper Object (BHO) and some registry entries that I could not get rid of. That is when I knew it would be bad. Derek recommended that I try Macafee Avert Stinger. That was no help either. I tried HijackThis. That was informative, but not as helpful as I had hoped. So I did some more digging online, and an article recommended Malwarebytes’ Anti-Malware (MBAM). That was a big step forward. It clearly identified my problem as the Virtumonde Trojan. There were 59 DLLs, BHOs, data files, and registry entries all over my computer from this one trojan. I used MBAM to remove all of them, but the BHO registry entry was stubborn. This meant there was still more. I did some research on Virtumonde, and found that a tool called ComboFix will wipe it out entirely. It took about 20 minutes to run, rebooted my machine, and took another 20 minutes to complete. But when it was all done, I was trojan free. No more popups when I use Firefox, and my machine is fast again. Now… if only I knew what I did that was so bad…

Software Metrics

Being able to measure success for a software development group is a difficult thing. But not being able to show the success of your development group is a dangerous thing. Your management team will want to be able to measure quality, show improvement, and benchmark against industry standards. All the popular maturity models (if you put stock into them) emphasize the ability to measure, react, and improve your quality and processes. My department is no different. We try to remain as flexible and lightweight as possible, and add formality where improvement is necessary. Here are some of the metrics that we collect to measure our success and find ares of improvement.

Code Coverage

There will always be bugs. No matter how hard you try, there will always be bugs in your software. That does not mean that you cannot try to prevent them. One of the easiest ways to do that is to write automated unit tests to validate your code is doing what is expected. You can write your unit tests lots of different ways, and Steve Cornett’s Code Coverage Analysis paper gives lots of different ways to break down code coverage. A great place to start is to aim for 50% coverage of all of your lines of code. And, as Tony Obermeit says in his Code Coverage Lessons paper, your ultimate goal of course is to always hit 100% coverage. You will need to pick a code coverage tool to help measure your success. In my department, developing in a Visual Studio and C# environment lends itself to the nCover open source solution. This solution works well with our CruiseControl environment. Test Driven Development methodologies and mocking tools can help you get closer and closer to covering as much of your code as possible with automated tests.

Defect Tracking

I use the words defects and bugs interchangeably. This I am sure is something that some people would disagree with, but I think that it is close enough. If it is not working as expected, then it is a defect, and it is a bug. Regardless, defects are identified in the development process, system and user acceptance testing process, and in the production environment. The objective is to minimize the number of bugs that are found in the live environments. To do that, you need to encourage the identification and mitigation of bugs in earlier and earlier stages. This sounds fundamental, but becomes difficult to implement. There are lots of methods that you can do to identify, solve, remove, and prevent bugs. You must have a way to measure that these methods are improving your success rate. And that means measuring the number of defects found in each environment – development, system test, user acceptance test, and production. The easiest set of numbers to get in a corporate environment is production defects. There is always a problem management system or help desk that tracks these kinds of things. But as a software development organization, you need to implement bug tracking throughout the entire lifecycle of your software. You can trend the numbers, and make sure you are finding more bugs before UAT, particularly in development. Tools like Bugzilla, an open source bug tracking tool, can help you track, trend, and manage defects in your software throughout its lifecycle.

Support Ticket Management

Software is not a static entity. It is always changing. Just think of all the patches, updates, service packs, and bug fixes that Microsoft releases on its suite of software. In a corporate environment, it is no different. Software management does not end once it is released. Teams of developers will be constantly updating desktop, web based, and console applications based on new requirements and requests from their clients. Problem Management software can be used to help track and trend all of these requests bydata points such as severity (Urgent, High, Medium, Low), priority (Urgent, High, Medium, Low), assessment (Customer Assistance, Required Modification, Elective Enhancement, Break Fix), difficulty (High, Medium, Low), etc. You can measure success agains more complex metrics such as the number of tickets created, number of open tickets, time to resolution, etc. All of these metrics will help you determine how fluid, stable, usable, sustainable, and maintainable your software is. Do not ignore your software or its users once it is released to production.


Web Analytics tools can tell you how many users you have had on your site, how long they visited, where they came from, where they went, how they found your site, did they reach your goal pages, did they convert, and did they return. There are free web based tools like Google Analytics, and over the shelf packages like WebTrends and CoreMetrics that can help you measure site activity. Do not ignore these metrics to help you define current activity, make improvements to your site, track your new results, and continue to improve. They directly measure your clients’ interaction with yoru software, and can identify trends that with simple changes can vastly improve your software and development processes.


So… these are some of the ways that we track the success of our software. There are a host of other methods to measure software, such as function points, lines of code, complexity, interfaces, velocity, etc. What ways do you measure your software? how do you define success? What are your plans for the future?

Leave me a comment and let me know what you think.