Leveraging a process is the basis to defining and executing a test strategy.  This allows the development team to focus on repeatability, stability, speed, and results.  While researching the landscape of test design techniques, I came across three very good articles that outline a clearly defined set of test design strategies (articles from Invensis, Art of Testing, and Test Automation Resources).  These articles outline techniques based on static code and compiled dynamic code; manual and automated testing; black box, white box, and experience based testing.  Below is a breakdown of each of the test design techniques.  Some of these have been discussed in my articles on Testing 101 and 200 Level Testing.  Below is a summary of each, but follow up with each of the original articles to get more details, as well as examples.

Static Test Design Techniques

Manual

• Walk through – a formal step by step review of all the features and documentation by the authors to better understand the software.
• Informal review – as stated, these are more informal discussions to gather information without the documentation or code.
• Technical review – more of a peer review of the application.
• Audit – a formal review comparing code to documentation by an external source.
• Inspection – a formal review by trained moderators, documenting defects in code and documentation through a detailed process.
• Management review – a review of the project documents – project plan, budget, metrics, objectives and results, etc.

With Help of Tools

• Analysis of coding standards (using compiler) – comparing the code against a set of rules, conventions, and standards defined within a tool or document.
• Analysis of code metrics – analysis of things like cyclomatic numbers, complexity, nesting, lines of code, code coverage, etc.
• Analysis of code structure – an analysis of the application by following the flow of data or paths through the code.  Also analyzes the structure of the data and the code itself.

Dynamic Test Design Techniques

Specification-based or Black-Box techniques

• Boundary Value Analysis – test all field input values at the boundaries – highest, lowest, etc.
• Decision Table Testing – Also called Classification Tree Method, build a decision tree for the logic of the application, and write tests for each of them.
• State Transition Diagrams – test each of the states of the application, particularly workflow steps.
• Equivalence Partitioning – reduce your number of tests by determining ones that test the same thing, and return the same results.
• Use Case Testing – define scenarios based on business functionality or user functionality.
• Combinatorial Testing – Randomly selected values, all possible values, each choice in at least one test, all-pairs or pair-wise or n-wise combinations, etc.

Structure-based or White-Box techniques

• Statement Coverage or Line Coverage – similar t code metrics, analyzing the amount of code that has been exercised by tests.
• Condition Coverage or Predicate Coverage – all conditions (i.e. true or false) are tested.
• Decision Coverage or Branch Coverage – all conditions in each decision table are tested.
• Multiple Condition Coverage – all values in all conditions are tested.

Experience-based techniques

• Exploratory Testing – similar to an informal review, this testing is based on a general understanding of the application, product, domain, company, etc. and the experience and intuition of the tester.
• Error Guessing or Fault Attack – leveraging prior experience and expertise, guess where the cracks are in the application, and focusing the testing there.

How to Choose the Right Technique

Once you have a general understanding of test design techniques, choosing the right approach is the most critical next step.  Here are some of the decision points to pick the right one:

• Application Type – based on requirements for the domain as well as  mobile vs. web applications.
• Regulatory standards – must follow conventional rules based on IT, countries, government agencies, etc.
• Customer’s requirements– based on relationships or contracts with customers.
• Risk Level and Type – This includes business risk, legal risk, compliance risk, brand risk, etc.
• Objectives – Focus on the objectives of your testing.
• Test Expertise – knowledge of the application, availability of documentation, familiarity with the techniques, etc.
• Time and budget – What will provide the biggest value that fits your schedule.
• SDLC – Waterfall, Agile, Scrum, Kanban, Extreme… each affects which technique will fit.
• Defect History – What kind of bugs have you found already for this app, in other apps, across the domain, etc.

All developers know that they need to test their code.  What new developers don’t understand is that the longer a bug exists in their code throughout the software developer lifecycle, the more expensive it costs to fix it.  So what do you do?  You start with the basic techniques I discussed in my last post. But that is not the end, only the beginning of the quality journey. Your next step is to bake Quality into your process, whether it is waterfall, agile, kanban, lean, etc.  You think about quality in every stage.  See some examples below of how to include testing throughout each phase of your software development process.

1. Architectural Design – Plan your solution out before you write your code.  This includes how you will test your application.  Sometimes this means writing your functions or methods differently, or creating some test harness code.  Plan this out beforehand.
2. Code Standards, Conventions, and Style Guides – Set the standards with the team, making code easier to read, modify, test, and predict.  Do you want to follow certain naming conventions?  Are you a fan of Hungarian Style?  Particular about indenting? Does bracket placement matter?  How about spacing?  Document it all, and stick to it.
3. Use your User Personas – I bet at one time or another, you or yours  UX department has defined a series of personas to help define and prioritize features.  You should do the same for your code – use the journey of your personas to help you define what is critical for testing.
4. Unit test your front end code, too – You have probably written lots of unit tests against your service layer and database code.  And, you probably have functional tests to exercise all that front end code. But don’t forget to unit test that front end code, too.  Libraries to facilitate this are mocha, karma, jasmine, jest, enzyme, selenium webdriver, cypress, puppeteer, protractor, and many more.
5. Improve your Definition of Ready -Improving your code quality starts with purpose – what is the objective of your effort?  Ensure you have documented enough of your expected outcomes.  Just be sure not to take it too far – this could become an agile anti-pattern.
6. Improve Acceptance Criteria –  Two great mnemonics to improve your acceptance criteria are INVEST and SMART.  Read more about them here. Remember the thoroughness of your acceptance criteria directly impacts code quality.
7. Peer Code Reviews – One of the best ways to identify bugs as early in the process as possible.  Two heads are better than one.  Learn from your peers and catch errors early.  And here is a great article you can read about peer code reviews.
8. Static Code Analysis Tools – Just like your coding standards and style guide, these tools analyze your source code (usually straight from your source code repository), compare your code against a wide range of rules, and help you identify areas that need help. Some examples of popular Static Code Analysis tools are Lint, AppScan, FxCop, StyleCop, Resharper, NUnit, SonarQube, and others.
9. Dynamic Code Analysis / Vulnerability Test Tools – These tools are run against code actively running on a server in an attempt to measure resources used, find complexity, identify errors, or uncover vulnerabilities.
10. Test Driven Development – Following this methodology, you think about your code first, write test cases that fail, and finally write your code until your tests pass.  Then you can write more test cases, or adjust, until you are complete.
11. Behavior Driven Development – This is a methodology to define test scenarios using natural language, following a specific pattern, just like the user story. In fact, these scenarios can become your acceptance criteria, and using a tool like Gherkin, can become tests directly by Cucumber. Read more about BDD, and the benefits of BDD and TDD.
12. Refactor – Just as much an art as a science, refactoring your code helps you to reorganize, simplify, provide focus and improve.
13. Test Data, Test Data, Test Data – Sometimes, the key to testing all your scenarios is setting up your test data. Setting up all your automated test scenarios? Need an existing user to test a new feature? Need to mock results from a downstream system? Running a load test? You need data. Plan this out just as diligently as you would your test scenarios.
14. Run your tests in all environments – Can you run your tests on your local machine? In Dev, Test, and Production? Both Green and Blue environments? Can you test all your downstream systems too? Is there data to be purged, or added to the environment? Need a flag to indicate this is test data? Plan this ahead – with your infrastructure teams, your partner systems, databases, and product teams to make this happen.
15. Plan your Rollback before you need to – Not every release is a success. Faced with a production issue, you have to make the tough decision… roll back, or forward fix? What if you used Flagr, your entire feature was behind a feature flag, and could be turned off at the flip of a switch? What if you deployed using a blue / green strategy, and could roll back by flipping environments? Or kept a library of deployed code to re-implement at a moment’s notice? Plan your strategy ahead of time.

My next blog post will cover test strategies and techniques that can help you better define your test cases, and approach your code in unique, thorough, and thoughtful ways.

Things certainly have changed.  I remember a day when testing meant writing up a set of scenarios to manually test, or having a checklist of features to repeatedly test against on deployment day.  Now, with modern tools, there are lots more ways to test your code.  Automation can repeatedly test those scenarios for you, and ensure you didn’t break your code when new changes are made.  You can use these automation tests in lots of different ways, along with some basic things to keep in mind to ensure those tests are as valuable as possible.

Basic Types of Automation

1. Unit Testing with Code Coverage – Write reusable test cases that test your functions, methods, and classes.  Integrate them into your automated build pipeline, and measure how much of your code is tested.  Aim for 100%, but expect a bit less.  Some unit testing tools are jUnit, nUnit, CA LISA, Rhino Mocks, and many more.
2. Functional / System Testing – Write tests from your customer’s perspective.  walk through your application in the browser, fill in the fields with predefined values, and submit.  Test if the data is correct, if the validations worked, if the data was submitted.  Common tools for this are Selenium, Cucumber, CA Lisa, Rhino Mocks, etc.
3. Browser and Device Testing – The Browser Wars may be over, but we still have a fragmented landscape of browsers used, and you have to test them all – Chrome, Edge, IE, Safari, Firefox, Android, iOS, phones, tablets, desktops… To cover them all, you can automate your tests and integrate with tools like BrowserStack, SauceLabs, Selenium, etc.
4. Beta / Acceptance Testing – You need to ensure that you have built a quality project that meets the requirements of the business and the customer.  This is usually a set of manual tests completed by your customer before and / or after launch.
5. Install Testing – This subset of existing automation tests ensures you exercise as much of your codebase as possible in a new environment, particularly production, to ensure your server installation has completed properly.
6. Smoke Testing – A small subset of existing automation tests, most probably used in other cases, that ensures any environment changes have not had an impact on your codebase and its functionality.  These tests usually follow the happy paths (or red routes).
7. Regression Testing – a complete and thorough suite of tests that exercise all of your code, new features and old, happy path and all known edge cases, including past and present bugs, ensuring you are prepared for a launch to production.
8. Performance / Load / Stress Tests – Identify break points and thresholds in your software and hardware, and improve.  Common tools include jMeter, LoadRunner, Silk Performer, and many others.
9. Security / Penetration tests – Make your code more secure, prevent hacking and attacks.  Tools include SonarQube, OWASP ZAP, Nexpose, WireShark, Retina, Aquila, and many more.

Tips To Keep In Mind

10. Keep your code simple – Keep your code focused on solving one problem at a time.  This makes it easier to read and understand, code correctly, debug, and maintain.
11. Pair Programming – Two heads are better than one.  Write your code cleaner, faster, and with less bugs.  This method comes with built in code reviews.
12. Use Design Patterns and Architecture Patterns – Follow these, and you are using proven methods that other developers will understand.  The Gang of Four is a good place to start learning design patterns, and there are many others.  MVC, MVP, MVVM, and VIPER are some common architecture patterns that you can use to organize your code at an application level.
13. Comment and Document your Code – Do this as close to the code as possible.  This makes your code easier to read and understand.  One way I like to do this is write pseudocode first, then fill in with actual code.  Then the pseudocode becomes natural code commenting.
14. Automated Builds and Continuous Integration – Run your code and test cases as often as possible, make sure they all work together.  Jenkins is the most common way to achieve this.  Another common tool is Bamboo.
15. Demos – this forces you to think holistically about your code, how you will communicate your solution to others, and gather feedback from your audience.  If you follow an agile methodology, chances are this is already part of your process.