Now that we have walked through the basics in my posts on Testing 101, 200 Level, and 300 Level testing, the number of bugs in your production environment should be falling. To continue to eliminate the nasty bugs, it is time to dig deep, get creative, and use skills from other fields. Here are some out-of-the-box ideas to continue the momentum and focus on quality and stability of your software products.

1. Shift Left – this is the general principle to find bugs as soon as possible in the software development process – write automation tests to find bugs in QA, write unit tests to find bugs in development, use TDD to find bugs before you write code, architect testability into your platform before you even create the solution.  TechBeacon has a great article about Shift Left.
2. Crowdsource Testing – sometimes you need to get a fresh, new perspective on testing.  Using the general public can help you get there.  I have used a  company called uTest, and they were great.  They have all kinds of purchase options.  There are lots of other crowdsource testing companies to choose from.
3. Fuzzing – I have done this before and never knew it was called fuzzing.  Build automation tests that randomly generate characters (ISO-8859-1 and UTF-8) and submit forms, to test positive and negative cases you never thought of.  Run it thousands of times, and you are bound to find some errors.  Read F-Secure to learn more about fuzzing.
4. Innovation – When most people think of innovation, they think of a bolt of lightning, striking randomly. While this may be true some of the time, that isn’t always the case. There are methods to developing and trying new ideas, and that kind of innovation needs a culture to support it. This great article from The Medium outlines some of the methodologies to use to find innovation.
5. Design Thinking – Over the last few years, I have been working with Rutgers on their Advisory Board for their Design Thinking certificate program. I stay on top of the subject, and tweet a lot of great articles to read on the topic. When I came across this article on DT and Software Quality, I knew I needed to share it.
6. Bug Bash / Bug Hunt / Bug Day – Make finding bugs a social event. Invite your team, or other teams, to find bugs in your product. Set aside an afternoon, order some pizza, plan some prizes, and find some bugs! Read more about how to run a Bug Hunt.
7. Quality Hackathon – another way to generate ideas about new approaches to testing and quality, you could hold a hackathon, one that focuses specifically on testing. Are there new ways to automate? New tools to try? New approaches to testing? Challenge your team, set some parameters, pick some judges, give away some prizes, and make it fun!
8. Apply Gamification to Testing – In World of Warcraft, there are monsters to kill, skills to learn, reputation to gain, equipment to discover.  You can do the same thing with Quality and Gamification – each person on the team has a QE Score, and gets points for bugs found, classes taken, Tech Talks conducted, junior teammates mentored.   Define a reward system, how to grow, and publicly praise the growth – different levels, badges, and achievements to earn and the steps to get there.  Read more about gamification on CIO.com.
9. Machine Learning and Testing – Can we teach a machine to find bugs in software that humans write? Or identify what sections of our software systems will most likely have the most bugs? Or what sections of code are the most complex? Maybe we should give it a shot…
10. Measure your Test Maturity – According to CIO Magazine, “The Capability Maturity Model Integration (CMMI) is a process and behavioral model that helps organizations streamline process improvement and encourage productive, efficient behaviors that decrease risks in software, product and service development.” Initially developed by the Software Engineering Institute at Carnegie Mellon University, in conjunction with the DoD and U.S. Government, CMMI is currently administered by the CMMI Institute as a process improvement tool for projects, divisions or organizations. CMM defines 5 levels of maturity – (1) Initial, (2) Repeatable, (3) Defined, (4) Managed, and (5) Optimizing. If thought of in a different way, this maturity model could be specifically applied to software quality and testing, and used to measure current state and set goals for improvement.
11. Think Different – While this may be the slogan for a famous ad campaign for a large tech company named after Newton’s fruit, it is also a technique to help people solve problems by taking a different mental approach – Creative, Analytical, Critical. Concrete, Abstract, Divergent, Convergent, Sequential, and Holistic thinking.  Read more about software testing and different thinking types on testingexcellence.com.
12. Complexity = Errors – Exploit the correlation between complexity and errors. A rule of thumb is that one component should do one thing, and do it well. Make complex classes smaller. Break monolithic services into micro services. Break database tables into multiple, smaller tables. The same is also true for tests. Make unit tests smaller. Test smaller sections of the user journey. And aim your spotlight of testing onto complex areas of your application – UI, services, or data, integration points, service signatures, common failure points.
13. Wear different Thinking Caps – It’s a common enough expression… “put on your thinking cap.” The 1985 book Six Thinking Hats by Edward de Bono promotes multi-dimensional thinking by mapping different colored hats to different cognitive styles – blue for an overview, white for facts, red for emotion, yellow for optimism, black for risks, green for creativity. Put on different thinking caps, and innovate in whole new ways.
14. World Quality Report – “The World Quality Report is the only global report analyzing software testing and quality engineering trends. It presents an analysis of developments in agile and DevOps, artificial intelligence, automation, test environments, data, security and budgets, showing once again the importance of quality, and of the measures that are put in place to maintain it.” – sogeti (part of capgemini)
15. Online Resources – Things change, evolve, improve. It is our obligation as technologists to stay on top of those changes. There are a lot of resources that can help you do that. Training, conferences, certifications, and industry trends are some ways to stay on top of those changes. Sometimes it is just staying on tho of the news as it happens. While doing research for my blog posts, some of the best resources for me were Ministry of Testing, Software Testing Help, StickyMinds, InfoWorld, and Software Testing Magazine.
16. Training – Continuing your education as a technologist and quality engineer is critical. There are lots of online video training resources available online. Pick your favorite and keep learning. Two of my favorite are Udemy (and their class on Software Testing and Innovation): and Lynda (and their class on Exploratory Testing).
17. Conferences – Learn about new techniques, hear from industry experts, plan for future innovation, look at products from vendors, continue your education. Check out a list of conferences from TechBeacon and testingconferences.org .
18. Certifications – Certifications are more than just a test and a score. They are managed by a company with resources, continuing education, advanced certifications, and a community of professionals you can tap into. Check out a list of certifications for Software Testing professionals.
19. Industry Trends – Remain on top of industry trends allows you to adapt to current challenges, leverage new tools and techniques, and benefit your skills and your company. Read more about quality assurance trends in 2019.
20. Beyond The Bugs – What I have focused on in these last four posts is how to find bugs in your code. Ones that would functionally prevent users from using your site. But there are lots of non-functional ways to test your site. Security testing, penetration testing, performance testing, reliability testing, efficiency testing, maintainability testing, portability testing.. and plenty more I am sure. Keep on expanding your knowledge, wide and deep.

In the list above, I just scratch the surface on each one of these ideas. Spend time researching each of them, dig deeper into the ideas, or come up with your own. Have an idea, technique, or best practice I haven’t covered? Leave me some comments and let me know.

Leveraging a process is the basis to defining and executing a test strategy.  This allows the development team to focus on repeatability, stability, speed, and results.  While researching the landscape of test design techniques, I came across three very good articles that outline a clearly defined set of test design strategies (articles from Invensis, Art of Testing, and Test Automation Resources).  These articles outline techniques based on static code and compiled dynamic code; manual and automated testing; black box, white box, and experience based testing.  Below is a breakdown of each of the test design techniques.  Some of these have been discussed in my articles on Testing 101 and 200 Level Testing.  Below is a summary of each, but follow up with each of the original articles to get more details, as well as examples.

Static Test Design Techniques

Manual

• Walk through – a formal step by step review of all the features and documentation by the authors to better understand the software.
• Informal review – as stated, these are more informal discussions to gather information without the documentation or code.
• Technical review – more of a peer review of the application.
• Audit – a formal review comparing code to documentation by an external source.
• Inspection – a formal review by trained moderators, documenting defects in code and documentation through a detailed process.
• Management review – a review of the project documents – project plan, budget, metrics, objectives and results, etc.

With Help of Tools

• Analysis of coding standards (using compiler) – comparing the code against a set of rules, conventions, and standards defined within a tool or document.
• Analysis of code metrics – analysis of things like cyclomatic numbers, complexity, nesting, lines of code, code coverage, etc.
• Analysis of code structure – an analysis of the application by following the flow of data or paths through the code.  Also analyzes the structure of the data and the code itself.

Dynamic Test Design Techniques

Specification-based or Black-Box techniques

• Boundary Value Analysis – test all field input values at the boundaries – highest, lowest, etc.
• Decision Table Testing – Also called Classification Tree Method, build a decision tree for the logic of the application, and write tests for each of them.
• State Transition Diagrams – test each of the states of the application, particularly workflow steps.
• Equivalence Partitioning – reduce your number of tests by determining ones that test the same thing, and return the same results.
• Use Case Testing – define scenarios based on business functionality or user functionality.
• Combinatorial Testing – Randomly selected values, all possible values, each choice in at least one test, all-pairs or pair-wise or n-wise combinations, etc.

Structure-based or White-Box techniques

• Statement Coverage or Line Coverage – similar t code metrics, analyzing the amount of code that has been exercised by tests.
• Condition Coverage or Predicate Coverage – all conditions (i.e. true or false) are tested.
• Decision Coverage or Branch Coverage – all conditions in each decision table are tested.
• Multiple Condition Coverage – all values in all conditions are tested.

Experience-based techniques

• Exploratory Testing – similar to an informal review, this testing is based on a general understanding of the application, product, domain, company, etc. and the experience and intuition of the tester.
• Error Guessing or Fault Attack – leveraging prior experience and expertise, guess where the cracks are in the application, and focusing the testing there.

How to Choose the Right Technique

Once you have a general understanding of test design techniques, choosing the right approach is the most critical next step.  Here are some of the decision points to pick the right one:

• Application Type – based on requirements for the domain as well as  mobile vs. web applications.
• Regulatory standards – must follow conventional rules based on IT, countries, government agencies, etc.
• Customer’s requirements– based on relationships or contracts with customers.
• Risk Level and Type – This includes business risk, legal risk, compliance risk, brand risk, etc.
• Objectives – Focus on the objectives of your testing.
• Test Expertise – knowledge of the application, availability of documentation, familiarity with the techniques, etc.
• Time and budget – What will provide the biggest value that fits your schedule.
• SDLC – Waterfall, Agile, Scrum, Kanban, Extreme… each affects which technique will fit.
• Defect History – What kind of bugs have you found already for this app, in other apps, across the domain, etc.

All developers know that they need to test their code.  What new developers don’t understand is that the longer a bug exists in their code throughout the software developer lifecycle, the more expensive it costs to fix it.  So what do you do?  You start with the basic techniques I discussed in my last post. But that is not the end, only the beginning of the quality journey. Your next step is to bake Quality into your process, whether it is waterfall, agile, kanban, lean, etc.  You think about quality in every stage.  See some examples below of how to include testing throughout each phase of your software development process.

1. Architectural Design – Plan your solution out before you write your code.  This includes how you will test your application.  Sometimes this means writing your functions or methods differently, or creating some test harness code.  Plan this out beforehand.
2. Code Standards, Conventions, and Style Guides – Set the standards with the team, making code easier to read, modify, test, and predict.  Do you want to follow certain naming conventions?  Are you a fan of Hungarian Style?  Particular about indenting? Does bracket placement matter?  How about spacing?  Document it all, and stick to it.
3. Use your User Personas – I bet at one time or another, you or yours  UX department has defined a series of personas to help define and prioritize features.  You should do the same for your code – use the journey of your personas to help you define what is critical for testing.
4. Unit test your front end code, too – You have probably written lots of unit tests against your service layer and database code.  And, you probably have functional tests to exercise all that front end code. But don’t forget to unit test that front end code, too.  Libraries to facilitate this are mocha, karma, jasmine, jest, enzyme, selenium webdriver, cypress, puppeteer, protractor, and many more.
5. Improve your Definition of Ready -Improving your code quality starts with purpose – what is the objective of your effort?  Ensure you have documented enough of your expected outcomes.  Just be sure not to take it too far – this could become an agile anti-pattern.
6. Improve Acceptance Criteria –  Two great mnemonics to improve your acceptance criteria are INVEST and SMART.  Read more about them here. Remember the thoroughness of your acceptance criteria directly impacts code quality.
7. Peer Code Reviews – One of the best ways to identify bugs as early in the process as possible.  Two heads are better than one.  Learn from your peers and catch errors early.  And here is a great article you can read about peer code reviews.
8. Static Code Analysis Tools – Just like your coding standards and style guide, these tools analyze your source code (usually straight from your source code repository), compare your code against a wide range of rules, and help you identify areas that need help. Some examples of popular Static Code Analysis tools are Lint, AppScan, FxCop, StyleCop, Resharper, NUnit, SonarQube, and others.
9. Dynamic Code Analysis / Vulnerability Test Tools – These tools are run against code actively running on a server in an attempt to measure resources used, find complexity, identify errors, or uncover vulnerabilities.
10. Test Driven Development – Following this methodology, you think about your code first, write test cases that fail, and finally write your code until your tests pass.  Then you can write more test cases, or adjust, until you are complete.
11. Behavior Driven Development – This is a methodology to define test scenarios using natural language, following a specific pattern, just like the user story. In fact, these scenarios can become your acceptance criteria, and using a tool like Gherkin, can become tests directly by Cucumber. Read more about BDD, and the benefits of BDD and TDD.
12. Refactor – Just as much an art as a science, refactoring your code helps you to reorganize, simplify, provide focus and improve.
13. Test Data, Test Data, Test Data – Sometimes, the key to testing all your scenarios is setting up your test data. Setting up all your automated test scenarios? Need an existing user to test a new feature? Need to mock results from a downstream system? Running a load test? You need data. Plan this out just as diligently as you would your test scenarios.
14. Run your tests in all environments – Can you run your tests on your local machine? In Dev, Test, and Production? Both Green and Blue environments? Can you test all your downstream systems too? Is there data to be purged, or added to the environment? Need a flag to indicate this is test data? Plan this ahead – with your infrastructure teams, your partner systems, databases, and product teams to make this happen.
15. Plan your Rollback before you need to – Not every release is a success. Faced with a production issue, you have to make the tough decision… roll back, or forward fix? What if you used Flagr, your entire feature was behind a feature flag, and could be turned off at the flip of a switch? What if you deployed using a blue / green strategy, and could roll back by flipping environments? Or kept a library of deployed code to re-implement at a moment’s notice? Plan your strategy ahead of time.

My next blog post will cover test strategies and techniques that can help you better define your test cases, and approach your code in unique, thorough, and thoughtful ways.

Things certainly have changed.  I remember a day when testing meant writing up a set of scenarios to manually test, or having a checklist of features to repeatedly test against on deployment day.  Now, with modern tools, there are lots more ways to test your code.  Automation can repeatedly test those scenarios for you, and ensure you didn’t break your code when new changes are made.  You can use these automation tests in lots of different ways, along with some basic things to keep in mind to ensure those tests are as valuable as possible.

Basic Types of Automation

1. Unit Testing with Code Coverage – Write reusable test cases that test your functions, methods, and classes.  Integrate them into your automated build pipeline, and measure how much of your code is tested.  Aim for 100%, but expect a bit less.  Some unit testing tools are jUnit, nUnit, CA LISA, Rhino Mocks, and many more.
2. Functional / System Testing – Write tests from your customer’s perspective.  walk through your application in the browser, fill in the fields with predefined values, and submit.  Test if the data is correct, if the validations worked, if the data was submitted.  Common tools for this are Selenium, Cucumber, CA Lisa, Rhino Mocks, etc.
3. Browser and Device Testing – The Browser Wars may be over, but we still have a fragmented landscape of browsers used, and you have to test them all – Chrome, Edge, IE, Safari, Firefox, Android, iOS, phones, tablets, desktops… To cover them all, you can automate your tests and integrate with tools like BrowserStack, SauceLabs, Selenium, etc.
4. Beta / Acceptance Testing – You need to ensure that you have built a quality project that meets the requirements of the business and the customer.  This is usually a set of manual tests completed by your customer before and / or after launch.
5. Install Testing – This subset of existing automation tests ensures you exercise as much of your codebase as possible in a new environment, particularly production, to ensure your server installation has completed properly.
6. Smoke Testing – A small subset of existing automation tests, most probably used in other cases, that ensures any environment changes have not had an impact on your codebase and its functionality.  These tests usually follow the happy paths (or red routes).
7. Regression Testing – a complete and thorough suite of tests that exercise all of your code, new features and old, happy path and all known edge cases, including past and present bugs, ensuring you are prepared for a launch to production.
8. Performance / Load / Stress Tests – Identify break points and thresholds in your software and hardware, and improve.  Common tools include jMeter, LoadRunner, Silk Performer, and many others.
9. Security / Penetration tests – Make your code more secure, prevent hacking and attacks.  Tools include SonarQube, OWASP ZAP, Nexpose, WireShark, Retina, Aquila, and many more.

Tips To Keep In Mind

10. Keep your code simple – Keep your code focused on solving one problem at a time.  This makes it easier to read and understand, code correctly, debug, and maintain.
11. Pair Programming – Two heads are better than one.  Write your code cleaner, faster, and with less bugs.  This method comes with built in code reviews.
12. Use Design Patterns and Architecture Patterns – Follow these, and you are using proven methods that other developers will understand.  The Gang of Four is a good place to start learning design patterns, and there are many others.  MVC, MVP, MVVM, and VIPER are some common architecture patterns that you can use to organize your code at an application level.
13. Comment and Document your Code – Do this as close to the code as possible.  This makes your code easier to read and understand.  One way I like to do this is write pseudocode first, then fill in with actual code.  Then the pseudocode becomes natural code commenting.
14. Automated Builds and Continuous Integration – Run your code and test cases as often as possible, make sure they all work together.  Jenkins is the most common way to achieve this.  Another common tool is Bamboo.
15. Demos – this forces you to think holistically about your code, how you will communicate your solution to others, and gather feedback from your audience.  If you follow an agile methodology, chances are this is already part of your process.