I grew up in the <a href="/bliki/WaterfallProcess.html">waterfall era </a>, where testing was seen as a separate activity to programming, done by a different group of people, and carried out after programming was done. The shift towards iterative and agile approaches, particularly the influence of <a href="/bliki/ExtremeProgramming.html">Extreme Programming </a>, has changed the role of testing - raising its importance, and integrating it with the core development work.

computerscience

Computer Science

At the core of my philosophy to testing is that we want to write <a href="/bliki/SelfTestingCode.html">self-testing code </a>, meaning we have a suite of automated tests that be easily run against the code base. We should be confident that this suite will catch almost all bugs in the software, so that when this suite “goes green” (passes), we can release that version of the software into production. This kind of environment can both drastically improve the robustness of our software, but also enables us to use <a href="https://refactoring.com">refactoring </a>to improve our productivity and responsiveness in the coming months.

<img src="https://martinfowler.com/bliki/images/selfTestingCode/sketch.png"/> Self-Testing Code is the name I used in <a href="/books/refactoring.html">Refactoring </a>to refer to the practice of writing comprehensive automated tests in conjunction with the functional software. When done well this allows you to invoke a single command that executes the tests - and you are confident that these tests will illuminate any bugs hiding in your code.

<img src=""/> Test-Driven Development (TDD) is a technique for building software that guides software development by writing tests. It was developed by <a href="https://twitter.com/KentBeck">Kent Beck </a>in the late 1990's as part of Extreme Programming. In essence you follow three simple steps repeatedly: by Martin Fowler 5 Mar 2005 <a href="/bliki/TestDrivenDevelopment.html">Read more… </a>

<img src="https://martinfowler.com/bliki/images/testPyramid/test-pyramid.png"/> The test pyramid is a way of thinking about how different kinds of automated tests should be used to create a balanced portfolio. Its essential point is that you should have many more low-level <a href="/bliki/UnitTest.html">UnitTests </a>than high level <a href="/bliki/BroadStackTest.html">BroadStackTests </a>running through a GUI. by Martin Fowler 1 May 2012 <a href="/bliki/TestPyramid.html">Read more… </a>

<img src="https://martinfowler.com/articles/practical-test-pyramid/title.png"/> The "Test Pyramid" is a metaphor that tells us to group software tests into buckets of different granularity. It also gives an idea of how many tests we should have in each of these groups. Although the concept of the Test Pyramid has been around for a while, teams still struggle to put it into practice properly. This article revisits the original concept of the Test Pyramid and shows how you can put this into practice. It shows which kinds of tests you should be looking for in the different levels of the pyramid and gives practical examples on how these can be implemented.

Testing Strategies in a Microservice Architecture

<img src="https://martinfowler.com/articles/microservice-testing/meta-image.png"/> There has been a shift in service based architectures over the last few years towards smaller, more focussed "micro" services. There are many benefits with this approach such as the ability to independently deploy, scale and maintain each component and parallelize development across multiple teams. However, once these additional network partitions have been introduced, the testing strategies that applied for monolithic in process applications need to be reconsidered. Here, we plan to discuss a number of approaches for managing the additional testing complexity of multiple independently deployable components as well as how to have tests and the application remain correct despite having multiple teams each acting as guardians for different services.

<img src="https://martinfowler.com/bliki/images/unitTest/sketch.png"/> Unit testing is often talked about in software development, and is a term that I've been familiar with during my whole time writing programs. Like most software development terminology, however, it's very ill-defined, and I see confusion can often occur when people think that it's more tightly defined than it actually is.

<img src=""/> A broad-stack test is a test that exercises most of the parts of a large application. It's often referred to as an end-to-end test or full-stack test . It lies in contrast to a <a href="/bliki/ComponentTest.html">ComponentTest </a>, which only exercises a well-defined part of a system. by Martin Fowler 22 Apr 2013 <a href="/bliki/BroadStackTest.html">Read more… </a> bliki

<img src="https://martinfowler.com/bliki/images/integrationTesting/sketch.png"/> Integration tests determine if independently developed units of software work correctly when they are connected to each other. The term has become blurred even by the diffuse standards of the software industry, so I've been wary of using it in my writing. In particular, many people assume integration tests are necessarily broad in scope, while they can be more effectively done with a narrower scope.

<img src=""/> Story tests are <a href="/bliki/BusinessFacingTest.html">BusinessFacingTests </a>used to describe and verify the software delivered as part of a <a href="/bliki/UserStory.html">UserStory </a>. When a story is elaborated the team creates several story tests that act as acceptance criteria for the story. The story tests can be combined into a regression suite for the software and provide traceability from the requirements (user stories) to tests and (through execution) to the behavior of the system. Story tests are usually <a href="/bliki/BroadStackTest.html">BroadStackTests </a>.

<img src="https://martinfowler.com/bliki/images/contractTest/sketch.png"/> One of the most common cases of using a <a href="/bliki/TestDouble.html">TestDouble </a>is when you are communicating with an external service. Typically such services are being maintained by a different team, they may be subject to slow, and unreliable networks, and maybe unreliable themselves. That's why a test double is handy, it stops your own tests from being slow and unreliable. But testing against a double always raises the question of whether the double is indeed an accurate representation of the external service, and what happens if the external service changes its contract?

<img src=""/> A business-facing test is a test that's intended to be used as an aid to communicating with the non-programming members of a development team such as customers, users, business analysts and the like. When automated, they describe the system in domain-oriented terms, ignoring the component architecture of the system itself. Business-facing tests are often used as acceptance criteria, having such tests pass indicates the system provides the functionality that the customer expects.

<img src=""/> I use subcutaneous test to mean a test that operates just under the UI of an application. This is particulary valuable when doing functional testing of an application: when you want to test end-to-end behavior, but it's difficult to test through the UI itself. by Martin Fowler 14 Feb 2011 <a href="/bliki/SubcutaneousTest.html">Read more… </a>

<img src="https://martinfowler.com/bliki/images/thresholdTest/sketch.png"/> A threshold test is a test inserted into a <a href="/bliki/DeploymentPipeline.html">DeploymentPipeline </a>that monitors some measurable phenomenon by comparing the value in the current build against a threshold value. Should the current build's value pass the threshold, the test fails, failing the build. by Martin Fowler 12 Sep 2013 <a href="/bliki/ThresholdTest.html">Read more… </a>

<img src=""/> Gerard Meszaros is <a href="/books/meszaros.html">working on a book </a>to capture patterns for using the various <a href="/bliki/Xunit.html">Xunit </a>frameworks. One of the awkward things he's run into is the various names for stubs, mocks, fakes, dummies, and other things that people use to stub out parts of a system for testing. To deal with this he's come up with his own vocabulary which I think is worth spreading further.

<img src=""/> The term 'Mock Objects' has become a popular one to describe special case objects that mimic real objects for testing. Most language environments now have frameworks that make it easy to create mock objects. What's often not realized, however, is that mock objects are but one form of special case test object, one that enables a different style of testing. In this article I'll explain how mock objects work, how they encourage testing based on behavior verification, and how the community around them uses them to develop a different style of testing.

<img src="/books/meszaros.jpg"/> This century has seen a significant rise in programmer testing, where programmers take the responsibility for writing tests of their own code. We've found this helps reduce bugs, makes it easier to think about API design, and enables Refactoring. A large part of making this happen is the xUnit family of simple testing frameworks, originating in the work of Kent Beck and Erich Gamma. But although xUnit is a simple tool, it takes experience and skill to use it effectively. Gerard Meszaros was an early adopter of these tools and of the idea of using patterns to help people learn about software design techniques. It provides a vocabulary to help us communicate about how we use tests, and a foundation for experienced practitioners to pass their knowledge onto a new generation.

Goto Fail, Heartbleed, and Unit Testing Culture

<img src=""/> Two computer security flaws were discovered in early 2014: Apple’s “goto fail” bug and OpenSSL’s “Heartbleed” bug. Both had the potential for widespread and severe security failures, the full extent of which we may never know. Given their severity, it is important for the software development profession to reflect on how they could have been detected so we can improve our ability to prevent these kinds of defects in the future. This article considers the role unit testing could play, showing how unit tests, and more importantly a unit testing culture, could have identified these particular bugs. It goes on to look at the costs and benefits of such a culture and describes how such a culture was instilled at Google.

<img src=""/> An automated regression suite can play a vital role on a software project, valuable both for reducing defects in production and essential for evolutionary design. In talking with development teams I've often heard about the problem of non-deterministic tests - tests that sometimes pass and sometimes fail. Left uncontrolled, non-deterministic tests can completely destroy the value of an automated regression suite. In this article I outline how to deal with non-deterministic tests. Initially quarantine helps to reduce their damage to other tests, but you still have to fix them soon. Therefore I discuss treatments for the common causes for non-determinism: lack of isolation, asynchronous behavior, remote services, time, and resource leaks.

<img src=""/> As my career has turned into full-time authorship, I often worry about distancing myself from the realities of day-to-day software development. I've seen other well-known figures lose contact with reality, and I fear the same fate. My greatest source of resistance to this is ThoughtWorks, which acts as a regular dose of reality to keep my feet on the ground.

ThoughtWorks also acts as a source of ideas from the field, and I enjoy writing about useful things that my colleagues have discovered and developed. Usually these are helpful ideas, that I hope that some of my readers will be able to use. My topic today isn't such a pleasant topic. It's a problem and one that we don't have an answer for.

<img src="https://martinfowler.com/bliki/images/testCoverage/card.png"/> From time to time I hear people asking what value of test coverage (also called code coverage) they should aim for, or stating their coverage levels with pride. Such statements miss the point. Test coverage is a useful tool for finding untested parts of a codebase. Test coverage is of little use as a numeric statement of how good your tests are.

<img src="https://martinfowler.com/articles/rise-test-impact-analysis/prod_to_tests_map.png"/> Test Impact Analysis (TIA) is a modern way of speeding up the test automation phase of a build. It works by analyzing the call-graph of the source code to work out which tests should be run after a change to production code. Microsoft has done some extensive work on this approach, but it's also possible for development teams to implement something useful quite cheaply.

David Heinemeier Hansson, the creator of Ruby on Rails, gave a keynote at RailsConf where he declared that TDD is Dead. This caused a predictably large amount of controversy in both the Rails and wider software development community. It also led to some interesting conversations between David, Kent, and myself. We decided that these conversations were interesting enough that others might like to watch them too, so recorded a series of video hangouts where we discuss the role of TDD in software development.

<img src="https://martinfowler.com/bliki/images/pageObject/pageObject.png"/> When you write tests against a web page, you need to refer to elements within that web page in order to click links and determine what's displayed. However, if you write tests that manipulate the HTML elements directly your tests will be brittle to changes in the UI. A page object wraps an HTML page, or fragment, with an application-specific API, allowing you to manipulate page elements without digging around in the HTML.

<img src=""/> If you need to get the current date or time in your code, don't access the system routines for that data directly. Put some form of wrapper around it that allows you to override it by setting the "current date/time" to a particular value. This is important to simplify testing. by Martin Fowler

Although I advocate automated self-tests as a core part of an effective development effort - such tests cannot do all you need to do in testing or quality assurance. Automated tests provide a fine bug catching net, but you need exploratory testing to figure out if the net really covers all you need it to. While testing used to be something you should complete before code goes into production, we now see monitoring and observability as serious tools to determine the health of our running software.

<img src="https://martinfowler.com/articles/qa-in-production/qa-in-prod.png"/> Traditionally, QA focuses on testing the software before release into production to see if it's ready for such release. But increasingly, modern QA organizations are also focusing attention onto the software running in production. By analyzing logs and other monitoring tools, they find quality problems to highlight to the development organization. This approach works particularly well with organizations that use continuous delivery to put new versions of the software into production rapidly and reliably.

<img src="https://martinfowler.com/bliki/images/exploratoryTesting/sketch.png"/> Exploratory testing is a style of testing that emphasizes a rapid cycle of learning, test design, and test execution. Rather than trying to verify that the software conforms to a pre-written test script, exploratory testing explores the characteristics of the software, raising discoveries that will then be classified as reasonable behavior or failures.

<img src="https://martinfowler.com/articles/domain-oriented-observability/twitter-card.png"/> Observability in our software systems has always been valuable and has become even more so in this era of cloud and microservices. However, the observability we add to our systems tends to be rather low level and technical in nature, and too often it seems to require littering our codebase with crufty, verbose calls to various logging, instrumentation, and analytics frameworks. This article describes a pattern that cleans up this mess and allows us to add business-relevant observability in a clean, testable way.

<img src="https://martinfowler.com/bliki/images/syntheticMonitoring/sketch.png"/> Synthetic monitoring (also called semantic monitoring ) runs a subset of an application's automated tests against the live production system on a regular basis. The results are pushed into the monitoring service, which triggers alerts in case of failures. This technique combines automated testing with monitoring in order to detect failing business requirements in production.

Testing Guide

Gigel Șmecherie's ideas are part of this journey:

Related collections

Similar ideas