One of the important parts of any DevOps process is the regular release of working software. In Scrum, iterations tend to be only one or two weeks long. When you use Kanban you release whenever a reasonable package is ready – often multiple times a week. When you do that, you will inevitably see that manual testing becomes a bottleneck. Always.
If your team is not able to test everything as soon as it is ready, they soon will ask you to introduce one or two week long test cycles. They tell you that in that time they can do all the testing and that you’ll have a stable release afterwards.
Unfortunately, nothing could be further from the truth. The later you test, the more effort you’ve got to spend fixing bugs introduced weeks ago. And as the code is changing during the testing weeks, every test cycle you do has to be repeated. In the end, your software is no more stable then it was before the test cycle.
The only way to support a rapid cadence of releases is to automate testing. Only if you use unit testing during implementation you can be sure that a bug fix or refactoring does not break anything later. And only if you use tools to do integration testing can you afford to skip the manual regression tests.
Fast releases drive manual testing efforts through the roof. Automate the critical parts step by step and you’ll be ready for continuous deployment.
Automation Testing (Acceptance) Strategy
Acceptance tests should be run when your system is in a production-like model. Manual acceptance testing is typically done by putting an application in a user acceptance testing (UAT) environment which is as similar as possible to production – both in configuration and in terms of the state of the application – although it might use mock versions of any external service. The tester uses the application standard user interface in order to perform testing. Automated acceptance tests should similarly be run in a production like environment, which the test harness interacting with the application the same way that a user would.
Automated acceptance tests have a number of valuable properties:
- They make the feedback loop faster – developers can run automated tests to find out if they have completed a particular requirement without having to go to the testers.
- They reduce the workload on testers
- They free testers to concentrate on exploratory testing and higher value activities instead of repetitive tasks.
- Your acceptance tests represent a powerful regression test suite. This is particularly important when writing large applications or working in large teams where frameworks or many modules are being used and changes one part of the application and likely to affect the other features.
- By using human-readable test and test suite names, as advocated by behavior-driven development, it is possible to auto-generate requirements documentation from your tests.
Automated Acceptance tests can be costly to maintain. Done badly, they can inflict a significant cost on your delivery team. For this reason, some people recommend against creating large complex suites of automated tests. However, by following good practices and using appropriate tools, it is possible to dramatically reduce the cost of creating and maintaining automated acceptance to the point where the benefits clearly exceed the costs.
Behaviour Driven Development (BDD)
In software engineering, behavior-driven development (BDD) is a software development process that emerged from test-driven development (TDD). Behavior-driven development combines the general techniques and principles of TDD with ideas from domain-driven design and object-oriented analysis and design to provide software development and management teams with shared tools and a shared process to collaborate on software development.
Although BDD is principally an idea about how software development should be managed by both business interests and technical insight, the practice of BDD does assume the use of specialized software tools to support the development process. Although these tools are often developed specifically for use in BDD projects, they can be seen as specialized forms of the tooling that supports test-driven development. The tools serve to add automation to the ubiquitous language that is a central theme of BDD.
BDD is largely facilitated through the use of a simple domain-specific language (DSL) using natural language constructs (e.g., English-like sentences) that can express the behavior and the expected outcomes. Test scripts have long been a popular application of DSLs with varying degrees of sophistication. BDD is considered an effective technical practice especially when the “problem space” of the business problem to solve is complex.
BDD does not have any formal requirements for exactly how these user stories must be written down, but it does insist that each team using BDD come up with a simple, standardized format for writing down the user stories which includes the elements listed above. A template for a textual format is given below and is used widely in many BDD software tools.
Story: Returns go to stock As a store ownerIn order to keep track of stockI want to add items back to stock when they’re returned.
Scenario 1: Refunded items should be returned to stock
Given that a customer previously bought a black sweater from me
And I have three black sweaters in stock.
When he returns the black sweater for a refund
Then I should have four black sweaters in stock.
Scenario 2: Replaced items should be returned to stock
Given that a customer previously bought a blue garment from me
And I have two blue garments in stock
And three black garments in stock.
When he returns the blue garment for a replacement in black
Then I should have three blue garments in stock
And two black garments in stock.
The scenarios are ideally phrased declaratively rather than imperatively — in the business language, with no reference to elements of the UI through which the interactions take place.
This format is referred to as the Gherkin language, which has a syntax similar to the above example. The term Gherkin, however, is specific to the Cucumber and JBehave software tools