Hey there, welcome to Agile in a Nutshell I’m your host Jonathan Rasmusson. One framework that is very handy for teams when they first get into automated testing is the testing pyramid.
The testing pyramid is great because helps get teams on the same page around how they’d like tackle automated testing for their project.
It can help by
In this episode we are going to look at:
Let’s get started.
So here we have a humble login page. How would you go about writing an automated test for this.
Well if you take the point-of-view of an end user, you’d probably like to see a test that:
Which is exactly what the User Interface Test does.
User Interface, or UI tests, are tests that test the system just like a real live end user would. They mimic the user’s interactions in the form or a script, we run that script in the form or a test, and it basically interacts with the system just like a regular user would.
For in this example here, written in RSpec, we visit the login page, fill in the user’s name and password, and then click the Sign in button just like a real user would.
It’s a beautiful, simple test, that tests the complete system end-to-end. That’s the UI test.
Now we could write another kind of test, very similar to the UI test, only this one wouldn’t go through the user interface. Instead it would call the underlying services that back it up. We call this, the integration test.
Integrations tests, like UI tests, slice through various layers of the application, but in this case they don’t go through the UI. Here, in the case of this web application, the integration tests mimics the HTTP calls the browser would normally be making on behalf of the user as they were logging.
So, for example to visit the login page, an integration test would send an HTTP GET request to the web server with the address of the login page it wants to navigate to.
get login_path
Then it send the username and password, the user would have normally entered, with something called an HTTP POST which would include the user’s credentials
post login_path, session: { email: ‘user@test.com’, ‘password: ‘password’ }
It could then follow the successful login to the welcome a some welcome page, where it could then check and see that that it ended up in the right place. For example by looking an HTML header with the word ‘Welcome’.
Similar test to the UI test. Just testing the same functionality in a different way.
Now what about all the other things we could test when it comes to login pages. Things like field length, required fields, and checking for duplicate email addresses. We could write all those as UI and integration tests. But low level details like these are usually better served with faster, more precise type of test. Something we call a unit test.
Unit tests are tests developers write to prove to themselves, that the code they are writing works. In enables them to make changes, without fear of breaking everything. And they are really what enables teams to make changes to code aggressively.
So instead of going end-to-end through the various layers of the application like a UI or integration test would, unit test are much more local. These are low method levels tests developers write on objects. And they usually don’t hit other parts of the system.
Now the interesting question, is out of these three types of tests, how to do we when to choose one kind of test over the others?
So which is right?
You see we could have written all of the tests we just covered at each layer of the pyramid. They all would have worked. And teams that are new to automated testing sometimes go about doing that. Without understanding the trade-offs each kind of test has, they blindly duplicate these tests at each level of the pyramid. That leads to higher maintenance costs. Not to mention lots duplicated effort and waste.
This is where the testing pyramid comes in. It’s a way to talk about the various trade-offs we need to sometimes make between different kinds of tests, and gives us some rules of thumb around where and we to use. Take UI tests for example.
UI tests are fantastic and going end-to-end through the entire application, and ensuring that everything is hooked up and working. They are the deepest, most integrated kind of test we can write. And if an UI tests works, chances are very good everything is going to work in production.
But this awesomeness comes with a price. UI tests are very slow. It takes orders of magnitude longer to run a UI test than a unit test. Which means once we start to collect a lot of these things, our build times start to get long, our ability to iterate really slows down.
They are always very fragile. Changing the user interface often breaks the corresponding UI tests. So we have to be careful about how we write these things, and try really hard not to make them overly fragile.
So because of their speed and fragility, we tend to use UI tests more sparingly, and save them end-to-end style smoke tests.
Integration tests like UI tests also test connectivity. But because they don’t go through the UI, there are no where near as slow or fragile. These things are great and testing things like web services, and giving some level of confidence that things are things hooked up, without paying an exorbitant price.
And while they are good at telling us when we have a problem, they unfortunately they can’t really tell us precisely where. As it could be anywhere in the system.
For low level precision, that tell us exactly where things went wrong, we rely on the unit test.
Unit tests are the fastest, most productive form of automated test we’ve got. These tests are fast, efficient, highly precise, and can pinpoint exactly where something fails when a test breaks. They give really rapid feedback. Are inexpensive. And developers write hundreds of these things all day long with testing their applications.
They’re only downside is that unit tests do periodically miss bugs that can sometimes occur only when you hook things up. Which is why integration and UI tests are still valuable. But for the bulk of our automated testing efforts, we should be relying on the unit test.
And when you bring these three forms of test together, and you try to balance some of these trade-offs that we just talked about, that’s when you begin to appreciate the testing pyramid.
The testing pyramid is a model that has become popular over the years because it visually captures some of the trade-offs we face when writing automated tests.
We may want to write everything as a UI test. But that would hurt, as our feedback and ability to iterate quickly would really slow down. So we tend to use UI tests more sparingly.
We know however that we are going to have integration bugs, so we do need some kind of test that enables us to test that various things are connected and hooked up. That’s why we have integration tests.
But usually, the fastest and cheapest test we can write is the unit test. Which is why we like to do the bulk of our automated testing down there. Where the tests are fast and cheap.
So that’s it! That’s the testing pyramid.
To learn more about the testing pyramid, and how to write really good UI, integration, and unit tests test, checkout my new book, The Way of the Web Tester---A beginners guide to the world of automated testing.
There you will not only learn how to write great automated tests, you will learn how the web works, some basics around programming, as well as how to organize your tests so they don’t collapse on you when you most need them.
Alright. That’s all for now. See you next time. Happy testing!