My work buddy and I were talking about tests. We wanted something to go through every route in our system and check for any obvious errors. That seemed like a pretty simple way to test an application.
So I wrote a generator to create tests for me automatically at work. I wanted to test every route we have in the system. We have hundreds of routes on this particular project I’m working on today. On our last large project we had over a thousand routes. What can I say? We like our routes.
The test to see if a route works is pretty simple and it looks something like this.
public function test_route()
So after running the test code generator I was left with about 300 tests for this project. Some failed, some succeed, others flat out gave me fatal errors and crashed phpunit. This was because we had changed database columns and it totally broke some sections of our the site. After fixing a couple of errors, I ran into a bigger issue. Out of Memory
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 41 bytes) in ../vendor/laravel/framework/src/Illuminate/Database/Connection.php on line 323
Basically, out of the box Laravel integration testing is a memory explosion waiting to happen. Testing will work so long as you have a little application. Don’t be alarmed though, I get emails all the time about pills we can take to increase size. Jokes aside, if you want to test each one of your routes, eventually when you have enough routes, you will run into this same memory problem I ran into.
There’s a quick fix to that problem. Just allow more memory, right?
However, that just traded one problem for another. After running some really slow tests, after about 150 tests I start getting a continious stream of error exceptions.
Well crap. What to do about this problem? My tests are erroring because the framework has created too many database connections. My guess is that Laravel stores it’s entire
$app globally and there may be some places where mysql database connections are not closed. This is causing memory leakage and eventually a bunch of PDOException: SQLSTATE  Too many connections errors.
One way to get around this too many connections problem is to not run so many tests all at once. We can limit our runs to only a subset of tests using phpunit suites. Here is a gist of someone who has done it this way.
However, I decided to tackle the problem head on. The real problem is too many database connections. Thus we will manually close the connections ourself by overwriting the default
TestCase::tearDown method as seen next.
class TestCase extends Illuminate\Foundation\Testing\TestCase
Now when we run our tests, we don’t get a bunch of PDOException errors due to the too many connections. That is because we are closing them. Alright! Should we stop here? Well, we could. And if you’re not that familar with Laravel’s insides, then you might do well to stop here. However, I still have a few bones to pick with these tests.
- They are pretty dang slow.
- They use way to much memory.
After some research I found that other people had ran into this memory issue before too. There is a nice closed github ticket which hasn’t really solved anything to date (even though the ticket dates over 2 years ago). Granted, we got around our memory problem by setting memory_limit to -1 but I want my tests to run faster and use less memory. It seemed to me that the obvious place to first tackle is the
public function createApplication()
Notice that everytime a new test is ran, Laravel is re-including and bootstrapping the application. However, we discovered earlier that the PHP garbage collection is not removing stuff under the hood of Laravel. There is a ton of stuff behind the curtain of Laravel. Some of which could remain, even after the test is completed. This means, the more tests you create, the more memory is used. I could get through about 20-30 tests before my php started to exploded due to being out of memory.
So in case you still don’t quite understand the issue. Let’s run a bench mark test. I’ll use the
ini_set('memory_limit', -1); hack I mentioned at the beginning of this article. It uses over 450Mb of memory.
class TestCase extends Illuminate\Foundation\Testing\TestCase
Now with my changes in place, let’s see how things run. Will we actually be able to complete without the framework blowing up and throwing error exceptions on over half our tests because it used up too many database connections?
YAY! We got a lot of failures and some skipped tests, but no weird ass errors. This looks promising. We will try this for a while and see how it turns out. We’ve ran over 600+ tests using a similar method to this on our Devise CMS tests. Hopefully, this will work well.
Also, take a look at the memory and time now. It is down to 27.96 seconds and 267.25Mb. That’s a bit better.
There is some merit to the way Taylor setup the
TestCase. So let’s try to understand why Taylor is resetting the application every time. If we understand that, then we’ll understand the dangers of the code I just showed you above to get around the memory problems.
The application is reset because Taylor wants us to test in isolation. When we create a new application, we are creating a new IoC container. There are variables stored in this IoC container. If those variables are tampered with because of our test, then it could affect every other test. If you don’t reset those everytime, we run the risk of our application container not being isolated and thus causing certain tests to mess with each other. For example, take these two tests,
class TestMe extends TestCase
Above we don’t actually have a ‘something-here’ route defined. We are mocking it with Mockery on the URL facade. I personally don’t use Mockery on Laravel’s facades often because I don’t use facades that often. However, this will show you the problem.
Above we are changing the URL facade to expect the
route method. In our second
test_me2() method we are just asserting that an exception is thrown because we don’t have a route defined for something-here.
This test should run successfully on the old out of the box TestCase.
However, when we change to our modified TestCase we run into a different situation.
Now our results show that mockery has been applied across both testing methods `test_me1’ and ‘testme_2’. See below.
See how now we no longer get an exception in
test_me2 because the mock we applied in
test_me1 bled over. This is why testing in isolation is a good thing. How do we deal with this? Here are some simple steps.
Don’t bootstrap Laravel on unit tests. Only on integration tests. We use Mockery a lot on unit tests. Unit tests however only require a single class to be loaded. Not an entire framework. If you are using Mockery in unit tests, then you should even be extending
createApplication()Feel free to make different classes such as,
Don’t mock things on integration tests.
Don’t try to unit test your models. If you really want to see your models working, write some integration tests on classes which heavily use the models. If you use a
Repositorythen often you can just make calls to your repository and ensure that the models are indeed working.
Use a real database and database transactions to rollback. This will save a lot of time compared to seeding your database over and over.
Use different phpunit testing suites for different types of tests.