Unit testing with VCR

You have an application that makes HTTP calls to an external service and you have no tests for it. You want to make sure that your application behaves correctly whether those services return correctly or not.

From an architectural point of view, these are two very different components in your system, and they should not be tightly coupled, generally. If the external service fails, your application needs to not fail as well.

Making sure that your application responds well to all possible responses from the service is thus a necessity. How can we ensure this?

• You hard code error prone requests to the service in your application, and you code the necessary error handling routines, for instance, intentionally mistyping an URL param, a wrong host name, etc;
• You create unit and/or integration tests to automate this process;

Option one presents the easiest way. However, this is a one-time testing strategy. This has clear disadvantages: every time you change code (for example, response handling methods), you’re going to have to manually test everything, again. There are a lot of other disadvantages to this approach which, if you take coding serious, you are well aware of. Let’s be smart, and let machines do the grunt work for us.

The obvious question arises: are we supposed to create tests that make HTTP calls every time we run our test suite? Won’t that make the tests slow as hell? What if I’m offline and still want to run tests, would I need connectivity to run my tests? What if my IP gets blocked from making too many requests?

Luckily, there’s a gem to ease this process, the VCR gem. In a nutshell, this gem lets you capture the result of a real HTTP call in your tests, saves the response to a file, and the next time the tests run, it will read the response from that file, if it exists, instead of making the request again, and the tested code can now work with that result.

I like this gem in particular because you won’t have to stub or mock any piece of code that makes service calls explicitly in your tests, making them nicer to read. Let’s see how with an example. I will assume that you already have the basic structure of a project configured to work with rspec.

We need to write a bit of configuration first in our spec_helper.rb file:

Create a file called vcr.rb in your spec/support folder with:

Don’t forget to create the folder spec/vcr , which is where we will put files with real service call responses, used by the tests. VCR also needs a tool to mock HTTP calls behind the scenes (don’t worry, you will still handle real responses):

Finally, let’s install the gems required for vcr to work.

> gem install vcr
> gem install webmock

Now that we have everything we need, let’s go ahead and write a simple application to test this.

Let’s bring back the code we wrote in my last blog post:

In order to test that we can correctly parse the real result from calling this reddit endpoint, we can create a simple test like so:

The important thing here is that we’re wrapping our test in a VCR.use_cassette block. Guess what, you now have a file called correct_result.yml in the folder spec/vcr . Here’s a truncated preview of the content of that file:

What’s so great about this? This request is recorded once and placed in this file. The next time you run your tests, the HTTP call will not be made, and your tests will work with the result in the body string of this yml file. No more slow tests! No more black listed IP’s!

Another advantage about this is that you can test that your code works with unique situations, such as handling wrong request URL’s. Let’s test this situation. First, we need to change the Fetcher class so that it accepts different reddit endpoints in the requests:

Let’s now test that our code works with incorrect request urls (hint: it does not because of line 12 in our fetcher class! Even though that hash access is really naive, it’s just to illustrate the point of when errors can occur when parsing service calls results):

When we run this test, we obviously get the error:

expected Fetcher::FetcherException with message matching /Unexpected response/, got #<NoMethodError: undefined method `[]’ for nil:NilClass> with backtrace:
 # ./lib/fetcher.rb:12:in `get_result’

The file incorrect_results.yml now contains the response of an unexpected call. This is nice because this situation is something that can happen very rarely or sporadically in your application, and you can still test that your app works correctly in those cases! (No cumbersome manual testing, or bug hunting by brute forcing requests with wrong urls).

# incorrect_results.yml# ...body:
  encoding: UTF-8
  string: '{"kind": "Listing", "data": {"modhash": "", "children": [], "after": null, "before": null}}'# ...

Remember that this is a real service call result! Let’s get the tests fixed.

We’re good now! We have tests for both cases: unexpected calls to wrong urls, and correct urls.

Some other nifty things about VCR that I’ve learned:

• If you want to disable VCR for a particular test, you can add VCR.turn_off! in a before block. If you’re using WebMock with VCR (like I am here), you may also to need to use this with WebMock.allow_net_connect! to turn off http stubs made by WebMock .
• IMPORTANT! If you’re using rails, take into account the warning from the vcr relish documentation website regarding the folder used to store your response yml files: do not use ‘test/fixtures’ as the directory if you’re using Rails and minitest (Rails will instead transitively load any files in that directory as models).
• If you’re doing requests in your code that requests zip files, for example, and you want to test the de-serialized response in your tests, you can pass the option decode_compressed_response: true in the method VCR.use_cassette . The yml file generated will contain the contents of the file as strings in the body key.
• If you already have a yml file created from previously executed tests, but you want to replay the call (maybe you changed the code) and update the yml file, you can pass the option record: :new_episodes to the method VCR.use_cassette . Nothing will change in your tests (other than your new expectations of the changed code, of course), but you don’t need to delete the yml file and create another one. You can just use the same. I use this options pretty much every time I write tests with VCR, it’s really handy.
• You can organize your response yml files in folders if you separate the name of the cassette with backslashes. For example, if you write VCR.use_cassete("Some/Folder/correct_response") do ... end , this would create the folders some and folder inside the directory you configured to store your yml files.
• An awesome thing about VCR is that you can use ruby inside your yml files! If, for some reason, something in the response yml can be dynamic, you can pass the option erb: true to the method VCR.use_cassette . This allows you to do stuff like:

# some_cassette.yml# ...- request:
    method: get
    uri: <%= normal ruby code here! %>/some/url# ...

That’s it! Thank you for reading. I hope you find using VCR a bliss to write unit tests on code that performs external requests. Comment bellow if you’ve used VCR in the past, if you liked it, and other nifty things that you’ve discovered about it!

Nowadays I work at Runtime Revolution. Working here has been, and continues to be, a great learning experience. I’ve matured professionally as a developer, focusing on building and maintaining large scale Ruby on Rails applications.