Friday, January 16, 2009

RSpec not unstubbing?

I've been really struggling lately with a testing situation. I get different results for a couple of tests when I run the following commands:

spec spec/models/document.rb
rake spec

I verified that in both tests, the Document model was including the same modules and that I was using the same RSpec source to execute the tests (before rake spec was running code from my plugins directory and spec spec/... was running code from the gem repository. I have configured both to be using the same source code now.

Research on the Internet has revealed that others have also found that RSpec doesn't always seem to unstub methods as it should. Every 'it' block should stub out methods (using before do's) and unstub methods so that unit tests are not order dependent.

I am trying to test that when save! is called, certain callback methods are invoked given certain circumstances. Running rake spec shows 2 failures while executing the file with the failed tests alone passes. It seems that something is being set in another test that is persisting over the course of many tests. If spec spec/... is fed all the model tests, the two test cases in question fail as in the rake spec execution. There is a -R option that can be passed to spec to reverse the order of the tests. When that option is taken, all the model tests pass. Another thing that I've found to be true is that the order of the tests is random. I'm not sure what exactly determines the randomness, but small changes to an existing file or the addition of new files seems to change the RSpec test execution order. Bottom line, currently, you cannot depend on the RSpec tests being run in any given order.

When I finally get down to the bottom of things, I will run tests with -R and without to make sure that I have eliminated my problem.

Currently, I've defined blocks for all possible callback methods. I want to see where the callback gets short-circuited, hoping to get some more insight into what's happening.

1 comment:

Unknown said...

Please let me know when you find out more, I lately had some problems but they seem to have vanished by themselves.

The nice thing about rspec is to be more in control of what you do, but when you find out that some tests aren't actually testing anything useful, that's annoying.