I have a development manager who wants us to mock almost every single component our software change interacts with because he insists that we are suppose to completely isolate our unit test from other components, which is what he calls a unit test.
We keep telling him that is not how to write unit tests, but he is the manager, and so we've been doing it this way.
For example, I agree with the following web page's "Reasons for use" of mocks.
Does anyone have the argument that supports the extreme position our manager is taking? Or, a good book or paper that argues against this extreme?
From what I've been seeing, it has created poorer quality software because we are actually not finding bugs that use our real databases and external web services until very late in the process. We just had a feature freeze date, and we found a few major problems that day largely because everyone was using mocks, and the mocked systems did not behave the way our mocks did, and the higher level functional tests created by QA missed a few things that a normal unit test would of caught had we not been using mocks for everything.
In article <9c9e542a-7573-4c37-9956-a0f457e6a...@q40g2000prh.googlegroups.com>, Tom Whittaker <tomwhittake...@yahoo.com> wrote:
> I have a development manager who wants us to mock almost every single > component our software change interacts with because he insists that > we are suppose to completely isolate our unit test from other > components, which is what he calls a unit test.
I agree with him so far. A unit test should test a single component in isolation from all other components.
> Does anyone have the argument that supports the extreme position our > manager is taking? Or, a good book or paper that argues against this > extreme?
People who are into XP tend to be extreme :-) What he's saying is not wrong, per-se, but it's also not the full picture.
It is useful to have true unit tests, which test components in isolation. But, it's also essential (as you have discovered) to have tests which test how the components interact. It sounds like your project is relying too much on unit tests and doesn't have enough integration (system, end-to-end, functional, whatever you want to call them) tests.
> From what I've been seeing, it has created poorer quality software > because we are actually not finding bugs that use our real databases > and external web services until very late in the process.
When dealing with external components like databases, it is important to test against the real thing. If you test only against mocks, you risk your code not working in the real world because:
* You have bugs in your mock (although a true mock should have such a simple implementation that this is unlikely).
* Your understanding of how the external database is supposed to behave was faulty.
* The database itself doesn't behave as it's supposed to behave (i.e. it has bugs). Testing against the real thing is the only way you find these problems.
So, your manager is right: real unit tests need mocks. And you're right too: relying entirely on mocks will lead to problems. It's like the beer commercial, "Less filling, tastes great!".
> I have a development manager who wants us to mock almost every single > component our software change interacts with because he insists that > we are suppose to completely isolate our unit test from other > components, which is what he calls a unit test.
> We keep telling him that is not how to write unit tests, but he is the > manager, and so we've been doing it this way.
> For example, I agree with the following web page's "Reasons for use" > of mocks.
> Does anyone have the argument that supports the extreme position our > manager is taking? Or, a good book or paper that argues against this > extreme?
> From what I've been seeing, it has created poorer quality software > because we are actually not finding bugs that use our real databases > and external web services until very late in the process. We just had > a feature freeze date, and we found a few major problems that day > largely because everyone was using mocks, and the mocked systems did > not behave the way our mocks did, and the higher level functional > tests created by QA missed a few things that a normal unit test would > of caught had we not been using mocks for everything.
Since you're posting in an XP group, I assume you're using Test Driven Development, possibly full XP.
I will point out that several leading XP proponents, including Kent Beck (who invented XP) and Ron Jeffries, do not use mocks except for isolating external interfaces.
Whether or not to use mocks extensively is a --very-- contentious issue, with the lines drawn rather firmly. I'm in the 'no, I don't use mocks' camp. I feel that they tend to force too much up front design and avoid facing the issue of continuous integration squarely.
As far as your manager is concerned, I suspect that he's confused by the term 'test' in Test Driven Development. It means "development using tests as a driver", not 'testing already written components.'
HTH
John Roth Yahoo XP mailing list moderator (NOT this list's moderator - it's unmoderated.)
> On Aug 2, 2:57 pm, Tom Whittaker <tomwhittake...@yahoo.com> wrote:
> > I have a development manager who wants us to mock almost every single > > component our software change interacts with because he insists that > > we are suppose to completely isolate our unit test from other > > components, which is what he calls a unit test.
> > We keep telling him that is not how to write unit tests, but he is the > > manager, and so we've been doing it this way.
> > For example, I agree with the following web page's "Reasons for use" > > of mocks.
> > Does anyone have the argument that supports the extreme position our > > manager is taking? Or, a good book or paper that argues against this > > extreme?
> > From what I've been seeing, it has created poorer quality software > > because we are actually not finding bugs that use our real databases > > and external web services until very late in the process. We just had > > a feature freeze date, and we found a few major problems that day > > largely because everyone was using mocks, and the mocked systems did > > not behave the way our mocks did, and the higher level functional > > tests created by QA missed a few things that a normal unit test would > > of caught had we not been using mocks for everything.
> Since you're posting in an XP group, I assume you're using Test > Driven Development, possibly full XP.
> I will point out that several leading XP proponents, including Kent > Beck (who invented XP) and Ron Jeffries, do not use mocks except > for isolating external interfaces.
> Whether or not to use mocks extensively is a --very-- contentious > issue, with the lines drawn rather firmly. I'm in the 'no, I don't use > mocks' camp. I feel that they tend to force too much up front design > and avoid facing the issue of continuous integration squarely.
> As far as your manager is concerned, I suspect that he's confused > by the term 'test' in Test Driven Development. It means "development > using tests as a driver", not 'testing already written components.'
> HTH
> John Roth > Yahoo XP mailing list moderator > (NOT this list's moderator - it's unmoderated.)
In looking at your message again, what occurs to me is that you may not be doing sufficient testing. By that I don't mean writing tests!
A fairly typical test setup in an XP shop might be to integrate and run the test suite on the developers system(s) every few minutes, trading off fidelity for speed (eg, an in-core data base), then run the test suite on each integration (you are integrating all work at least twice a day, right?) with an intermediate tradeoff of fidelity for speed, and run the full test suite overnight, with full fidelity except where it's absolutely necessary to compromise for stability (eg, an external e-mail server).
By full fidelity I mean a real data base configured the same way as production. Also the test suite should go all the way through packaging and installation on a test system. This latter avoids problems that are only discovered during deployment
On Aug 2, 4:57 pm, Tom Whittaker <tomwhittake...@yahoo.com> wrote: <snip>
> From what I've been seeing, it has created poorer quality software > because we are actually not finding bugs that use our real databases > and external web services until very late in the process.
<snip>
Hi. I think your problem is that you need to iterate from unit test (with Mocks) to integration test (without Mocks) as a daily part of your process. Sounds like you are only unit testing until late in the game.
Note that mocks or stubs or fakes can have subtly different meanings. Bottom line, though, is that you need them to unit test a system of any complexity.
For unit testing, I hope you are using the xUnit framework, e.g., JUnit, CppUnitLite, DBUnit, etc. That enables you to automate test execution and validation. Without that, unit testing does have nearly as much return on investment.
Regarding books, my favorite books on unit testing are "Test-Driven Development: A Practical Guide" and Michael Feather's "Working Effectively with Legacy Code." I've also just started reading "xUnit Test Patterns," and that is quite good. I suspect any of those would have answers to your questions.
On Aug 4, 9:11 am, John Roth <johnro...@gmail.com> wrote: <snip>
> As far as your manager is concerned, I suspect that he's confused > by the term 'test' in Test Driven Development. It means "development > using tests as a driver", not 'testing already written components.'
I disagree with this. The notion of refactoring legacy code fits quite nicely into TDD. The way you do that is when some new requirement comes in, any code you have to touch that is not already under test--gets put under test. Over time, you'll end up touching all your code, so eventually it'll all get under test. Feather's Working Effectively with Legacy Code is all about that.
> On Aug 4, 9:11 am, John Roth <johnro...@gmail.com> wrote: > <snip>
> > As far as your manager is concerned, I suspect that he's confused > > by the term 'test' in Test Driven Development. It means "development > > using tests as a driver", not 'testing already written components.'
> I disagree with this. The notion of refactoring legacy code fits > quite nicely into TDD. The way you do that is when some new > requirement comes in, any code you have to touch that is not already > under test--gets put under test. Over time, you'll end up touching > all your code, so eventually it'll all get under test. Feather's > Working Effectively with Legacy Code is all about that.
I may have not made myself clear. I was attempting a distinction between "classical testing theory" and "test driven development." What I was trying to say has nothing to do with refactoring.
BTW - Feather's book should be on every developer's shelf.
|
