On Dec 8, 2008, at 6:14 PM, Krzysztof Kosiński wrote:
Jon A. Cruz wrote:
At the moment there are reasons to keep things separate...
... however with CxxTest, it's simple to have things individually *and* combined. We might have some of that already set up.
OK, but what are those reasons to keep CxxTests separate? As I see it, this has a number of disadvantages:
- Lots of error-prone makefile boilerplate needed
- Test logs are scattered all over the build directory
- Building and linking the tests takes more time
- Each test has to specify which libraries it needs
- We need the test-stubs.cpp file with a stub preferences
implementation
The only advantages I see are:
- If some test crashes, others can run (but then it's an obvious
error either in the testand should be corrected right away) 2. The number of failed tests shown after "make check" is slightly more meaningful 3. Higher chance of conflicts between test headers (I encountered only one and corrected it already)
My single-test implementation uses a variable named CXXTEST_TESTSUITES, to which all test headers are appended in Makefile_inserts. The only test-related thing left in Makefile_inserts is the list of test headers. I think it's more convenient and simpler than the exisitng system.
Disadvantage 1: only when a new subdirectory is created. Happens so rarely as to not be a strong factor.
Disadvantage 2: logs can be directed to specific locations as needed. Not so strong an issue. (mainly the current behavior is due to nobody needing it to be different, as opposed to an inability to be different)
Disadvantage 3: on my linux box this time is only 8 seconds for an average test run of everything, and 33 seconds if I touch every single test source file. So seems like a non-issue
Disadvantage 5: This sounds like a problem in the recent introduction of the preferences. Should be made simpler.
Advantage 1: not really an advantage. Advantage 2: this number is so off at the moment as to not be a factor. Advantage 3: strongly possible.
Other advantages: 4: Since all tests are in .h files, they do not need to be explicitly added to the Makefile_instert, since they could be picked up due to placement in the resulting .cpp
5: an individual developer is likelier to only be working in a subset of the code. When doing so, they can work within a directory subtree and have test results for just that directory subtree. (this was one of the main reasons things were split to begin with)
6: certain tests can be run with dedicated mock object. (This has not yet been leveraged).
And most importantly, overall the most "error prone" operations are only done when adding a new test source file (*-test.h). That happens very rarely, with the addition of tests to an existing test suite source is far more common.