"make check" doesn't run unit tests
This wiki page: http://www.inkscape.org/wiki/index.php/TestingInkscape says that "make check" runs the unit tests written in the CxxTest framework on Unix systems, but in fact it doesn't. The only tests which are actually run are the ones using the old homegrown utest framework residing in src/utest. I'm no expert on Inkscape's build process so I'm not sure how to fix this.
Regards, Krzysztof Kosiński
On Dec 6, 2008, at 5:27 PM, Krzysztof Kosiński wrote:
This wiki page: http://www.inkscape.org/wiki/index.php/TestingInkscape says that "make check" runs the unit tests written in the CxxTest framework on Unix systems, but in fact it doesn't. The only tests which are actually run are the ones using the old homegrown utest framework residing in src/utest. I'm no expert on Inkscape's build process so I'm not sure how to fix this.
Are you sure?
Last time I checked, only a few weeks ago, both sets of tests were being run.
You just might be looking for the wrong type of output.
On Dec 6, 2008, at 6:03 PM, Jon A. Cruz wrote:
On Dec 6, 2008, at 5:27 PM, Krzysztof Kosiński wrote:
This wiki page: http://www.inkscape.org/wiki/index.php/TestingInkscape says that "make check" runs the unit tests written in the CxxTest framework on Unix systems, but in fact it doesn't. The only tests which are actually run are the ones using the old homegrown utest framework residing in src/utest. I'm no expert on Inkscape's build process so I'm not sure how to fix this.
Are you sure?
Last time I checked, only a few weeks ago, both sets of tests were being run.
You just might be looking for the wrong type of output.
Uh oh, Looks like recent 2geomification broke the tests. We'd better get on that.
Jon A. Cruz wrote:
Are you sure? Last time I checked, only a few weeks ago, both sets of tests were being run.
I checked again, they indeed are run - the test I was looking for wasn't appearing in the output because of a runtime error (caused by a typo), and the output of CxxTest tests isn't too appealing as there's no indication of passed tests. I have corrected this now. What remains to be done is making all tests compile again (I just suppressed the compilation of the offending tests).
By the way, attribute-test.h spews out a lot of errors, it may be worth looking at.
Regards, Krzysztof Kosiński
On Dec 7, 2008, at 12:56 PM, Krzysztof Kosiński wrote:
I checked again, they indeed are run - the test I was looking for wasn't appearing in the output because of a runtime error (caused by a typo), and the output of CxxTest tests isn't too appealing as there's no indication of passed tests. I have corrected this now. What remains to be done is making all tests compile again (I just suppressed the compilation of the offending tests).
Yes, that was by design. In the normal course of things nothing but a dot for a test and a summary per suite/program should show up. Only when things fail do you get warnings.
If you want more detailed info, it should be creating test output files in two or three different formats, including one "standard" format XML.
By the way, attribute-test.h spews out a lot of errors, it may be worth looking at.
Yes, that's one of the tests that pointed out some actual problems in our approach. Some things might need to be refactored, as certain data does not round-trip.
I have managed to fix the SVG tests, the only roadblock to successful make check compilation now is the XML test. There's a circular dependency: preferences.o -> libspxml.a -> libspsvg.a -> preferences.o. I tried linking with $(all_libs) but it fails. I wonder why does this work with the SVG test and when compiling the main executable.
Regards, Krzysztof Kosiński
Krzysztof Kosiński wrote:
I have managed to fix the SVG tests, the only roadblock to successful make check compilation now is the XML test. There's a circular dependency: preferences.o -> libspxml.a -> libspsvg.a -> preferences.o. I tried linking with $(all_libs) but it fails. I wonder why does this work with the SVG test and when compiling the main executable.
I'd just like to mention that you (and Jon) totally rock! Keep up the good work.
Unfortunately I don't know why this circular dependency is suddenly a problem (or perhaps it didn't exist before?), it worked over the summer... (And I'm not exactly an expert when it comes to building on Linux.)
Krzysztof Kosiński wrote:
I have managed to fix the SVG tests,...
I've just removed the sp_main_* stubs from the individual headers and put them in a separate file src/test-stubs.cpp to make the unit tests build again on Windows. If someone could add it in the right places in the Makefiles that would be great.
I have sanitized the build rules for "make check" and removed the obsolete tests based on utest. This will greatly simplify adding tests to new directories (there are now 4 simple rules to add to the directory's makefile and one rule to modify in Makefile.am). However, on Windows all CxxTests are built as a single executable. I would like to implement the same on Unix - we wouldn't several log files, but just one in the main build directory. This would be more convenient. What's your opinion?
Regards, Krzysztof Kosiński
Krzysztof Kosiński wrote:
on Windows all CxxTests are built as a single executable. I would like to implement the same on Unix - we wouldn't several log files, but just one in the main build directory. This would be more convenient.
Update: I have a commit-ready version of this, and the number of modifications needed to add an unit test to a directory which doesn't have any dropped from 5 in 2 files to 1 in 1 file. Additionally, there is only one log file to check after running the tests, and there's no need for a stub implementation of preferences to reside in test-stubs.cpp (it would only contain sp_main_gui and sp_main_console until those functions are removed).
Regards, Krzysztof Kosiński
At the moment there are reasons to keep things separate...
... however with CxxTest, it's simple to have things individually *and* combined. We might have some of that already set up.
On Dec 8, 2008, at 5:41 PM, Krzysztof Kosiński wrote:
Krzysztof Kosiński wrote:
on Windows all CxxTests are built as a single executable. I would like to implement the same on Unix - we wouldn't several log files, but just one in the main build directory. This would be more convenient.
Update: I have a commit-ready version of this, and the number of modifications needed to add an unit test to a directory which doesn't have any dropped from 5 in 2 files to 1 in 1 file. Additionally, there is only one log file to check after running the tests, and there's no need for a stub implementation of preferences to reside in test-stubs.cpp (it would only contain sp_main_gui and sp_main_console until those functions are removed).
Regards, Krzysztof Kosiński
View this message in context: http://www.nabble.com/%22make-check% 22-doesn%27t-run-unit-tests-tp20876856p20907203.html Sent from the Inkscape - Dev mailing list archive at Nabble.com.
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http:// 2009.visitmix.com/ _______________________________________________ Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Jon A. Cruz wrote:
At the moment there are reasons to keep things separate...
... however with CxxTest, it's simple to have things individually *and* combined. We might have some of that already set up.
OK, but what are those reasons to keep CxxTests separate? As I see it, this has a number of disadvantages: 1. Lots of error-prone makefile boilerplate needed 2. Test logs are scattered all over the build directory 3. Building and linking the tests takes more time 4. Each test has to specify which libraries it needs 5. We need the test-stubs.cpp file with a stub preferences implementation
The only advantages I see are: 1. If some test crashes, others can run (but then it's an obvious error either in the testand should be corrected right away) 2. The number of failed tests shown after "make check" is slightly more meaningful 3. Higher chance of conflicts between test headers (I encountered only one and corrected it already)
My single-test implementation uses a variable named CXXTEST_TESTSUITES, to which all test headers are appended in Makefile_inserts. The only test-related thing left in Makefile_inserts is the list of test headers. I think it's more convenient and simpler than the exisitng system.
Regards, Krzysztof Kosiński
On Dec 8, 2008, at 6:14 PM, Krzysztof Kosiński wrote:
Jon A. Cruz wrote:
At the moment there are reasons to keep things separate...
... however with CxxTest, it's simple to have things individually *and* combined. We might have some of that already set up.
OK, but what are those reasons to keep CxxTests separate? As I see it, this has a number of disadvantages:
- Lots of error-prone makefile boilerplate needed
- Test logs are scattered all over the build directory
- Building and linking the tests takes more time
- Each test has to specify which libraries it needs
- We need the test-stubs.cpp file with a stub preferences
implementation
The only advantages I see are:
- If some test crashes, others can run (but then it's an obvious
error either in the testand should be corrected right away) 2. The number of failed tests shown after "make check" is slightly more meaningful 3. Higher chance of conflicts between test headers (I encountered only one and corrected it already)
My single-test implementation uses a variable named CXXTEST_TESTSUITES, to which all test headers are appended in Makefile_inserts. The only test-related thing left in Makefile_inserts is the list of test headers. I think it's more convenient and simpler than the exisitng system.
Disadvantage 1: only when a new subdirectory is created. Happens so rarely as to not be a strong factor.
Disadvantage 2: logs can be directed to specific locations as needed. Not so strong an issue. (mainly the current behavior is due to nobody needing it to be different, as opposed to an inability to be different)
Disadvantage 3: on my linux box this time is only 8 seconds for an average test run of everything, and 33 seconds if I touch every single test source file. So seems like a non-issue
Disadvantage 5: This sounds like a problem in the recent introduction of the preferences. Should be made simpler.
Advantage 1: not really an advantage. Advantage 2: this number is so off at the moment as to not be a factor. Advantage 3: strongly possible.
Other advantages: 4: Since all tests are in .h files, they do not need to be explicitly added to the Makefile_instert, since they could be picked up due to placement in the resulting .cpp
5: an individual developer is likelier to only be working in a subset of the code. When doing so, they can work within a directory subtree and have test results for just that directory subtree. (this was one of the main reasons things were split to begin with)
6: certain tests can be run with dedicated mock object. (This has not yet been leveraged).
And most importantly, overall the most "error prone" operations are only done when adding a new test source file (*-test.h). That happens very rarely, with the addition of tests to an existing test suite source is far more common.
Jon A. Cruz wrote:
Disadvantage 1: only when a new subdirectory is created. Happens so rarely as to not be a strong factor.
Actually, this needs to be done not when a directory is created, but when tests are added to a directory that doesn't have any. Currently there are tests only in 6 directories. The single-test-executable solution just ensures that adding a test to a directory that doesn't have any unit tests yet is equally simple as adding one to a directory that has some - it's much simpler to maintain, and eliminates the need to maintain a list of test log files to be removed during "make distclean". It's also consistent with the Windows build system which uses a single test executable.
Jon A. Cruz wrote:
Disadvantage 2: logs can be directed to specific locations as needed. Not so strong an issue. (mainly the current behavior is due to nobody needing it to be different, as opposed to an inability to be different)
OK, so how to redirect all logs to a single file?
Jon A. Cruz wrote:
Disadvantage 5: This sounds like a problem in the recent introduction of the preferences. Should be made simpler.
A stub implementation of preferences also existed before I did the rewrite, the only difference was that it required less code.
Advantage 1: not really an advantage. Advantage 2: this number is so off at the moment as to not be a factor. Advantage 3: strongly possible.
The risk of conflicts can be mitigated if tests are kept contained in their respective objects, e.g. any objects the test needs are members of the test suite object, rather than global.
Jon A. Cruz wrote:
4: Since all tests are in .h files, they do not need to be explicitly added to the Makefile_instert, since they could be picked up due to placement in the resulting .cpp
I don't really understand what do you mean by this - the tests have to be listed in Makefiles regardless of the approach, because they have to be passed as arguments to cxxtestgen.pl. Wildcarding cannot be used because Automake doesn't support wildcards or pattern rules.
Jon A. Cruz wrote:
5: an individual developer is likelier to only be working in a subset of the code. When doing so, they can work within a directory subtree and have test results for just that directory subtree.
Your response to disadvantage 3 indicates that running a subset of the tests won't gain us much time, and running the whole test suite ensures that no unforeseen side effects were introduced. Also, you can't run "make check" in a subdirectory, you'd have to run the tests manually after building them.
My original disadvantage 3 doesn't really hold because on multi-core machines it may be faster to compile the tests in parallel (make -j). Either way, the differences would be minimal.
Regards, Krzysztof Kosiński
When it comes to Unicode, Jon is the expert, and I must agree with him. When a module reads one character out of GIO it must be at least 26 bits.
UTF8 is insufficient. If GIO does not consider a single atomic character to be at least 26 bits, then it is broken.
bob
Bob Jamison-2 wrote:
When it comes to Unicode, Jon is the expert, and I must agree with him. When a module reads one character out of GIO it must be at least 26 bits.
UTF8 is insufficient. If GIO does not consider a single atomic character to be at least 26 bits, then it is broken.
Er, isn't this the wrong thread? By the way, GIO works on byte streams and has no concept of a "character", so if there's a need to read a file character by character I'll just derive a custom GInputStream that reads whole UTF-8 characters only.
Regards, Krzysztof Kosiński
participants (4)
-
Bob Jamison -
Jasper van de Gronde -
Jon A. Cruz -
Krzysztof Kosiński