Bryce Harrington wrote:
... Actually the original intent is more about gaining a suite of useful tests, than setting up the test framework. We already have most of the infrastructure for running tests via cxxtest and make check; certainly it could be better/fancier, but that's far secondary to having more valuable tests in the first place. (Having worked on several test frameworks myself, I know how much more attractive they are to work on than "just" tests, but the ultimate goal is finding and fixing bugs, and for that we need tests.)
Great, thanks for the feedback. For me the main thing is to make sure the tests are actually used (obviously this goes hand in hand with actually *having* tests). That is, I'd like to make sure the tests are run often and systematically. Personally I don't think developers will run them themselves very often (I know how difficult it can be to get people to change their habits), so by running them as often as possible (preferably after each commit) on all three supported platforms I hope any future regressions might be picked up more easily. And with a bit of luck it will also increase the visibility of the tests.
Having said that, that is also pretty much all I want to do in the testing *framework* department, I'm not looking to create a fancy web interface with all sorts of nice features. If I can get away with a system which regularly reruns tests and uploads a few xml files, and perhaps sends a message to a mailing list if it encounters a regression, I'd be quite happy.
So, given that the summer moves along fast, I would encourage you to focus on #1 and #4, which are the things you could add the most unique value. The others are of course worthwhile, but other folks have set those up in the past and I expect could again in the future; so far the limiting factor has been our scarcity of tests, which reduce the usefulness of frameworks to begin with. So if you focused particularly on #1 and #4, I think it might stimulate the rest to fall into place, and would make the most valuable use of your time.
I'll definitely focus on those first, but I do intent to also incorporate higher level tests, like the W3C conformance tests and tests for verbs.
livarot would probably not be worth the while to instrument with tests as it's scheduled for removal anyway.
I was wondering about that. I know the plan is to use Cairo for rendering, but for what parts exactly, and on what time scale? For example, would nr-compose also become redundant? And if the current renderer will probably remain to be used for one or two years it may still be useful to make sure it is in good shape.
With testing, there are some general rules of what things are best to make tests for:...
Hopefully those heuristics can be used to prioritize which sections of code need tests written the most.
I had been thinking about this too, and I'll definitely have a look at things like this. But initially I thought I'd try to more or less close some "gaps" and more or less try to cover the code involved in converting SVG to a bitmap. I like systematic approaches :)
The types of tests needed, as Bulia and others mentioned, include both unit tests (via cxxtest), and high level functional tests (such as via the command line verbs). The former is more suited for running via make check; the latter could be hooked into make check but probably would be more useful to be a stand alone suite of test scripts.
Yes, indeed. It might also have different output. I could imagine it might be useful to have a more quantitative result for example. Instead of saying a test failed or passed it would simply report the similarity measure it uses for example, so people could more easily evaluate the effect of changes that only change the precision of the renderer.