bulia byak wrote:
On Fri, Mar 21, 2008 at 9:19 AM, Jasper van de Gronde <th.v.d.gronde@...528...> wrote:
Unfortunately I don't have a Linux system set up at the moment, so I can't actually execute 'make check'. So far I've come up with the following draft plan:
Great plan, and long overdue! :)
I think the important thing is to test both from bottom up (unit tests of classes and functions) and from top down (scripted tests and performance measurements of the entire program doing various tasks on various files).
It's a good idea to also make use of the command line access to verbs, this would indeed allow for some very interesting testing possibilities. The main problem would be how to test it. I would imagine comparing the SVG result to a reference output would be most interesting (after all, just looking good usually isn't good enough when executing verbs). But to compare SVGs it would be necessary to parse them and write comparison routines for different elements, which might be a bit much to do as part of this project.
Alternatively the resulting bitmaps could be compared, this would be a lot less work (it would be needed for the SVG conformance tests anyway). And it would still capture bugs resulting in visible problems.
For the latter, Ted's command line access to verbs is priceless, but you'll also need to find some kind of bitmap diff program to compare PNG renditions of various files to the reference versions and find differences. This will be especially critical when we switch main rendering to cairo.
One way I am planning to deal with this is letting humans decide whether something is acceptable and caching the result. The underlying assumption is that Inkscape's rendering output probably doesn't change that often (on most of the tests), so it would only require human intervention when something actually changes.
Another method to reduce human intervention I was thinking of was to define areas of the result image to disregard when comparing it. The idea behind this was that a lot of the conformance tests I found contained lots of labels which weren't actually important (for that test case). With a bit of luck the rest of the image is good enough to be compared directly to the reference image (although the image might not be exactly the same the difference will be considerably smaller, and methods like looking at MSE or maximum error will provide reasonable results).
In addition, I just stumbled across: http://pdiff.sourceforge.net/ I still have to try it, but it sounds interesting.
As a last resort it would definitely be feasible for me to program a simple bitmap diff program (one that computes the MSE for example).
Combining these methods I hope to minimize the amount of human intervention needed. If anyone has any other ideas I'd be glad to hear them.
I have a couple thousand misc SVG files from various sources, and before a release I ran a very simple script that would load each file and export it to PNG. Even without any bitmap diffing, this found at least a couple bugs.
I get the feeling there is no need to worry about getting enough test images :)