Re: [Inkscape-devel] W3C SVG test suite and trunk
On Wed, 2013-10-09 at 11:28 +0200, Guiu Rocafort wrote:
2013/10/9 Tavmjong Bah <tavmjong@...8...>
On Wed, 2013-10-09 at 00:51 +0200, Guiu Rocafort wrote:
Hi ! I've been looking at the GSOC and after solving a few problems I can run the tests now. Now I have to find out a way to update the tests.
I was able to run the tests without too much trouble... however I did not have a set of "good" PNGs to test against... so I gave up and went back to using the latest SVG 1.1 test suite.
To update the tests, I think you just need to put the new SVG files into the directory with the existing SVG files and then create reference PNGs in the appropriate files (known good rendering, known bad rendering). You can't just use the reference PNG files from the SVG test suite for doing automated pixel-by-pixel comparison as the results depend on the font used. (The SVG2 test suite will use a WOFF font, I think, to avoid this problem.) The existing files from the SVG test suite should be replaced by the files from the latest version of the SVG 1.1 2nd Ed test suite.
I see, I wasn't aware of that problems with the fonts. Is the problem related to the use of the SVGFreeSans font ? I saw this font in some tests so I'm guessing it's being used in all of them. I noticed this font is not in my system, so installing it in the machine is going to perform the tests should solve the problem ?
No. A search through the SVG files shows all kinds of different fonts being used. SVGFreeSans is required for some tests as is a number of other specially constructed fonts.
But since we're maybe going to use that ppa automated thing to perform the tests that probably is not an option. Would it be safe to replace all the fonts in the test by one of these safe fonts ? [1]
No. If you look carefully you'll see that each proposed font-family is actually a list of fonts. For example:
"Times New Roman", Times, serif.
If the first font "Times New Roman" isn't found, then Times is search for. If that is not found then the generic fallback font serif is used. While "Times New Roman" and Times are similar, they are not exactly the same. For the use on the web they are probably acceptable equivalents but for pixel by pixel comparison, they wont work.
The solution is to generate the PNG reference images on the same machine that the tests are going to be run on. Then the same font will be used for both the reference and the test images.
Tavmjong, i saw you're running some tests in [2], so, how are you solving these font problems ?
[1] http://www.w3schools.com/cssref/css_websafe_fonts.asp [2] http://tavmjong.free.fr/INKSCAPE/W3C_SVG_0.49/htmlInkscapeHarness/index.html
I look at each test manually and just filter out the differences in font. This is OK for a once in a while check but not for automated testing. Also, a few of the text tests really do need special fonts (found in the 'resources' directory).
I've set a personal branch in my launchpad account with the changes I made to the original code. https://code.launchpad.net/~neandertalspeople/+junk/inkscape-testsuite
After updating the test I'll try to find a way to automate the tests, and perform them regulaly. What do you think about that ? Any idea about how that could be done ? Maybe it is possible to set some kind of trigger when there is a commit in the launchpad inkscape trunk ?
This would be awesome.
Tav
Guiu
On 2013-10-09 11:59, Tavmjong Bah wrote:
On Wed, 2013-10-09 at 11:28 +0200, Guiu Rocafort wrote:
2013/10/9 Tavmjong Bah <tavmjong@...8...>
I see, I wasn't aware of that problems with the fonts. Is the problem related to the use of the SVGFreeSans font ? I saw this font in some tests so I'm guessing it's being used in all of them. I noticed this font is not in my system, so installing it in the machine is going to perform the tests should solve the problem ?
No. A search through the SVG files shows all kinds of different fonts being used. SVGFreeSans is required for some tests as is a number of other specially constructed fonts.
That's why I removed text from the tests: https://svn.code.sf.net/p/inkscape/code/gsoc-testsuite/tester/testcases/svgt... The whole test suite for running rendering tests is available at: https://svn.code.sf.net/p/inkscape/code/gsoc-testsuite/tester/ Note that this test suite allows explicit specification of reference images, and was made to especially detect regressions (so whenever something changes you need to decide whether or not this is a good change). It is best used with a "fuzzy" comparator, like described here: http://wiki.inkscape.org/wiki/index.php/Testing_Inkscape#Running_rendering_t...
Of course removing all text from the tests has its own issues, but the real problem is that even without text there turned out to be too many "small" changes that visibly (but not necessarily unacceptable) affect (Inkscape's) rendering, so a lot of upkeep is required to maintain the tests. Having a clearer definition (and an accompanying tool) of what an "acceptable" rendering is would help a lot, but it is quite difficult. For example, it might be acceptable to have a slight shift in the middle of a Bézier curve, but not at the endpoints, nor would you like a curve that is too thick or thin. Also, there is not really a generally agreed upon "best" way to antialias vector graphics, etc., etc.
In short: whatever method of testing you use, ultimately the best you can really hope for in terms of automation is something (very) forgiving, or something that simply compares to previous "known good" renders. The above improves a bit upon a basic implementation of the latter scheme by comparing both to good and bad renders when possible, and by allowing you to use a custom comparator. But in the end it still required quite a bit of upkeep.
participants (2)
-
Jasper van de Gronde -
Tavmjong Bah