More than two months of rendering tests
I've run the rendering tests for a bit more than two months now, usually roughly once a week. The results have actually been relatively stable, considering all the refactoring work going on. However, there have been a few notable changes.
The good news: the number of passed tests has gone from 96 to 105 and the number of failed tests from 65 to 56.
The bad news: we have three tests that crash at the moment (introduced between 2008-12-28 and 2009-01-04), and some of the changes in the number of passed tests were due to a rejudging of some the filter outputs (basically there appear to be at least one or two wrong references in the W3C test suite, and Inkscape currently simply doesn't handle the color-interpolation properties, while the actual filters are fine).
You can get more details at: http://home.hccnet.nl/th.v.d.gronde/inkscape/ResultViewer.html
I intend to continue running the tests as before in the near future, but am still most definitely interested in any additional help in further automating these tests and/or writing the tests. Specifically: - I currently run these tests by starting a batch file on my notebook, basically when I feel like it. A machine which could run this using a timer would be great (it would have to compile inkscape, run the tests and upload the result, possibly generating an e-mail if there are any regressions). - I currently judge any new results (so results which don't match any existing reference). Usually this is fine (most of the time there are only about 1-3 new results), but any help would be appreciated. Of course this would be a lot easier if the machine running the tests could also serve the output images (then a small on-line application could be made for letting people judge the new results). - I'd like to change the comparison algorithm used. Currently I'm using perceptualdiff, but I would like to use something which simply allows for a given error threshold (maximum error and/or rms error). Obviously extra bonus points for an algorithm that can do something sensible for comparing curves (the SVG standard says something about curves having to be within one pixel of the "true" curve or something). - Any additional tests (based on bug reports for example) would obviously be welcome. - I'd love to be able to link a bug report to a certain fail result (linking it to a file is probably not a good idea, as one test may exhibit different bugs over time). Any ideas and/or implementations for doing this would be great. Note that currently the system cannot distinguish between two fail references, so some easy way of specifying for which outputs the bug holds would be very nice.
On Sun, Jan 11, 2009 at 9:47 AM, Jasper van de Gronde <th.v.d.gronde@...528...> wrote:
The bad news: we have three tests that crash at the moment (introduced between 2008-12-28 and 2009-01-04)
I tried to reproduce the crash of https://bugs.launchpad.net/inkscape/+bug/181995 but could not - it loads fine for me and exports to png from command line without problems. Can you also include in the table the exact command line you use, so these crashes could be easily reproduced?
- I currently run these tests by starting a batch file on my notebook,
basically when I feel like it. A machine which could run this using a timer would be great (it would have to compile inkscape, run the tests and upload the result, possibly generating an e-mail if there are any regressions).
Absolutely support this. That's the best use we can put our server to. Who could help with setting this up?
- Any additional tests (based on bug reports for example) would
obviously be welcome.
I will certainly contribute some of the most demanding and bug-prone SVGs from my collection, and I urge others to do the same.
- I'd love to be able to link a bug report to a certain fail result
(linking it to a file is probably not a good idea, as one test may exhibit different bugs over time).
Right now I see links to both in the table.
bulia byak wrote:
On Sun, Jan 11, 2009 at 9:47 AM, Jasper van de Gronde <th.v.d.gronde@...528...> wrote:
The bad news: we have three tests that crash at the moment (introduced between 2008-12-28 and 2009-01-04)
I tried to reproduce the crash of https://bugs.launchpad.net/inkscape/+bug/181995 but could not - it loads fine for me and exports to png from command line without problems. Can you also include in the table the exact command line you use, so these crashes could be easily reproduced?
I just use the runtests.py from SVN, it's not a manually run test. But at the end of this mail I included the relevant part of the executionlog (including the exact command line used).
However, I clearly have been too busy with other things for a while (and have been using the command line too much), as currently Inkscape crashes for me as soon as I start it! (I made a clean build today, and do so for each test run.) I'll try to have a closer look tomorrow to see what the problem is. "Hopefully" it's just a local problem.
- Any additional tests (based on bug reports for example) would
obviously be welcome.
I will certainly contribute some of the most demanding and bug-prone SVGs from my collection, and I urge others to do the same.
Sounds good :) Note that for references you can either include one or more pngs (for failure and/or pass cases) and/or a "patch" file, which is a simpler SVG file that should yield the exact same output. This can even be useful when you know that in practice it won't produce the exact same output, to help judge the results (for example, for feComposite I did this).
- I'd love to be able to link a bug report to a certain fail result
(linking it to a file is probably not a good idea, as one test may exhibit different bugs over time).
Right now I see links to both in the table.
A few files are based on specific bug reports, and they link to the bug report so you can easily see what it's about, but especially for other files it might make more sense to link specific failures to bugs. (Also, at the moment I don't even have a way to link arbitrary files to bugs, as the links are simply generated based on the test name, so even doing that could be an improvement.)
----------- Begin Quote ----------- Executing: tester "inkscape -e "E:\Inkscape\gsoc-testsuite\tester.\output\bugs\bug181995.png" "E:\Inkscape\gsoc-testsuite\tester.\testcases\bugs\bug181995.svg"" "E:\Inkscape\gsoc-testsuite\tester.\output\bugs\bug181995.png" -compare perceptualdiff.exe -pass "E:\Inkscape\gsoc-testsuite\tester.\references\pass\bugs\bug181995-1.png" -pass "E:\Inkscape\gsoc-testsuite\tester.\references\pass\bugs\bug181995-2.png" -fail "E:\Inkscape\gsoc-testsuite\tester.\references\fail\bugs\bug181995.png" ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Background RRGGBBAA: ffffff00 Area 0:0:300:150 exported to 300 x 150 pixels (90 dpi) Bitmap saved as: E:\Inkscape\gsoc-testsuite\tester.\output\bugs\bug181995.png ** Message: Error: Er is een interne fout opgetreden in Inkscape. Het programma wordt afgesloten.
RegistryTool: Could not set the value 'e:\inkscape\trunk\inkscape\inkscape.exe'
Emergency save activated! Emergency save completed. Inkscape will close now. If you can reproduce this crash, please file a bug at www.inkscape.org with a detailed description of the steps leading to the crash, so we can fix it. ----------- End Quote -----------
On 01/11/2009 08:46 PM, Jasper van de Gronde wrote:
... as currently Inkscape crashes for me as soon as I start it! (I made a clean build today, and do so for each test run.) I'll try to have a closer look tomorrow to see what the problem is. "Hopefully" it's just a local problem.
Might be similar to this bug:
https://bugs.launchpad.net/inkscape/+bug/315089
Diederik
Diederik van Lierop wrote:
On 01/11/2009 08:46 PM, Jasper van de Gronde wrote:
... as currently Inkscape crashes for me as soon as I start it! (I made a clean build today, and do so for each test run.) I'll try to have a closer look tomorrow to see what the problem is. "Hopefully" it's just a local problem.
Might be similar to this bug:
I've had a look, and it definitely might be. A backtrace doesn't seem to help me much, probably because it apparently happens after exiting the idle loop for the first time. Perhaps using glib symbols would help (anyone know how to do this?). I've appended the output from a gdb run, augmented with some messages showing entering/exiting of some initialization related functions (and the idle function).
Also noteworthy seems to be the message "Error: dll starting at ... not found". I have no idea what dll this is supposed to be and whether or not it is connected to the problem, but it seems fishy.
----------------------------------------------------------- Starting program: E:\Inkscape\trunk\inkscape/inkscape.exe [New thread 6956.0x1e00] RegistryTool: Could not set the value 'E:\Inkscape\trunk\inkscape\inkscape.exe' [New thread 6956.0x1c78] Error: dll starting at 0x5630000 not found. [New thread 6956.0x13ec] ** Message: Entering sp_file_new_default ** Message: Starting to leave sp_file_new_default 1 ** Message: Entering sp_file_new ** Message: Leaving sp_file_new ** Message: Connecting idle ** Message: Running main instance ** Message: Entering idle loop ** Message: Leaving idle loop warning: HEAP[inkscape.exe]: warning: Invalid address specified to RtlFreeHeap( 012F0000, 097AFB58 )
Program received signal SIGTRAP, Trace/breakpoint trap. 0x77a17dff in ntdll!DbgUiConvertStateChangeStructure () from C:\Windows\system32\ntdll.dll (gdb) bt #0 0x77a17dff in ntdll!DbgUiConvertStateChangeStructure () from C:\Windows\system32\ntdll.dll #1 0x77a81c93 in ntdll!RtlpNtMakeTemporaryKey () from C:\Windows\system32\ntdll.dll #2 0x77a52b26 in ntdll!RtlTimeToTimeFields () from C:\Windows\system32\ntdll.dll #3 0x77a82cf3 in ntdll!RtlpNtMakeTemporaryKey () from C:\Windows\system32\ntdll.dll #4 0x77a4bfe8 in ntdll!RtlTimeToTimeFields () from C:\Windows\system32\ntdll.dll #5 0x77a38652 in ntdll!RtlUnhandledExceptionFilter () from C:\Windows\system32\ntdll.dll #6 0x7742c56f in KERNEL32!HeapLock () from C:\Windows\system32\kernel32.dll #7 0x772e9d6b in msvcrt!free () from C:\Windows\system32\msvcrt.dll #8 0x012f0000 in ?? () #9 0x00000000 in ?? ()
bulia byak wrote:
On Sun, Jan 11, 2009 at 9:47 AM, Jasper van de Gronde <th.v.d.gronde@...528...> wrote:
- I currently run these tests by starting a batch file on my notebook,
basically when I feel like it. A machine which could run this using a timer would be great (it would have to compile inkscape, run the tests and upload the result, possibly generating an e-mail if there are any regressions).
Absolutely support this. That's the best use we can put our server to. Who could help with setting this up?
I had asked Ian (inc) on IRC about this recently as he's our point person regarding the web server. He has to run it by the OSU-OSL staff to get approval if we're to do it. I will follow-up to see if he's had any word back on it.
Cheers, Josh
participants (4)
-
bulia byak -
Diederik van Lierop -
Jasper van de Gronde -
Josh Andler