On Wed, Nov 26, 2014 at 07:54:04AM +0100, Christoffer Holmstedt wrote:
2014-11-26 1:27 GMT+01:00 Bryce Harrington <bryce@...961...>:
On Tue, Nov 25, 2014 at 11:38:45PM +0100, Johan Engelen wrote:
[...]
To help with refactoring: reinvigorate unittests + rendertests. This is already started at jenkins.inkscape.org:8080, but we should have a larger effort in improving our testsuite (possibly move to other test framework, but at the very least move all unittests into a dedicated tree structure, separate from the src tree);
I've been involved in projects that keep unit tests in trunk, as well as a few where they lived in separate repositories. Universally the latter projects' test suites got ignored and quickly fell out of date. Indeed there got to be kind of a discrimination against test writing as a chore. Consequently the testsuites for those projects atrophied; and the more they atrophied the less seriously they got taken; and the less importance people placed on improving them.
Then there have been projects that not only kept the tests in trunk, but made them *the* priority when developing features. Launchpad was very gung ho on this. Cairo is too, to a lesser extent. The test suite actually ends up taking up an appreciable proportion of the codebase, and you may find yourself putting as much effort into writing the tests as on the feature itself (and maybe even more!) But the result is that when you make any arbitrary change to the codebase, and the testsuite still passes, you feel a very strong sense of confidence that your change is right and hasn't broken anything.
Anyway, this is all to say I feel strongly based on my experience that we should keep all tests that we want to actually work and be useful, IN-tree. And in fact in code reviews one of the first things we should check for is whether new or updated tests are being included.
I assumed that Johan meant "in the same repository but in another directory with proper structure". Moving tests into another repository sounds really weird, never heard of it before.
Ah, whoops my bad, sorry!
and also in extending and improving our rendertestsuite; and possibly also create a UI testsuite or something of the kind (or a cmdline testsuite). In general: TESTING.
+1
UI testing would be really interesting to dig into. I know there are tools for it for web interfaces but haven't seen any for desktop UI. Though I see this as a last step in the testing area.
There are indeed a variety of UI testing frameworks including at least a couple specific to gtk applications for testing the widgetry. I'm not sure how valuable tests will be - "Widget UI" usually either works or it doesn't most of the time, and it gets reworked often which risks tests bitrotting - but if done judiciously could well prove to be a boon.
For "On Canvas UI" you can test a) behavior, and b) rendering. Behavioral testing might be a bit beyond our capabilities right now, but rendering testing is doable by comparing the rendered screen against reference images. Cairo has this style of test suite, which I've mucked around with and am in process of implementing something similar for Wayland. Since this UI is in-house developed we may get more bang for our buck writing tests for this, but I'm not certain.
Having more comprehensive tests, /and checking them every day/ for early discovery, I think will greatly reduce release-cycle times. Less bugs need to be squashed and hopefully less nasty block bugs too. It is pretty rewarding too to fix a bug measurably by creating a test, and be sure it will not ever fail again without us knowing.
Perhaps this does not sound as much fun as creating a completely new feature. But I think it can be, and I think it is necessary.
I wonder what we can do as a project to stimulate more test writing activity. More visible posting of test writing activity might help. Maybe posting howto's and tips & tricks on writing good tests.
Bryce
The "only" thing missing for me is the workflow and automatic running of tests. My experience is mostly with Git, Github, Bitbucket, Gitlab, Bamboo CI, Travis CI and my own instances of Jenkins. This is purely my own preference and I think with more experience with Bazaar/Launchpad it would work good-enough as well but overall, launchpad seems outdated as a useful "tool".
Johan has been experimenting with Jenkins and automatic running of tests so if this is an area you're interested in, definitely touch base with him. I hope we can expand in this area significantly in the coming releases.
Launchpad is indeed a bit outdated, and unfortunately no longer actively maintained. I don't think we're at a part where we're ready to mass migrate off of it, but I do think we'll probably avoid increasing dependence on it in favor of looking at other services and tools.
Our culture here favors open source including for infrastructural services, as much as is feasible. One thing Launchpad does have going for it is that it *is* (technically) Open Source. But I don't see us on Launchpad in 10 years.
Bryce