2014-11-26 1:27 GMT+01:00 Bryce Harrington <bryce@...961...>:
On Tue, Nov 25, 2014 at 11:38:45PM +0100, Johan Engelen wrote:
[...]
To help with refactoring: reinvigorate unittests + rendertests. This is already started at jenkins.inkscape.org:8080, but we should have a larger effort in improving our testsuite (possibly move to other test framework, but at the very least move all unittests into a dedicated tree structure, separate from the src tree);
I've been involved in projects that keep unit tests in trunk, as well as a few where they lived in separate repositories. Universally the latter projects' test suites got ignored and quickly fell out of date. Indeed there got to be kind of a discrimination against test writing as a chore. Consequently the testsuites for those projects atrophied; and the more they atrophied the less seriously they got taken; and the less importance people placed on improving them.
Then there have been projects that not only kept the tests in trunk, but made them *the* priority when developing features. Launchpad was very gung ho on this. Cairo is too, to a lesser extent. The test suite actually ends up taking up an appreciable proportion of the codebase, and you may find yourself putting as much effort into writing the tests as on the feature itself (and maybe even more!) But the result is that when you make any arbitrary change to the codebase, and the testsuite still passes, you feel a very strong sense of confidence that your change is right and hasn't broken anything.
Anyway, this is all to say I feel strongly based on my experience that we should keep all tests that we want to actually work and be useful, IN-tree. And in fact in code reviews one of the first things we should check for is whether new or updated tests are being included.
I assumed that Johan meant "in the same repository but in another directory with proper structure". Moving tests into another repository sounds really weird, never heard of it before.
and also in extending and improving our rendertestsuite; and possibly also create a UI testsuite or something of the kind (or a cmdline testsuite). In general: TESTING.
+1
UI testing would be really interesting to dig into. I know there are tools for it for web interfaces but haven't seen any for desktop UI. Though I see this as a last step in the testing area.
Having more comprehensive tests, /and checking them every day/ for early discovery, I think will greatly reduce release-cycle times. Less bugs need to be squashed and hopefully less nasty block bugs too. It is pretty rewarding too to fix a bug measurably by creating a test, and be sure it will not ever fail again without us knowing.
Perhaps this does not sound as much fun as creating a completely new feature. But I think it can be, and I think it is necessary.
I wonder what we can do as a project to stimulate more test writing activity. More visible posting of test writing activity might help. Maybe posting howto's and tips & tricks on writing good tests.
Bryce
The "only" thing missing for me is the workflow and automatic running of tests. My experience is mostly with Git, Github, Bitbucket, Gitlab, Bamboo CI, Travis CI and my own instances of Jenkins. This is purely my own preference and I think with more experience with Bazaar/Launchpad it would work good-enough as well but overall, launchpad seems outdated as a useful "tool".
When pushing a feature branch to my own fork in Github and within seconds/minutes (depending on size of project) I know from Travis CI that my code didn't mess things up, and I can now send a pull request to mainline. This is enough for me to dig into refactoring code/writing unit tests. Gerrit is also a pretty good tool when it comes to code review.