On Sat, Sep 17, 2005 at 01:49:39PM +0100, Ben Fowler wrote:
Yes, but we have end users using Inkscape for their regular work.
The reality is distributions are shipping Inkscape and users are expecting a level of stability more than developers expect. There is also no way distributions can keep up with the fast releases coming from inkscape as they are stuck on a 6 month/yearly cycle.
If this is a major problem, then we switch to an even/odd system and urge packagers to only pass downstream the even ones, or we do once or twice a year a stable release specifically for packagers, This might be too much work until after 1.0, by which time the problems will have changed anyway!
I've thought a lot about this and paid close attention to how the project works from release to release since we started. I've noticed several things.
As a general rule, I've noticed that each release is much more stable than the one that went before, even the releases that are very development-heavy. This makes sense; whenever bugs are found in the previous release, the bug fixes go in for the next release.
So far, there hasn't been a lot of developer activity in backporting fixes. The 0.42.2 release was the first that really did this aggressively, although we've backported fixes in the past. I'm quite interested to see how things turn out following the 0.43 release. Will people take interest in backporting fixes and creating a 0.43.x series? Will we pour all of our focus into an extra-stable 0.44?
Most of the destabilization that occurs in new releases is localized to areas where new features are added. This makes sense; people aren't just randomly going through and fiddling with code for no reason. And for most of the feature work, the changes rarely break more than the specific functionality they're working on. This is good; this means that when a new feature comes out, if it brings new bugs along, those bugs are mostly only going to break that feature, so you could simply report the issue and then ignore that feature until the next release.
However, in practice I notice that new features really get early adopter users excited and they get really involved in playing with it and trying it out. They break it in all sorts of ways and talk directly to the developer, who often has a fix out within days. Thus, usually by the time a feature gets out into a release, it's been through a heavy "play testing" period and actually has a very high degree of quality. If there are remaining issues, it's usually due to lack of time to complete it rather than lack of review. This aspect of being able to team users, testers, and developers together in such a tightnit feedback loop is Inkscape's secret weapon and I think it's the reason we've been able to advance so aggressively. This is a killer competitive advantage open source has over commercial companies; I bet if we keep doing what we're doing, within a year or two Inkscape may start really making some serious inroads against the commercial apps.
Furthermore, while intuition would suggest that it is new features that bring the new bug reports, in reality I see just as many if not more bug reports arising due to use of Inkscape in new environments. For example, we got a whole nest of internationalization bugs once it started getting used by people in other countries. We've also gotten tons of bugs from users of operating systems, hardware platforms, or distros we've never tested on before. People also find unusual ways to configure Inkscape - e.g., with different input devices, with oddball sets of old or new libraries, etc. All these are legitimate bugs, but none are due to feature work; we could put out a release with no new features and see just as many of these bug reports.
I am not sure that users want to be managed, but the 'ear'y adopters' should be treated as full partners so far as directing the goes.
You're exactly right. In fact, I make a practice of when a new user expresses a need for some feature or other, if no one else is working on them, to ask, "Do you code?" If they say yes, I immediately invite them to work on that feature. I've been fairly astounded how often this actually works. I'd expect that 99% of the time the person wouldn't have the time, knowledge, or interest, but in reality I find that 25-50% of the users I ask actually react positively to the invitation to join. Even the ones that don't code will often put in time helping out in various non-coding areas.
People are already treating Inkscape like it is a stable product even if it is in fact a fast moving alpha/beta stage project. Something should be done to accomodate or compensate for that perception.
My suggestion is to accommodate it by having at all times a stable branch and/or occasionally having bug fix only releases - 'No new features' - to prove that we mean to keep the count of outstanding issues controlled.
I think both of these are excellent suggestions. Of course, like I said above, many of the bug reports we see are due not to new features but new environments. So while I think it would be good practice to have a release 100% focused on bug fixing, I suspect that restricting feature work would not actually have that much effect on the count of outstanding issues. Maybe 5-10% less, but not really enough to make it worth the impact to developers by making them stop working on cool new features. Still, sh*t happens, and you're right that this approach could be an effective way to improve quality.
Management of a stable branch would be best if we had someone dedicated to maintaining that branch. If you'd be interested in doing that, I think the other devs would be open to having you fill that role. If not, I think we'd need to find a volunteer, since most of the current devs are occupied with other efforts and wouldn't have enough time to devote to it. I don't know how much work it'd be to backport fixes and put out stable point releases, but I think you're right that users would get a lot of value out of it.
Now, I've been thinking of these same issues, but from a somewhat different context. Given that so many of our bugs come not from new features but from people just starting to test in new environments (such as on platforms that most devs don't have access to), my thought is that the most effective way to achieve better quality would not be a more rigid release process, but rather a more formalized, aggressive testing process.
I think that by applying formalized testing methods such as regular regression testing, usability testing, interoperability tests, and so forth, we will scare out a lot of these bugs before users have to run into them. As well, by doing this testing routinely, it will help us flag issues that might recur; for instance, issues that cause Inkscape to have trouble on 64-bit hardware, or on OSX, or when run with uncommon language settings.
One of the cool things about this sort of testing is that it's quite ameniable to automation. For instance, you can write a short script that sets the language setting to each of a dozen languages, then run Inkscape in various automated ways, checking if it crashes.
I'm also excited about the potential of building a tester community who can do testing work prior to each release. I think a lot of bugs that get through into a release do so simply because no one was actively looking for them. Developers were busy developing, users were busy using, and the rest were simply focused on watching the bug count drop. By having a group of a few people doing some organized banging on the app, we can have greater assurance that when the release comes out, that there will be fewer bugs.
This is why I suggest that from a developer's point of view, we either need to be able to slipstream older bug fixes (my thaw policy as outlined above) or have protected releases when we only do bug fixes. Obviously the latter might be ruled out if you felt it was anathema, unnecessary pre-1.0 or not possible owing to the large number of enhancements going through.
I'm not going to get into specifics but I think this discussion reveals a problem we are aware of on some level and needs to be managed.
This is where I expect Bryce to step in with some wise words, to crystalize the situation and suggest a workable plan to help manage it.
Hopefully the above dissertation fulfils this role. ;-)
You can probably see I was thinking along the same lines as all of this myself; I'd proposed a few ideas like bugfix only releases, etc. in the past, to gauge their viability, but came to the conclusion that a distinct, organized testing effort would be the most effective way of addressing these issues. That's why I proposed and set up the inkscape-tester community. I could really use all of your help in pushing that forward and solidifying it. We need a lot of good ideas to help get that group up and running. We need to find how to make it fun and exciting so people want to participate in it. And we need good tools and processes that make it convenient to find and report what we find in our analysis.
I would absolutely love it if we could start a regular inkscape-tester process of doing aggressive pre-release testing around the time of the hard freeze. I think this would fill a major needed gap in our release process, and it could be a lot of fun!
A chunk of the 'Roadmap' page on the wiki seems to be missing.
See http://wiki.inkscape.org/cgi-bin/wiki.pl?CreatingDists the Feature Freeze section.
Could you stick in questions on those pages? Then myself or someone can flesh those things out when we run across them.
Thanks, Bryce