Extensions and binary distribution
Inkscape, like many open source programs, uses Python as a scripting language for extensions. This is all fine and good, but unfortunately raises serious issues for binary distribution.
A few are obvious:
* Windows users don't have it * Mac users probably have an out of date version * Linux users almost certainly have it, but it's probably got a different ABI/API to what is expected
The first two do not concern me here. The last one does.
The libpython C API varies between systems in an incompatible manner. Python 2.2, 2.3 and 2.4 are all in the wild, and they export different APIs. Worse, Python can be configured to use either UCS2 or UCS4 unicode strings. The upstream default is UCS2 but some distributions configure it to use UCS4 - this changes the exported ABI.
I'm not sure what the solution to this is. Currently we recommend people do add Python for scripting to their applications, but that horse is long since gone for Inkscape. So we have a few alternatives:
1) Disable Python scripting in autopackage build. That'd be a shame. 2) Enable it in such a way that if libpython cannot be loaded, Inkscape still starts and works minus scripting 3) Figure out some new magic tricks so we can be compatible with all of them (relaytool can help here), for instance by avoiding the unstable parts of the API 4) Drop Python and port the scripts to some other system, like Lua or Mono
(3) is the best for the user but requires the most in-depth knowledge of the Python API and versioning history.
Statically linking Python isn't really an option, unfortunately.
thanks -mike
Mike Hearn wrote:
I'm not sure what the solution to this is. Currently we recommend people do add Python for scripting to their applications, but that horse is long since gone for Inkscape.
This is a common misconception. Inkscape doesn't really have Python for scripting. All of the current functionality is just shelled out to the installed system python.
Aaron Spike
On Fri, 25 Nov 2005 11:44:07 -0600, aaron-XW8b5UTxoeLYtjvyW6yDsg wrote:
This is a common misconception. Inkscape doesn't really have Python for scripting. All of the current functionality is just shelled out to the installed system python.
Hmm, are you sure? The configure script checks for various functions in libpython itself like Py_Initialize. Seems odd to check for anything if all it does is run scripts using the command line.
thanks -mike
Mike Hearn wrote:
On Fri, 25 Nov 2005 11:44:07 -0600, aaron-XW8b5UTxoeLYtjvyW6yDsg wrote:
This is a common misconception. Inkscape doesn't really have Python for scripting. All of the current functionality is just shelled out to the installed system python.
Hmm, are you sure? The configure script checks for various functions in libpython itself like Py_Initialize. Seems odd to check for anything if all it does is run scripts using the command line.
thanks -mike
Actually, the lib bundle on win32 does contain complete Perl and Python implementations. Also, the stubs for scripting Inkscape via Perl or Python do actually start interpreters for them. So there is no reason that I can perceive why such code can't start external scripts just as well. Maybe people can experiment with what we would need to add to our Inkscape distro bundle to have running interpreters on Win32 and others. In fact, those stubs can probably be a means of starting such scripts in a cross-platform manner, rather than shelling out.
bob
On Fri, Nov 25, 2005 at 07:40:51PM +0000, Mike Hearn wrote:
On Fri, 25 Nov 2005 11:44:07 -0600, aaron-XW8b5UTxoeLYtjvyW6yDsg wrote:
This is a common misconception. Inkscape doesn't really have Python for scripting. All of the current functionality is just shelled out to the installed system python.
Hmm, are you sure? The configure script checks for various functions in libpython itself like Py_Initialize. Seems odd to check for anything if all it does is run scripts using the command line.
There are actually two different Python scripting systems.
The first is the commandline version Aaron is talking about. This is the one that's in routine use, and that has had many scripts written for it. The limitation (and power) of this system is that it's very self-contained. We interact with these scripts via their STDIN / STDOUT, and that's it. This makes it quite straightforward to create scripts, however it limits the amount of flexibility with what can be done with them, since the scripts couldn't e.g. manipulate internal Inkscape data structures or invoke internal Inkscape commands.
The second is the embedded scripting interpreter. This is what the configure script is set up to check. It is also available via the Inkscape menu (View -> Scripts), if you've compile it in. In theory, this will be able to connect up to Inkscape's internals, but presently it is still at proof-of-concept stage; you can run a script to pop up the about screen, and of course perform any other generic python operations, but that's about it.
The plan is to get the DOM work completed. This will establish a consistent internal API in Inkscape that the scripting system (among other things) can hook into. Without the DOM, we'd have to maintain the scripting language bindings in a more manual, haphazard fashion that would be very difficult to maintain.
Anyway, so the long and short of all this is that yes, you're right that solving the scripting ABI incompatibility issue is definitely important, but not yet critical for Inkscape since it's not something that is going to affect users in the near term. But, say, a year from now, it will be a very big deal, so if you can lay out the best long term scheme for handling these compatibility issues now, it'll help put everyone on the right path for long term needs.
(Btw, also note we have Perl bindings, which I imagine may have similar issues as Perl 6 begins to be deployed.)
Bryce
On Fri, Nov 25, 2005 at 02:28:50PM +0000, Mike Hearn wrote:
Inkscape, like many open source programs, uses Python as a scripting language for extensions. This is all fine and good, but unfortunately raises serious issues for binary distribution.
A few are obvious:
- Windows users don't have it
- Mac users probably have an out of date version
- Linux users almost certainly have it, but it's probably got a different ABI/API to what is expected
The first two do not concern me here. The last one does.
The libpython C API varies between systems in an incompatible manner. Python 2.2, 2.3 and 2.4 are all in the wild, and they export different APIs. Worse, Python can be configured to use either UCS2 or UCS4 unicode strings. The upstream default is UCS2 but some distributions configure it to use UCS4 - this changes the exported ABI.
I'm not sure what the solution to this is. Currently we recommend people do add Python for scripting to their applications, but that horse is long since gone for Inkscape. So we have a few alternatives:
- Disable Python scripting in autopackage build. That'd be a shame.
- Enable it in such a way that if libpython cannot be loaded, Inkscape still starts and works minus scripting
#2 may be the best solution for now, as I imagine it'd be more straightforward to implement than #3, but would leave users with at least the hope of having Python. As aaron points out, while the build system permits linking against libpython, the scripting support in Inkscape is not hooked up to Inkscape's internals, so is not of any practical use anyway at current. (It's worth keeping though, since once the DOM work is in place, we'll be able to hook into that and suddenly have a great deal of power available to pythonites.)
- Figure out some new magic tricks so we can be compatible with all of them (relaytool can help here), for instance by avoiding the unstable parts of the API
Would 'shim layers' be of any use? I.e., separate bits of code that sit between Inkscape and libpython to compensate for ABI incompatibilities? You'd have three of these shims, one for each of the three libpython versions, that allow Inkscape to speek consistently to all.
Also, this sounds like a larger problem than just for Inkscape... aren't other Python-using applications running into similar issues? What approaches have they taken to work around it?
- Drop Python and port the scripts to some other system, like Lua or Mono
4 sounds like a pass the buck solution. ;-) I don't know anything about Lua or Mono but would imagine that there could just as conceivably be an ABI issue with them (unless they're taking special precautions against that...) Our initial vision was for Inkscape to be scripted with a variety of common scripting languages (Python, Perl, ...) to enable as wide a variety of people as possible to be able to contribute in ways they are both skillful and comfortable with. Of course, a balance is needed due to the fact that every new scripting language increases the challenge for packagers.
(3) is the best for the user but requires the most in-depth knowledge of the Python API and versioning history.
Statically linking Python isn't really an option, unfortunately.
Agreed.
thanks -mike
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click _______________________________________________ Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
On Fri, 25 Nov 2005 11:47:49 -0800, Bryce Harrington wrote:
#2 may be the best solution for now, as I imagine it'd be more straightforward to implement than #3, but would leave users with at least the hope of having Python.
Yeah. I'm not a big fan of the unpredictability though.
Would 'shim layers' be of any use? I.e., separate bits of code that sit between Inkscape and libpython to compensate for ABI incompatibilities? You'd have three of these shims, one for each of the three libpython versions, that allow Inkscape to speek consistently to all.
Yes, that's an option. Somebody has to write it though.
Also, this sounds like a larger problem than just for Inkscape... aren't other Python-using applications running into similar issues?
Yes.
What approaches have they taken to work around it?
They haven't. They just pass the problem on to the distributions to work around - some stuff like the Unicode fiasco changes the ABI but not the API so recompiling magically "fixes" it.
4 sounds like a pass the buck solution. ;-) I don't know anything about Lua or Mono but would imagine that there could just as conceivably be an ABI issue with them (unless they're taking special precautions against that...)
Well, Lua is designed as an embedded scripting language and can be statically linked easily. The downside is you don't get a big standard library.
I don't know how stable the Mono embedding API is, I must admit.
thanks -mike
On 11/25/05, Mike Hearn <mike@...869...> wrote:
Statically linking Python isn't really an option, unfortunately.
If not linking, can we at least provide our own binary of python in the package? This would solve it for all platforms.
-- bulia byak Inkscape. Draw Freely. http://www.inkscape.org
On Fri, 25 Nov 2005 18:41:30 -0400, bulia byak wrote:
If not linking, can we at least provide our own binary of python in the package? This would solve it for all platforms.
It may be possible, but Python is enormous, it'd hugely increase the download size. And that approach doesn't really scale when other projects start doing it too. The Python unicode issues were hit first with Scribus.
thanks -mike
On Saturday 26 November 2005 16:38, Mike Hearn wrote:
On Fri, 25 Nov 2005 18:41:30 -0400, bulia byak wrote:
If not linking, can we at least provide our own binary of python in the package? This would solve it for all platforms.
It may be possible, but Python is enormous, it'd hugely increase the download size. And that approach doesn't really scale when other projects start doing it too. The Python unicode issues were hit first with Scribus.
The Scribus 1.3.x Win32 build currently includes Python, but when we finally build an installer, we hope to offer to install if an existing one is not found or link to the existing one.
Craig
On 11/26/05, Mike Hearn <mike@...869...> wrote:
On Fri, 25 Nov 2005 18:41:30 -0400, bulia byak wrote:
If not linking, can we at least provide our own binary of python in the package? This would solve it for all platforms.
It may be possible, but Python is enormous, it'd hugely increase the download size. And that approach doesn't really scale when other projects start doing it too. The Python unicode issues were hit first with Scribus.
Not so enormous if we inlclude only what WE need. And disk space is cheap. Much cheaper than the efforts of developers trying to work around innumerable incompatibilities and bugs. Given that even you cannot identify a perfect solution, I think this one is the best in the long run. Especially important is that it's actually a solution for all platforms, without which no solution is actually a solution at all.
Of course installers can offer the option of not installing Python if the user so desires. And/or, we could offer -with-python and -without-python installers to choose from.
-- bulia byak Inkscape. Draw Freely. http://www.inkscape.org
On 11/26/05, bulia byak <buliabyak@...400...> wrote:
On 11/26/05, Mike Hearn <mike@...869...> wrote:
On Fri, 25 Nov 2005 18:41:30 -0400, bulia byak wrote:
If not linking, can we at least provide our own binary of python in the package? This would solve it for all platforms.
It may be possible, but Python is enormous, it'd hugely increase the download size. And that approach doesn't really scale when other projects start doing it too. The Python unicode issues were hit first with Scribus.
Not so enormous if we inlclude only what WE need. And disk space is cheap. Much cheaper than the efforts of developers trying to work around innumerable incompatibilities and bugs. Given that even you cannot identify a perfect solution, I think this one is the best in the long run. Especially important is that it's actually a solution for all platforms, without which no solution is actually a solution at all.
concerning mac os X, both 10.3 and 10.4 comes with 2.3 but both of them don't have pyXML. I downloaded and installed it and everything works like a charm now (for python a least because there are still Perl problems). I tried to produce a "binary" (a dmg that people can just open and copy to their python directory) but no one tested it yet. If it works it will be the solution of OSX: it's just 1 Mb.
-- JiHO --- Windows, c'est un peu comme le beaujolais nouveau : a chaque nouvelle cuvee on sait que ce sera degueulasse, mais on en prend quand meme par masochisme. ---
On Sat, 26 Nov 2005 14:44:13 -0400, bulia byak wrote:
Not so enormous if we inlclude only what WE need.
The main problem is how do you know what extension authors will want to use? Python is partly useful because of the standard library, if you start dropping parts of that library at random, Python becomes less useful and you might as well just use Lua. It's the std library rather than the interpreter itself which is large.
And disk space is cheap.
Agreed. However, memory and bandwidth still aren't for many people.
Much cheaper than the effrts of developers trying to work around innumerable incompatibilities and bugs. Given that even you cannot identify a perfect solution, I think this one is the best in the long run. Especially important is that it's actually a solution for all platforms, without which no solution is actually a solution at all.
That's true, though given the different nature of Python support on each major platform, I think we'll end up with platform specific solutions for now anyway.
Of course installers can offer the option of not installing Python if the user so desires. And/or, we could offer -with-python and -without-python installers to choose from.
This is true. Autopackage can easily cope with checking for the systems python and only downloading an included copy if it's incompatible.
thanks -mike
participants (7)
-
unknown＠example.com -
Bob Jamison -
Bryce Harrington -
bulia byak -
Craig Bradney -
jiho -
Mike Hearn