Dear all,
I've separated out the extensions directory into a new repository:
https://gitlab.com/inkscape/extensions
I haven't yet removed the files from the master branch, but I will as soon as I get a couple of nods of approval from other developers and packagers as this change needs some consideration.
My first job is to upgrade extensions to python3 and to do this I'm going to write tests. Please let me know if there are any extensions tests already written I can use.
Best Regards, Martin Owens
On 03/21/2018 07:43 PM, Martin Owens wrote:
I've separated out the extensions directory into a new repository:
https://gitlab.com/inkscape/extensions
I haven't yet removed the files from the master branch, but I will as soon as I get a couple of nods of approval from other developers and packagers as this change needs some consideration.
Yes, do you have a proposal for how all those things should work? Don't forget to worry about translations as well.
Ted
On Wed, 2018-03-21 at 21:48 -0500, Ted Gould wrote:
On 03/21/2018 07:43 PM, Martin Owens wrote:
I've separated out the extensions directory into a new repository:
https://gitlab.com/inkscape/extensions
I haven't yet removed the files from the master branch, but I will
as
soon as I get a couple of nods of approval from other developers
and
packagers as this change needs some consideration.
Yes, do you have a proposal for how all those things should work? Don't forget to worry about translations as well.
Yes.
The non-hard things is just translating to python3, I've found the test directory which has some serious problems, but have started rehab (one of which is it being called 'test' which conflicts with python's test module)
Once that part is done, we start making extensions a community thing, rather than a developer only thing.
Firstly we put any non-python extensions into their own repository. We don't want to drop support for just running weird shell scripts and perl stuff, but I'd put it in auxiliary and ask packagers to add them to separate packages.
This is so we can focus the core extensions repository on making it a python installable module set with the inx coming along as data.
The reason we'd want to move in this direction is really to allow inkscape to install and uninstall extensions while not having to write our own dependency software.
If we take advantage of pip/virtualenv in our .local/share/inkscape/extensions (Or do we stuff everything in .config like naughty XDG users 😉) then we can install extensions from zi p files, from git repositories on github and do uninstalls by just managing the pip repository from the inkscape UI.
This will install all the required dependencies (numpy anyone?) and keep files separated.
Just imagine being able to invite all those github repositoryinkscape extensions and inkscape.org zip file extensions into being used by Inkscape users. It'll be amazing.
Best Regards, Martin Owens
On 03/21/2018 10:54 PM, Martin Owens wrote:
Firstly we put any non-python extensions into their own repository. We don't want to drop support for just running weird shell scripts and perl stuff, but I'd put it in auxiliary and ask packagers to add them to separate packages.
Frankly, you're thinking like old school Linux packaging. Multiple packages with dependencies isn't really how most modern OSes are thinking right now. So I think that relying on that is perhaps a play towards the past.
I'd agree on the goals of making it so that an extension can be published separately, on a separate schedule, and work well in Inkscape. But, I don't think we're to the point of that working well with things like dropping to command line instructions, much less being user friendly enough to be practical.
This is so we can focus the core extensions repository on making it a python installable module set with the inx coming along as data.
The reason we'd want to move in this direction is really to allow inkscape to install and uninstall extensions while not having to write our own dependency software.
If we take advantage of pip/virtualenv in our .local/share/inkscape/extensions (Or do we stuff everything in .config like naughty XDG users 😉) then we can install extensions from zi p files, from git repositories on github and do uninstalls by just managing the pip repository from the inkscape UI.
This will install all the required dependencies (numpy anyone?) and keep files separated.
Installing executables in your home directory is a security nightmare. We really don't want to be in that business. And I have doubts that'll work in Windows, but there may be some magic there I don't know about. PIP does some fun things with compiling executables and libraries when it needs them.
I hate to be stop energy on porting the python extensions to Python 3, that clearly needs to happen. But I don't think we're ready to have them shipping and developed out of a different repository. The distribution issues are critical there. I think that we need a test extension that works well in all the repos/stores/etc before we can split out the extensions that we have.
Ted
On Thu, 2018-03-22 at 00:15 -0500, Ted Gould wrote:
I hate to be stop energy on porting the python extensions to Python 3, that clearly needs to happen. But I don't think we're ready to have them shipping and developed out of a different repository. The distribution issues are critical there. I think that we need a test extension that works well in all the repos/stores/etc before we can split out the extensions that we have.
I understand reservations, but I don't agree with your conclusions Ted. I'm suggesting standardization on existing technologies that we're already using. Using well worn and supported formats and existing multi-platform code.
I'll have to check all the bits will work, sure, and the good thing about having a separate repository is that it's going to let us be more experimental.
As for deps. I'll let you make a snap for every extension contributor ��� ��� It's a bloody minded way of making people do lots of work though.
Overall the proposed direction is tentative, I'm fishing for better ideas. It's still the best one so far.
Best Regards, Martin Owens
Thanks Mark,
that list shows some serious thought beyond the often limited views of an Inkscape developer/packager and highlights some of the actual issues that need to be tackled if such a splitting should have a chance to be successful!
@Martin: I have *a lot* of reservations on the technical side (maybe I have some time to describe some of them in detail later), most importantly that the whole process you describe for distribution those extensions *might* work on Linux (but even there seem to be different opinions and I doubt it's possible to evaluate all possible distribution channels), while almost every step will break on Windows (there is no package manager, no reliable pip, no reliable virtualenv, ... not even a reliable Python for that matter, so we usually need to bring our own!) and other platforms might be affected similarly.
I have a feeling it will not make packaging easier but will make it actually a lot harder for many platforms and distribution channels. I'm afraid this will quickly result in extensions being "dead" for those platforms, which will a little slower but eventually lead to extensions being "dead" altogether.
Also copying my thoughts from a while back from a bug on launchpad from https://bugs.launchpad.net/ubuntu/+source/inkscape/+bug/1735363 (not sure they were considered or even read...)
One thing we'd need to consider is that some more or less basic Inkscape functionality currently relies on Python extensions, e.g.
- Several templates
- Help links
although I already wondered if this was even really necessary - Â Â I assume there are better solutions available to open a link from GTK
- Export/Import extensions
They are not strictly Inkscape core but a lot of users are not aware   they might be running an extension to export to their favorite format.
Second important questions is if this would only affect packaging or if we really want to split out the extension code from the Inkscape source repository into a more or less self sufficient sub-project. In the latter case I fear that the bundled extensions - which are in a pretty desolate state even now - might bit-rot even faster (less users, less developers having the repo on their radar, less obvious affiliation with the core program). If we could achieve to make the extensions into a "community project" (that is a lot more user driven, allowing easy contributions and additions of new extensions with well-organized categorization and documentation) that could also improve upon the status-quo but as a start this would require a "core team" of people who would be willing to do the necessary maintenance (not sure we have that right now...).
As a reality check:
* Which problems are we actually trying to solve? (I know the way we currently bundle extensions is not perfect - but it *works*!) * How likely is it that we'll introduce more new problems by splitting the extension repo than we can possibly hope to solve?
Regards, Eduard
Am 22.03.2018 um 08:53 schrieb Mark Schafer:
I have been slowly working on all of the third party extensions I can find (with help from Brynn and Maren) to bring them into compatibility with 0.91. As the units changed for 0.92 all the work needed to be done again and I lost energy for a while. I have just moved to a new house and that's also taken up a lot of time this last year. Nevertheless I do intend to help the authors prepare, as many of these extensions as I can, for useful work in Inkscape. Many have incorporated my pull requests into their code.
I don't know if the proposed mechanism is the best way or what mechanism might be better but IMHO these are the issues that extensions - in their current mechanism) could do with some help with solving:
- Stable approach to units - many extensions seem to deal with real
world dimensions and care about this. Â - probably no issue with the installer mechanism so ignore. Â - currently in 0.92 it appears all internal numbers (as seen in the xml editor) are in mm. In 0.48 they were in 90dpi px. Â - I am unsure of what 0.93 proposes but currently assuming internal mm as in 0.92.
- Many extensions use the supplied code pieces like simplecurve.py etc
- these need to be stable and when updated make new names and not reuse existing function names  - we need some new ones to inherit such things as drawn dimensions and graphs which have been quite nicely worked out in user side only (not in inkscape repo) extensions. Most useful ones of these make new classes which inherit from inkex.
- At least one of which does not work well anymore in 0.92 due to
units change. I will help fix.
- Localisation. Many extension writers have tried to work out how to
supply translations and have them work. Currently this is too hard (separate package for each language or script that runs and renames files - too dangerous for Windows etc.
- Adding their own UI. In the cases where the current inx based UI
falls down (conditional values, and many more) the users want to able to use, say, Kivy or even tkinter to do dialogs. This is super hard and only one I know that works is the very recent embroidery extension which needs own installer and is so complex I can't follow it.
- Python versions - In 0.48 the version of python was quite old - now
newer but as pointed out, want to move to 3.x. This means rewrite of all user extensions also. but as print is seldom used, not neccessarily a lot of work for each one. I will help them out. Python 3.7 is in beta now.  - even better would be to allow user to choose the python installed on the machine - so deps like numpy which may already be there don't need to be reinstalled. Instead of bundling a specific version of python with inkscape.  - but keeping 2.7 and 3.x at same time would be even more useful.  - e,g, including the line |#!/usr/bin/env python2| (or |python3|) in any script. See PEP 397 which was included in python 3.3 and works on Windows.  - https://python-forum.io/Thread-Install-Python-2-and-3-on-same-Computer
- There's probably a couple more but I can't recall right now.
FYI - The long term issues not expected to be enabled any time soon are:
- calling internal inkscape functions - I.e. an exposed python API
- getting onto the internal UI.
- interactive control of a parameter while adjusting.
Cheers Mark...
On 3/22/2018 7:11 PM, Martin Owens wrote:
On Thu, 2018-03-22 at 00:15 -0500, Ted Gould wrote:
I hate to be stop energy on porting the python extensions to Python 3, that clearly needs to happen. But I don't think we're ready to have them shipping and developed out of a different repository. The distribution issues are critical there. I think that we need a test extension that works well in all the repos/stores/etc before we can split out the extensions that we have.
I understand reservations, but I don't agree with your conclusions Ted. I'm suggesting standardization on existing technologies that we're already using. Using well worn and supported formats and existing multi-platform code.
I'll have to check all the bits will work, sure, and the good thing about having a separate repository is that it's going to let us be more experimental.
As for deps. I'll let you make a snap for every extension contributor ��� ��� It's a bloody minded way of making people do lots of work though.
Overall the proposed direction is tentative, I'm fishing for better ideas. It's still the best one so far.
Best Regards, Martin Owens
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org!http://sdm.link/slashdot _______________________________________________ Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Hi Mark,
just two bits of info:
Am 22.03.2018 um 08:53 schrieb Mark Schafer:
- probably no issue with the installer mechanism so ignore. Â - currently in 0.92 it appears all internal numbers (as seen in the xml editor) are in mm. In 0.48 they were in 90dpi px.
- This is due to the default template. Each template can have a different one. If you use the methods supplied by inkex.py for unit conversion, you do not need to care about the actual values.
- Adding their own UI. In the cases where the current inx based UI
falls down (conditional values, and many more) the users want to able to use, say, Kivy or even tkinter to do dialogs. This is super hard and only one I know that works is the very recent embroidery extension which needs own installer and is so complex I can't follow it.
- On Windows, Tkinter is included from 0.92.3 on.
Great to see you back again :)
Maren
Cheers Mark...
On 3/22/2018 7:11 PM, Martin Owens wrote:
On Thu, 2018-03-22 at 00:15 -0500, Ted Gould wrote:
I hate to be stop energy on porting the python extensions to Python 3, that clearly needs to happen. But I don't think we're ready to have them shipping and developed out of a different repository. The distribution issues are critical there. I think that we need a test extension that works well in all the repos/stores/etc before we can split out the extensions that we have.
I understand reservations, but I don't agree with your conclusions Ted. I'm suggesting standardization on existing technologies that we're already using. Using well worn and supported formats and existing multi-platform code.
I'll have to check all the bits will work, sure, and the good thing about having a separate repository is that it's going to let us be more experimental.
As for deps. I'll let you make a snap for every extension contributor ��� ��� It's a bloody minded way of making people do lots of work though.
Overall the proposed direction is tentative, I'm fishing for better ideas. It's still the best one so far.
Best Regards, Martin Owens
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Hi Maren,
Yes many people just used number in their python code for widths. E.g 6. This meant 6px in V 0.48 and 6mm in 0.92 The proper way is to convert all numbers to units in all extensions going forward - its just that many writers did not understand that. The SVG is constructed directly in extensions.
On 3/23/2018 3:09 AM, Maren Hachmann wrote:
Hi Mark,
just two bits of info:
Am 22.03.2018 um 08:53 schrieb Mark Schafer:
- probably no issue with the installer mechanism so ignore. Â - currently in 0.92 it appears all internal numbers (as seen in the xml editor) are in mm. In 0.48 they were in 90dpi px.
- This is due to the default template. Each template can have a
different one. If you use the methods supplied by inkex.py for unit conversion, you do not need to care about the actual values.
- Adding their own UI. In the cases where the current inx based UI
falls down (conditional values, and many more) the users want to able to use, say, Kivy or even tkinter to do dialogs. This is super hard and only one I know that works is the very recent embroidery extension which needs own installer and is so complex I can't follow it.
- On Windows, Tkinter is included from 0.92.3 on.
Great to see you back again :)
Maren
Cheers Mark...
On 3/22/2018 7:11 PM, Martin Owens wrote:
On Thu, 2018-03-22 at 00:15 -0500, Ted Gould wrote:
I hate to be stop energy on porting the python extensions to Python 3, that clearly needs to happen. But I don't think we're ready to have them shipping and developed out of a different repository. The distribution issues are critical there. I think that we need a test extension that works well in all the repos/stores/etc before we can split out the extensions that we have.
I understand reservations, but I don't agree with your conclusions Ted. I'm suggesting standardization on existing technologies that we're already using. Using well worn and supported formats and existing multi-platform code.
I'll have to check all the bits will work, sure, and the good thing about having a separate repository is that it's going to let us be more experimental.
As for deps. I'll let you make a snap for every extension contributor ��� ��� It's a bloody minded way of making people do lots of work though.
Overall the proposed direction is tentative, I'm fishing for better ideas. It's still the best one so far.
Best Regards, Martin Owens
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
This is great Mark, thanks for writing it up and working on the units conversations. Do you have a list of the extensions that are out there? In the Wiki or something similar?
On 03/22/2018 02:53 AM, Mark Schafer wrote:
- Many extensions use the supplied code pieces like simplecurve.py etc
- these need to be stable and when updated make new names and not reuse existing function names  - we need some new ones to inherit such things as drawn dimensions and graphs which have been quite nicely worked out in user side only (not in inkscape repo) extensions. Most useful ones of these make new classes which inherit from inkex.
- At least one of which does not work well anymore in 0.92 due to
units change. I will help fix.
How are the extensions typically finding things like inkex.py and simplecurve.py? AFAIK we're not installing them as Python modules so there are some path issues. Are they hardcoding them?
- Localisation. Many extension writers have tried to work out how to
supply translations and have them work. Currently this is too hard (separate package for each language or script that runs and renames files - too dangerous for Windows etc.
This is going to be hard, and we're gonna have to handle it in Inkscape with loading a translations directory and looking for them there. This is probably towards the top of my list of hard problems.
- Adding their own UI. In the cases where the current inx based UI
falls down (conditional values, and many more) the users want to able to use, say, Kivy or even tkinter to do dialogs. This is super hard and only one I know that works is the very recent embroidery extension which needs own installer and is so complex I can't follow it.
I think for us doing any kind of installer this would be tricky. This usually requires some amount of the GUI libraries to be installed and/or bindings compiled. We want extension authors to choose their toolkit, but I'm not sure how we can handle all that choice easily.
- Python versions - In 0.48 the version of python was quite old - now
newer but as pointed out, want to move to 3.x. This means rewrite of all user extensions also. but as print is seldom used, not neccessarily a lot of work for each one. I will help them out. Python 3.7 is in beta now.  - even better would be to allow user to choose the python installed on the machine - so deps like numpy which may already be there don't need to be reinstalled. Instead of bundling a specific version of python with inkscape.  - but keeping 2.7 and 3.x at same time would be even more useful.  - e,g, including the line |#!/usr/bin/env python2| (or |python3|) in any script. See PEP 397 which was included in python 3.3 and works on Windows.  - https://python-forum.io/Thread-Install-Python-2-and-3-on-same-Computer
I'm not sure how we'd make everyone happy here. I think the alternate solution would be to allow every extension to bundle its own version of Python (or use the system python) as the author there wanted. Some people are die hard 2.7 folks, some like the latest and greatest, and that doesn't even include the Perl people :-)
FYI - The long term issues not expected to be enabled any time soon are:
- calling internal inkscape functions - I.e. an exposed python API
- getting onto the internal UI.
- interactive control of a parameter while adjusting.
When talking about using the internal Inkscape API, do you know much of what people want there? I'm curious if we're not better of pushing people to using lib2geom than thinking how to make Inkscape into a library. I think that most of the use-cases I've seen could be handled by lib2geom.
Ted
On Thu, 2018-03-22 at 09:32 -0500, Ted Gould wrote:
How are the extensions typically finding things like inkex.py and simplecurve.py? AFAIK we're not installing them as Python modules so there are some path issues. Are they hardcoding them?
Something that could be separated out into a package, or module, at least for development and testing. I don't believe any of the libs need inkscape to run.
FYI - The long term issues not expected to be enabled any time soon are:
- calling internal inkscape functions - I.e. an exposed python API
- getting onto the internal UI.
- interactive control of a parameter while adjusting.
When talking about using the internal Inkscape API, do you know much of what people want there? I'm curious if we're not better of pushing people to using lib2geom than thinking how to make Inkscape into a library. I think that most of the use-cases I've seen could be handled by lib2geom.
A python wrapper for lib2geom would be AWESOME. Especially if it could be used without inkscape. +1
Best Regards, Martin Owens
Am 22.03.2018 um 15:32 schrieb Ted Gould:
This is great Mark, thanks for writing it up and working on the units conversations. Do you have a list of the extensions that are out there? In the Wiki or something similar?
On 03/22/2018 02:53 AM, Mark Schafer wrote:
- Many extensions use the supplied code pieces like simplecurve.py etc
- these need to be stable and when updated make new names and not reuse existing function names  - we need some new ones to inherit such things as drawn dimensions and graphs which have been quite nicely worked out in user side only (not in inkscape repo) extensions. Most useful ones of these make new classes which inherit from inkex.
- At least one of which does not work well anymore in 0.92 due to
units change. I will help fix.
How are the extensions typically finding things like inkex.py and simplecurve.py? AFAIK we're not installing them as Python modules so there are some path issues. Are they hardcoding them?
- They don't need to do anything - just plain 'import inkex' or 'import simplepath'. Everything that's in /.config/extensions and /share/extensions seems to already be available. I /think/ that both directories are just added to the path when one executes a script extension (but couldn't find that in the code..., maybe it's even a gtk thing that deals with the configuration hierarchy?).
- Btw. as far as I understand, our Windows versions come with pip as part of the python installation, to facilitate installing extension dependencies (Eduard will know more about this). Doesn't look like a virtualenv to me, though.
Maren
- Localisation. Many extension writers have tried to work out how to
supply translations and have them work. Currently this is too hard (separate package for each language or script that runs and renames files - too dangerous for Windows etc.
This is going to be hard, and we're gonna have to handle it in Inkscape with loading a translations directory and looking for them there. This is probably towards the top of my list of hard problems.
- Adding their own UI. In the cases where the current inx based UI
falls down (conditional values, and many more) the users want to able to use, say, Kivy or even tkinter to do dialogs. This is super hard and only one I know that works is the very recent embroidery extension which needs own installer and is so complex I can't follow it.
I think for us doing any kind of installer this would be tricky. This usually requires some amount of the GUI libraries to be installed and/or bindings compiled. We want extension authors to choose their toolkit, but I'm not sure how we can handle all that choice easily.
- Python versions - In 0.48 the version of python was quite old - now
newer but as pointed out, want to move to 3.x. This means rewrite of all user extensions also. but as print is seldom used, not neccessarily a lot of work for each one. I will help them out. Python 3.7 is in beta now.  - even better would be to allow user to choose the python installed on the machine - so deps like numpy which may already be there don't need to be reinstalled. Instead of bundling a specific version of python with inkscape.  - but keeping 2.7 and 3.x at same time would be even more useful.  - e,g, including the line |#!/usr/bin/env python2| (or |python3|) in any script. See PEP 397 which was included in python 3.3 and works on Windows.  - https://python-forum.io/Thread-Install-Python-2-and-3-on-same-Computer
I'm not sure how we'd make everyone happy here. I think the alternate solution would be to allow every extension to bundle its own version of Python (or use the system python) as the author there wanted. Some people are die hard 2.7 folks, some like the latest and greatest, and that doesn't even include the Perl people :-)
FYI - The long term issues not expected to be enabled any time soon are:
- calling internal inkscape functions - I.e. an exposed python API
- getting onto the internal UI.
- interactive control of a parameter while adjusting.
When talking about using the internal Inkscape API, do you know much of what people want there? I'm curious if we're not better of pushing people to using lib2geom than thinking how to make Inkscape into a library. I think that most of the use-cases I've seen could be handled by lib2geom.
Ted
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
On Fri, 2018-03-23 at 01:22 +0100, Maren Hachmann wrote:
- Btw. as far as I understand, our Windows versions come with pip as
part of the python installation, to facilitate installing extension dependencies (Eduard will know more about this). Doesn't look like a virtualenv to me, though.
Pip may be all that's required, you don't need virtualenv to have a custom prefix:
https://stackoverflow.com/questions/2915471/install-a-python-package-in to-a-different-directory-using-pip
I believe this eliminates one of the problems, no extra deps (on windows at least) and removing binaries from the conf.
Martin,
Am 23.03.2018 um 03:44 schrieb Martin Owens:
On Fri, 2018-03-23 at 01:22 +0100, Maren Hachmann wrote:
- Btw. as far as I understand, our Windows versions come with pip as
part of the python installation, to facilitate installing extension dependencies (Eduard will know more about this). Doesn't look like a virtualenv to me, though.
Pip may be all that's required, you don't need virtualenv to have a custom prefix:
https://stackoverflow.com/questions/2915471/install-a-python-package-in to-a-different-directory-using-pip
I believe this eliminates one of the problems, no extra deps (on windows at least) and removing binaries from the conf.
Martin,
I'll stop you right there (I'd prefer if we stopped "eliminating" problems that swiftly without actual investigation):
* pip is currently *not* included in Windows builds (it was for a short time when I last updated the devlibs - which we do not use anymore - but it was completely experimental back then and usage was not straightforward with some things not working as one would expect). Investigating the usage of pip is on my todo list but I did not find the time so far. * Even *if* we get pip do work properly you need to be aware that it's generally not possible to install any modules with it that contain binary components. pip is not magically allowing us to compile stuff on-the-fly. It can only work for pure Python modules (and even that fails occasionally, especially if any non-standard setups are attempted - like the one you seem to suggest).
Regards, Eduard
On Fri, 2018-03-23 at 10:41 +0100, Eduard Braun wrote:
pip is currently *not* included in Windows builds (it was for a short time when I last updated the devlibs - which we do not use anymore - but it was completely experimental back then and usage was not straightforward with some things not working as one would expect). Investigating the usage of pip is on my todo list but I did not find the time so far.
Very good to know. I'm very ignorant of the windows builds.
Even *if* we get pip do work properly you need to be aware that it's generally not possible to install any modules with it that contain binary components. pip is not magically allowing us to compile stuff on-the-fly. It can only work for pure Python modules (and even that fails occasionally, especially if any non-standard setups are attempted - like the one you seem to suggest).
We may not need binary builds, there are only a handful of cases where they'd be useful. Our own libs if they got python support would be shipped with Inkscape, while things like numpy if they were ever needed, could use either wheels or external installation.
Supporting that internally would be a perfect solution, but it shouldn't be the enemy of the good for the majority. ~90% of extensions are likely to not need any external dependencies, and ~90% of them will use pure python libraries. We can forgo ~1% of support being automatic.
Best Regards, Martin Owens
Am 23.03.2018 um 10:41 schrieb Eduard Braun:
Am 23.03.2018 um 03:44 schrieb Martin Owens:
On Fri, 2018-03-23 at 01:22 +0100, Maren Hachmann wrote:
- Btw. as far as I understand, our Windows versions come with pip as
part of the python installation, to facilitate installing extension dependencies (Eduard will know more about this). Doesn't look like a virtualenv to me, though.
Pip may be all that's required, you don't need virtualenv to have a custom prefix:
https://stackoverflow.com/questions/2915471/install-a-python-package-in to-a-different-directory-using-pip
I believe this eliminates one of the problems, no extra deps (on windows at least) and removing binaries from the conf.
Martin,
I'll stop you right there (I'd prefer if we stopped "eliminating" problems that swiftly without actual investigation):
- pip is currently *not* included in Windows builds (it was for a short time when I last updated the devlibs - which we do not use anymore - but it was completely experimental back then and usage was not straightforward with some things not working as one would expect). Investigating the usage of pip is on my todo list but I did not find the time so far.
- Eduard, I had looked and had found the following in Inkscape 0.92.2, isn't that pip? Else I wouldn't have written anything about it here. I also understood that some people had been using it for installing dependencies - but it might be they had to install it separately.
https://i.imgur.com/R9YSqiE.png
Or maybe it just doesn't work, but is included? Or it's something else...? Sorry, I don't know enough.
Maren
- Even *if* we get pip do work properly you need to be aware that it's generally not possible to install any modules with it that contain binary components. pip is not magically allowing us to compile stuff on-the-fly. It can only work for pure Python modules (and even that fails occasionally, especially if any non-standard setups are attempted - like the one you seem to suggest).
Regards, Eduard
On 03/22/2018 01:11 AM, Martin Owens wrote:
On Thu, 2018-03-22 at 00:15 -0500, Ted Gould wrote:
I hate to be stop energy on porting the python extensions to Python 3, that clearly needs to happen. But I don't think we're ready to have them shipping and developed out of a different repository. The distribution issues are critical there. I think that we need a test extension that works well in all the repos/stores/etc before we can split out the extensions that we have.
I understand reservations, but I don't agree with your conclusions Ted. I'm suggesting standardization on existing technologies that we're already using. Using well worn and supported formats and existing multi-platform code.
No? We don't use pip, virtualenv, and we don't install any executables in a user's home folder. And we don't package any extensions with the required Python packaging for such things.
I'll have to check all the bits will work, sure, and the good thing about having a separate repository is that it's going to let us be more experimental.
A place to experiment is good, but with a single extension, not all of them. Not as a fork. IMHO, we're going to need to ship them with main project for a bit longer, so putting them in a different repo adds complexity.
As for deps. I'll let you make a snap for every extension contributor ��� ��� It's a bloody minded way of making people do lots of work though.
At the end of the day they need to be packaged somehow that goes into all of the various stores. Snaps are one of those. Frankly, I'm more worried about the Windows Store because I understand it less and it is where the majority of our users are. Also, MS has shown that is likely to be the only user-friendly way to install software on Windows in the future.
Overall the proposed direction is tentative, I'm fishing for better ideas. It's still the best one so far.
I think that looking for an ideas is a good plan. We should look for ideas, find a good one, implement it, prove it works, and then fork the repository. Starting at the end isn't a good plan.
Ted
On 03/22/2018 09:39 AM, Martin Owens wrote:
On Thu, 2018-03-22 at 09:21 -0500, Ted Gould wrote:
Starting at the end isn't a good plan.
I see it as the start.
Thanks for the thoughtful replies though, there's much to consider.
Okay. Please don't delete the extensions from the main repository until the other issues are worked out. That way we can do releases as we have done until we get all the pieces of the puzzle together.
Ted
It's good to see progress starting on this. The roadmap lists this as a task for 0.93, and this is a good point in the release process to start hacking on it.
As Ted and Eduard point out, the task may sound simpler than it really is. This might be one of those types of things where the hard part is coming up with a plan of attack. It's good that some actual work has been done so there's something to focus critique on. I do agree we need to find consensus on an approach before moving from experimentation to integration of the change into the main codebase.
I'd like to see some additional thought into how this is hosted in gitlab. I.e. it may make sense to establish a subteam that owns the extensions, so that for example contributor permissions can be managed differently than we'd want for the core codebase. I don't really have a solid idea in mind here, but perhaps we can brainstorm a bit next week?
As an old school Linux packager, the proposed approach feels natural to me, but Ted and Eduard make good points that it may not be as clean a fit on other platforms that Inkscape needs to be on. It may be worth doing some analysis into how similar software projects solve this problem. It's entirely possible there is no perfect solution that would address everyone's desires - but I'm hopeful we can at least find an approach that is at least workable.
One difficulty here is that if we provide a distribution/installation system for 3rd party extension writers to use, then if we change that system they'll be quite grumpy. Yet getting all the requirements satisfied in one go would be a formiddable challenge.
It may benefit us to do this in a few stages. For instance, a first step might have the extensions housed in their own gitlab repository and subteam, but still be integrated into the source tarball at release time, thus sidestepping all of the packaging/platform problems but gaining some of the team membership flexibility. A next step would focus on modularizing the extension installation, but keeping distribution centralized as we do it now (i.e. everything is released in concert with the main release). A later step would work on modularizing distribution to work asynchronously, so users can install and update modules at will.
Let's definitely plan on talking about extensions and modularization in general at the hackfest.
Bryce
On Thu, Mar 22, 2018 at 12:15:42AM -0500, Ted Gould wrote:
On 03/21/2018 10:54 PM, Martin Owens wrote:
Firstly we put any non-python extensions into their own repository. We don't want to drop support for just running weird shell scripts and perl stuff, but I'd put it in auxiliary and ask packagers to add them to separate packages.
Frankly, you're thinking like old school Linux packaging. Multiple packages with dependencies isn't really how most modern OSes are thinking right now. So I think that relying on that is perhaps a play towards the past.
I'd agree on the goals of making it so that an extension can be published separately, on a separate schedule, and work well in Inkscape. But, I don't think we're to the point of that working well with things like dropping to command line instructions, much less being user friendly enough to be practical.
This is so we can focus the core extensions repository on making it a python installable module set with the inx coming along as data.
The reason we'd want to move in this direction is really to allow inkscape to install and uninstall extensions while not having to write our own dependency software.
If we take advantage of pip/virtualenv in our .local/share/inkscape/extensions (Or do we stuff everything in .config like naughty XDG users 😉) then we can install extensions from zi p files, from git repositories on github and do uninstalls by just managing the pip repository from the inkscape UI.
This will install all the required dependencies (numpy anyone?) and keep files separated.
Installing executables in your home directory is a security nightmare. We really don't want to be in that business. And I have doubts that'll work in Windows, but there may be some magic there I don't know about. PIP does some fun things with compiling executables and libraries when it needs them.
I hate to be stop energy on porting the python extensions to Python 3, that clearly needs to happen. But I don't think we're ready to have them shipping and developed out of a different repository. The distribution issues are critical there. I think that we need a test extension that works well in all the repos/stores/etc before we can split out the extensions that we have.
Ted
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Am 23.03.2018 um 00:08 schrieb Bryce Harrington:
<...>
As an old school Linux packager, the proposed approach feels natural to me, but Ted and Eduard make good points that it may not be as clean a fit on other platforms that Inkscape needs to be on. It may be worth doing some analysis into how similar software projects solve this problem. It's entirely possible there is no perfect solution that would address everyone's desires - but I'm hopeful we can at least find an approach that is at least workable.
- I've often seen them using git submodules. For example, MyCroft AI (smart assistant), pelican (python static website generator) and Atom (editor) use this approach for their (partially user-contributed) extensions (but they do have the extensions in a separate repo, and pull **those** in via submodules).
I seem to remember though that this thought had been rejected due to the complexity it brings. Currently, I wouldn't know how to work with it either, but probably it wouldn't be too hard to find out.
Maren
One difficulty here is that if we provide a distribution/installation system for 3rd party extension writers to use, then if we change that system they'll be quite grumpy. Yet getting all the requirements satisfied in one go would be a formiddable challenge.
It may benefit us to do this in a few stages. For instance, a first step might have the extensions housed in their own gitlab repository and subteam, but still be integrated into the source tarball at release time, thus sidestepping all of the packaging/platform problems but gaining some of the team membership flexibility. A next step would focus on modularizing the extension installation, but keeping distribution centralized as we do it now (i.e. everything is released in concert with the main release). A later step would work on modularizing distribution to work asynchronously, so users can install and update modules at will.
Let's definitely plan on talking about extensions and modularization in general at the hackfest.
Bryce
On Thu, Mar 22, 2018 at 12:15:42AM -0500, Ted Gould wrote:
On 03/21/2018 10:54 PM, Martin Owens wrote:
Firstly we put any non-python extensions into their own repository. We don't want to drop support for just running weird shell scripts and perl stuff, but I'd put it in auxiliary and ask packagers to add them to separate packages.
Frankly, you're thinking like old school Linux packaging. Multiple packages with dependencies isn't really how most modern OSes are thinking right now. So I think that relying on that is perhaps a play towards the past.
I'd agree on the goals of making it so that an extension can be published separately, on a separate schedule, and work well in Inkscape. But, I don't think we're to the point of that working well with things like dropping to command line instructions, much less being user friendly enough to be practical.
This is so we can focus the core extensions repository on making it a python installable module set with the inx coming along as data.
The reason we'd want to move in this direction is really to allow inkscape to install and uninstall extensions while not having to write our own dependency software.
If we take advantage of pip/virtualenv in our .local/share/inkscape/extensions (Or do we stuff everything in .config like naughty XDG users 😉) then we can install extensions from zi p files, from git repositories on github and do uninstalls by just managing the pip repository from the inkscape UI.
This will install all the required dependencies (numpy anyone?) and keep files separated.
Installing executables in your home directory is a security nightmare. We really don't want to be in that business. And I have doubts that'll work in Windows, but there may be some magic there I don't know about. PIP does some fun things with compiling executables and libraries when it needs them.
I hate to be stop energy on porting the python extensions to Python 3, that clearly needs to happen. But I don't think we're ready to have them shipping and developed out of a different repository. The distribution issues are critical there. I think that we need a test extension that works well in all the repos/stores/etc before we can split out the extensions that we have.
Ted
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
On Fri, Mar 23, 2018 at 12:28:14AM +0100, Maren Hachmann wrote:
Am 23.03.2018 um 00:08 schrieb Bryce Harrington:
- I've often seen them using git submodules. For example, MyCroft AI
(smart assistant), pelican (python static website generator) and Atom (editor) use this approach for their (partially user-contributed) extensions (but they do have the extensions in a separate repo, and pull **those** in via submodules).
I seem to remember though that this thought had been rejected due to the complexity it brings. Currently, I wouldn't know how to work with it either, but probably it wouldn't be too hard to find out.
They can make a lot of sense for projects where the primary distribution mechanism is git itself. However, the vast majority of our users are using packages (heck they've probably never even heard of git).
(Part of my earlier concern with git submodules was that in transitioning from bzr to git, we were imposing a bit of a learning curve on our developers, and adding git submodules would increase the complexity of the transition. I suspect now, though, everyone is conversant enough in git, that this particular concern is no longer really a factor. Of course, even so it might end up being more complication than benefit, but I don't think we necessarily need to rule it out just out of principle if we have a good situation where it would make sense.)
Bryce
If we come to separate extensions, git submodules sounds like a reasonable way to go. We could also extend the approach to other things like 2geom and adaptagram libraries, where we want to stay up to date as long as they are not packaged independently.
Also re virtualenv for python, iirc there may be a way to just provide a pyvenv.cfg file ?
People,
On 2018-03-23 10:08, Bryce Harrington wrote:
It's good to see progress starting on this. The roadmap lists this as a task for 0.93, and this is a good point in the release process to start hacking on it.
As Ted and Eduard point out, the task may sound simpler than it really is. This might be one of those types of things where the hard part is coming up with a plan of attack. It's good that some actual work has been done so there's something to focus critique on. I do agree we need to find consensus on an approach before moving from experimentation to integration of the change into the main codebase.
I'd like to see some additional thought into how this is hosted in gitlab. I.e. it may make sense to establish a subteam that owns the extensions, so that for example contributor permissions can be managed differently than we'd want for the core codebase. I don't really have a solid idea in mind here, but perhaps we can brainstorm a bit next week?
As an old school Linux packager, the proposed approach feels natural to me, but Ted and Eduard make good points that it may not be as clean a fit on other platforms that Inkscape needs to be on. It may be worth doing some analysis into how similar software projects solve this problem. It's entirely possible there is no perfect solution that would address everyone's desires - but I'm hopeful we can at least find an approach that is at least workable.
One difficulty here is that if we provide a distribution/installation system for 3rd party extension writers to use, then if we change that system they'll be quite grumpy. Yet getting all the requirements satisfied in one go would be a formiddable challenge.
It may benefit us to do this in a few stages. For instance, a first step might have the extensions housed in their own gitlab repository and subteam, but still be integrated into the source tarball at release time, thus sidestepping all of the packaging/platform problems but gaining some of the team membership flexibility. A next step would focus on modularizing the extension installation, but keeping distribution centralized as we do it now (i.e. everything is released in concert with the main release). A later step would work on modularizing distribution to work asynchronously, so users can install and update modules at will.
Sorry to distract from the fairly heavy-duty main discussion but I thought a comment on a particular extension might be useful: Some time ago I needed to create animations from SVGs and while I was able to work out how to do this in a basic way with Inkscape, I would like to see this area improved upon with a more sophisticated extension. I am probably not up to doing any coding but I am certainly happy to help with anything else that would speed up the process.
Let's definitely plan on talking about extensions and modularization in general at the hackfest.
Maybe improving animation could be a test case?
Regards,
Phil.
Bryce
On Thu, Mar 22, 2018 at 12:15:42AM -0500, Ted Gould wrote:
On 03/21/2018 10:54 PM, Martin Owens wrote:
Firstly we put any non-python extensions into their own repository. We don't want to drop support for just running weird shell scripts and perl stuff, but I'd put it in auxiliary and ask packagers to add them to separate packages.
Frankly, you're thinking like old school Linux packaging. Multiple packages with dependencies isn't really how most modern OSes are thinking right now. So I think that relying on that is perhaps a play towards the past.
I'd agree on the goals of making it so that an extension can be published separately, on a separate schedule, and work well in Inkscape. But, I don't think we're to the point of that working well with things like dropping to command line instructions, much less being user friendly enough to be practical.
This is so we can focus the core extensions repository on making it a python installable module set with the inx coming along as data.
The reason we'd want to move in this direction is really to allow inkscape to install and uninstall extensions while not having to write our own dependency software.
If we take advantage of pip/virtualenv in our .local/share/inkscape/extensions (Or do we stuff everything in .config like naughty XDG users 😉) then we can install extensions from zi p files, from git repositories on github and do uninstalls by just managing the pip repository from the inkscape UI.
This will install all the required dependencies (numpy anyone?) and keep files separated.
Installing executables in your home directory is a security nightmare. We really don't want to be in that business. And I have doubts that'll work in Windows, but there may be some magic there I don't know about. PIP does some fun things with compiling executables and libraries when it needs them.
I hate to be stop energy on porting the python extensions to Python 3, that clearly needs to happen. But I don't think we're ready to have them shipping and developed out of a different repository. The distribution issues are critical there. I think that we need a test extension that works well in all the repos/stores/etc before we can split out the extensions that we have.
Ted
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Inkscape-devel mailing list Inkscape-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/inkscape-devel
participants (8)
-
Bryce Harrington -
Eduard Braun -
Marc Jeanmougin -
Maren Hachmann -
Mark Schafer -
Martin Owens -
Philip Rhoades -
Ted Gould