2 a.m.
Hi,
Just wanted to point out that technically URI classes are not usable
for our assets.
For example, it is tempting to use an URI to resolve an asset to the
base document path. The problem is that a URI according to the RFC's
is not usable for file paths. Glib and friends properly implement URI,
therefore we can't use them for files.
In the future we can probably use IRI's or our own classes for
references and assets, etc. However URI's are not usable.
I'm pointing this out since we are just about to release, but I
tracked down a critical bug to being caused by this. In the near term
we should search through the codebase for improper uses of URI
classes. The tricky thing is that they will appear to function since
common path names are "safe." However as soon as unexpected characters
come in (like ampersands), all bets are off.
To summarize:
If we are linking something that is represented by a file (images,
color profiles, etc.) can't be processed with a URI class or URI
functions. It will work for simple cases, but not all.
7:57 a.m.
On 9/29/2009 1:00 AM, Jon A. Cruz wrote:
Hi,
Just wanted to point out that technically URI classes are not usable
for our assets.
For example, it is tempting to use an URI to resolve an asset to the
base document path. The problem is that a URI according to the RFC's
is not usable for file paths. Glib and friends properly implement URI,
therefore we can't use them for files.
In the future we can probably use IRI's or our own classes for
references and assets, etc. However URI's are not usable.
I'm pointing this out since we are just about to release, but I
tracked down a critical bug to being caused by this. In the near term
we should search through the codebase for improper uses of URI
classes. The tricky thing is that they will appear to function since
common path names are "safe." However as soon as unexpected characters
come in (like ampersands), all bets are off.
To summarize:
If we are linking something that is represented by a file (images,
color profiles, etc.) can't be processed with a URI class or URI
functions. It will work for simple cases, but not all.
Jon,
(for those uninterested in URI's, please ignore :-)
I suspect there is a problem either with the URI impl or how it is being
used.
But I'm kinda pro URIs. Yes, the URI API as stated in the W3C doc does not
have a getNativePath() method as far as I know. However, Java's
runtime, XML2, and
other libs -do- include such a call, and it is quite useful. I think it
has become a defacto
requirement.
In conjunction with resolve() and normalize(), you should be able to
handle paths in a -predictable-
manner. Maybe not what you expect, but always the same.
As far as IRI's go, in the specs themselves IRI's are described as URI
classes which have been
hacked to be able to handle Unicode and international information in the
full resource paths. We
could take either of the two URI impl's in Inkscape and hack them to
work nicely in just
a few days' work. I just call them all URI's whether they can handle
IRI stuff or not.
The case that you mention, an asset pointing back to the owner doc. Is
that feasible, or
even promised anywhere? I can easily imagine that resolving owner
doc->resource can
only be guaranteed 1-way. Especially considering symlinks.
Once again, the "sandbox" comes up. It would be a wonderful thing if we
had docs operate in
a JAR-like sandbox. If you import an image, it gets copied into the
sandbox with the
SVG doc (unless you request otherwise). HTML editors work this way. The
links within the
sandbox are simple and predictable.
And when you JAR the directory into an archive, the META-INF directory
and manifest
are already ready to go.
This is something we needed for collaboration, too.
bob
2:24 p.m.
On Sep 29, 2009, at 4:57 AM, Bob Jamison wrote:
I suspect there is a problem either with the URI impl or how it is
being
used.
But I'm kinda pro URIs. Yes, the URI API as stated in the W3C doc
does not
have a getNativePath() method as far as I know. However, Java's
runtime, XML2, and
other libs -do- include such a call, and it is quite useful. I
think it
has become a defacto
requirement.
Actually Mental and I both have looked into this a fair bit. URI as
defined by the spec is not really usable for arbitrary filenames.
Java's implementation is interesting, but also falls down. When doing
some Java work a while back I recall hitting some of those issues.
For the glib world, I seem to recall that these break down:
g_filename_from_uri()
g_filename_to_uri()
One of the trickiest things to deal with is converting to and from
file names when those names are *not* in UTF-8. We need to remember to
keep those in our testing, as all data internal to Inkscape needs to
be Unicode + UTF-8.
Basically I recall that IRI's should work nicely for our needs, with
an implicit assumption that they will work as URIs for most APIs we
care about. But using IRIs will keep us from failing in some edge cases.
2:46 p.m.
2009/9/29 Jon A. Cruz <jon@...18...>:
For the glib world, I seem to recall that these break down:
g_filename_from_uri()
g_filename_to_uri()
On what cases? Why?
One of the trickiest things to deal with is converting to and from
file names when those names are *not* in UTF-8. We need to remember to
keep those in our testing, as all data internal to Inkscape needs to
be Unicode + UTF-8.
On Linuxes the filename encoding is usually UTF-8 (e.g. anything newer
than 5 years). On Windows the filename encoding is UTF-8 by
definition. However, the arguments from the command line are in locale
encoding (different from filename encoding), which is usually a legacy
8-bit codepage - this is the only place where the conversion is
non-trivial, and the arguments of main() do not actually contain
sufficient information to determine what files were actually meant -
we should therefore never use the arguments of main() for *anything*
on Windows.
The problem I see with keeping everything in UTF-8 is that it's a
convention unsupported by Glib. Glib has functions to work with native
filenames and URIs, but not with native filenames converted to UTF-8.
We should just use URIs everywhere; this would add network
transparency via GIO, as we wouldn't be limited to local files.
By the way, there is some code that deals with versions of Windows
that do not have the wide versions of Win32 API functions (Windows 95,
Windows 98 and Windows ME) - this is totally superfluous since the
version of Glib we depend on does not work on such systems.
Regards, Krzysztof
2:48 p.m.
On Sep 29, 2009, at 11:46 AM, Krzysztof Kosiński wrote:
On Linuxes the filename encoding is usually UTF-8 (e.g. anything
newer
than 5 years).
Actually that is not quite accurate.
The filename encoding is UTF-8 on *new* systems.
When a user upgrades a system, it keeps the filename encoding. We have
several core Inkscape developers who have this issue. It is also far
more prevalent for corporate users.
3:05 p.m.
On 9/29/09, Jon A. Cruz <jon@...18...> wrote:
When a user upgrades a system, it keeps the filename encoding. We
have
several core Inkscape developers who have this issue.
If you refer to me, then I have just installed a shiny new UTF-8
Ubuntu on a new machine after my old one died. So, until a couple
weeks ago, indeed I had KOI8-R as my filename encoding, but not
anymore.
--
bulia byak
Inkscape. Draw Freely.
http://www.inkscape.org
3:19 p.m.
On Sep 29, 2009, at 12:05 PM, bulia byak wrote:
On 9/29/09, Jon A. Cruz <jon@...18...> wrote:
> When a user upgrades a system, it keeps the filename encoding. We
> have
> several core Inkscape developers who have this issue.
If you refer to me, then I have just installed a shiny new UTF-8
Ubuntu on a new machine after my old one died. So, until a couple
weeks ago, indeed I had KOI8-R as my filename encoding, but not
anymore.
Yes, you were generally taken as the pathological worst-case, since
your filesystem also had files encoded in a few encodings. Was CP-1251
in there at some point?
But I've seen others. At least some of my boxen have been upgraded,
and one common place to see "interesting" file system encodings has
been Japan. There were many encodings used over there, and I've
personally hit support issues with people having other than UTF-8
filesystems on boxes running modern distros that list "UTF-8" as their
default filesystem.
The good news, though, is that as long as we do the right thing it
will work 100% for everyone, including those with UTF-8 filesystems.
It's minor work to do the right thing, and actually more work to do it
wrong.
3 p.m.
On Sep 29, 2009, at 11:46 AM, Krzysztof Kosiński wrote:
The problem I see with keeping everything in UTF-8 is that it's
a
convention unsupported by Glib. Glib has functions to work with native
filenames and URIs, but not with native filenames converted to UTF-8.
We should just use URIs everywhere; this would add network
transparency via GIO, as we wouldn't be limited to local files.
No, you're missing the point.
The only sane way to craft an application is to have a consistent data
approach internally. The alternative is to have each individual string
also marked with its encoding and do multiple conversions on the fly
as things are processed. (Look to both early Windows 95 APIs and older
Perl versions for examples of this).
The simple way is to say "All data inside the program is Unicode".
Volia! Problem solved.
Now Unicode data can be encoded in many ways (here we hit the
important difference of "encoding" vs. "character set"). Unicode can
be UTF-8, UTF-7, UTF-16BE, UTF-16LE, UTF-32, etc. Some data can be
UCS-2, but that is *not* the same as UTF-16 and can burn people who
don't realize it.
On MS Windows, they went with UTF-16 as the standard. They also
defined wchar_t as 16-bit when everyone else in the world followed the
standard's recommendation and implemented it as 32-bit. Java also uses
UTF-16 as far as the programmer can tell. IBM's ICU also uses UTF-16.
On Linux, however, and with GTK+ the base encoding is UTF-8.
Everything we do on the UI *must* be in UTF-8. Therefore it makes
sense for Inkscape to use UTF-8 as its Unicode encoding of choice.
You mention filename encoding, but miss some issues. First and
foremost is that as far as Inkscape is concerned each and every
filename *must* be presentable in the UI anyway (MRU lists, titlebars,
media tracker list, etc.). Therefore we have to be able to handle
UTF-8 for those. There are also safe round-trip conversions for *most*
of the user scenarios. Therefore it vastly simplifies our code to just
keep internal data in a single consistent format - Unicode.
And conceptually URI's don't actually even support Unicode. What the
authors of those API's you cite have done, though, is follow RFC-3987
and implement their URI's *as* IRIs'. Thus we are 100% compatible with
those API's you care so much about as long as we properly support IRIs.
By the way, there is some code that deals with versions of Windows
that do not have the wide versions of Win32 API functions (Windows 95,
Windows 98 and Windows ME) - this is totally superfluous since the
version of Glib we depend on does not work on such systems.
For years this code was not superfluous. It only became redundant
later on. Thus it now can be safely dropped, but only as long as care
is taken in cutting it out.
Also early on Inkscape implemented code that did not exist in Glib and
was adopted by the Glib maintainers because of us.
2:27 p.m.
2009/9/29 Bob Jamison <ishmalius@...400...>:
On 9/29/2009 1:00 AM, Jon A. Cruz wrote:
> Hi,
>
> Just wanted to point out that technically URI classes are not usable
> for our assets.
>
> For example, it is tempting to use an URI to resolve an asset to the
> base document path. The problem is that a URI according to the RFC's
> is not usable for file paths. Glib and friends properly implement URI,
> therefore we can't use them for files.
We can, and in fact we must in order to support editing directly on
network shares (GIO). The convention used by absolutely everyone is to
use the file:/// scheme for local files.
> In the future we can probably use IRI's or our own classes
for
> references and assets, etc. However URI's are not usable.
The only difference between URIs and IRIs is the set of allowed
characters. You can use Unicode characters in IRIs without escaping
them, but you can use Unicode characters in URIs as well, you just
need to escape their UTF-8 representation with %xx.
http://en.wikipedia.org/wiki/Internationalized_Resource_Identifier
But I'm kinda pro URIs. Yes, the URI API as stated in the W3C
doc does not
have a getNativePath() method as far as I know. However, Java's
runtime, XML2, and
other libs -do- include such a call, and it is quite useful. I think it
has become a defacto
requirement.
Seconded. URIs might not be originally intended for file paths but it
is a de-facto standard to use the file:/// URI scheme for local paths.
It is even used in the clipboard in GTK.
The case that you mention, an asset pointing back to the owner doc.
Is
that feasible, or
even promised anywhere?
SVG provides mechanisms to embed the necessary data (e.g. images) in
the SVG itself, so I don't see why we need this.
And when you JAR the directory into an archive, the META-INF
directory
and manifest
are already ready to go.
This is something we needed for collaboration, too.
Multipage as well. But we should also fix embedding data in SVG (the
data URI). For example, pasted and imported images should be embedded
in the document by default. There must be a separate option to embed a
link to the image, and there must be a clear distinction between those
two concepts.
Regards, Krzysztof
2:47 p.m.
On Sep 29, 2009, at 11:27 AM, Krzysztof Kosiński wrote:
We can, and in fact we must in order to support editing directly on
network shares (GIO). The convention used by absolutely everyone is to
use the file:/// scheme for local files.
No...
There are many things that are *close*
We want 100%, not 95%
So for our internals we must use something that is consistent and also
covers 100% of the use cases we need. As we touch each API, we can
either feed something in directly (as I mentioned, both Mental and I
have researched this URI/IRI issue) or we can do a quick conversion
per API.
Also... at runtime we might need to track some additional data. Bob
mentioned some additions to URI classes that people do in order to
make things work. That is what we need to do. However, instead of
having redundant code all over the place doing potentially buggy
conversions, we need to have a consistent class that we use internally.
To quote RFC 3987 "which means that IRIs can be used instead of URIs,
where appropriate, to identify resources." Later on it also mentions
"Every URI is by definition an IRI."
You also seem to not be grasping what I'm trying to convey. Let me try
to clarify:
* URI's *can* be used for all sorts of things, but as is the standard
library calls for conversions *fail* in our use cases.
* There are many ways to tweak data in and out of URI's to make them
more useful
* RFC 3987 codifies the best practices of "useable URIs" and calls
them "IRIs"
* We need to use IRIs internally
* We need to use a class that is more than just a bare string
* We should use them to reference files
5118
days inactive
5118
days old
inkscape-devel@lists.inkscape.org
9 comments
4 participants
participants (4)
-
Bob Jamison -
bulia byak -
Jon A. Cruz -
Krzysztof Kosiński