What about something like libdepixelize? Although the name does not exactly mention pixel art, it seems it does not yield any hits in google yet.
My concern was about the name of the symbols/variables/classes/..., not the library per se.
The library could implement (as you stated) other algorithms in future and should use a generic name.
The potrace tool ( http://potrace.sourceforge.net/ ) uses the name tracing to refer to this technique. I like both names (libtracer, libdepixelize) and other names also (libvectorizer) and whatever you guys choose is fine to me.