Obviously compile with empty CXXFLAGS and CFLAGS (i.e. no optimization, no debugging annotations), and use ccache as you mention. These won't help link time, though.
To slightly speed up link time at cost of compile time proper, you could set CXXFLAGS and CFLAGS to -Os (just to reduce I/O). Possibly using -O/-O2 instead of -Os would speed up linking by doing more inlining during compilation, reducing the amount of symbol lookup needed during linking; I haven't tested.
I gather that the development version of binutils includes the option of using a much faster (more specialized) linker called `gold'. It's disabled by default for now; enable it by configuring with `--enable-gold'. People are reporting that it runs 5-6 times faster for sizable programs (such as inkscape), and that it seems to work correctly for linking everything but Linux. (Linux source makes heavy use of ELF features, and it even has some freedom to misuse ELF, whereas I don't think Inkscape source makes any special use of ELF or even cares what object file format is used.)
pjrm.