I think what we need is a new Alt+drag mode
which would mean "drag what is selected now, without selecting
anything else no matter where I start the drag."
I implemented this in CVS. So here are the release notes for both features:
* Select Under: in Selector, clicking with Alt selects the object at
cursor which is below (in z-order) the currently selected object at
cursor; if the bottom object is selected, Alt+click again selects the
top object. Thus, several Alt+clicks will cycle selection through the
z-order stack at the click point. Combining Alt with Ctrl ("select in
groups") and Shift ("add to selection") works, too. Note that on
Linux, some window managers steal Alt+click by default; reconfigure
your WM so you can use Alt+click in Inkscape.
* Drag Selected: dragging with Alt in Selector moves the currently
selected object(s) no matter where you start the drag, unlike regular
drag that first selects the object under cursor. This is convenient
for dragging objects that are behind other objects in z-order. On
Linux you may have to disable dragging the window with Alt if you want
to use this.