I think what we need is a new Alt+drag mode which would mean "drag what is selected now, without selecting anything else no matter where I start the drag."
I implemented this in CVS. So here are the release notes for both features:
* Select Under: in Selector, clicking with Alt selects the object at cursor which is below (in z-order) the currently selected object at cursor; if the bottom object is selected, Alt+click again selects the top object. Thus, several Alt+clicks will cycle selection through the z-order stack at the click point. Combining Alt with Ctrl ("select in groups") and Shift ("add to selection") works, too. Note that on Linux, some window managers steal Alt+click by default; reconfigure your WM so you can use Alt+click in Inkscape.
* Drag Selected: dragging with Alt in Selector moves the currently selected object(s) no matter where you start the drag, unlike regular drag that first selects the object under cursor. This is convenient for dragging objects that are behind other objects in z-order. On Linux you may have to disable dragging the window with Alt if you want to use this.