If the selected object is bigger than the viewport, could the handles be drawn at the viewport's edges, and not at the object's outside of viewport bounding box like currently ?
While that's better than it currently is, depending on how large the graphic is it could still be a lot of resizing still, click, drag, click, drag. I'd still prefer a hotkey to resize the selection to view (centered, minus about 10% of the screen on each side, so it's easy to grab the handles.
One other thing that may work quite well is using the Ctrl+Alt keys along with the mouse wheel to scale the current selection down. That requires a mouse wheel, or multitouch trackpad though.
Even with that added, I'd still like a hotkeyed Fit to View. That covers you in case the very large imported/pasted graphic is off-center by a lot.
-C