If the selected object is bigger than the viewport, could the handles be
drawn at the viewport's edges, and not at the object's outside of
viewport bounding box like currently ?

While that's better than it currently is, depending on how large the graphic is it could still be a lot of resizing still, click, drag, click, drag. I'd still prefer a hotkey to resize the selection to view (centered, minus about 10% of the screen on each side, so it's easy to grab the handles.

One other thing that may work quite well is using the Ctrl+Alt keys along with the mouse wheel to scale the current selection down.

That requires a mouse wheel, or multitouch trackpad though.

Even with that added, I'd still like a hotkeyed Fit to View. That covers you in case the very large imported/pasted graphic is off-center by a lot.

-C