perhaps this would make sense to me if the defs list was shown in a way that users could easily access and clearly shown to the user as the library/palette that conveyed that these were an abstract set of resuble symbols, gradients, and patterns, associated with this document.
That's what we have. Open fill & stroke and you'll find the selectable lists of the document's gradients, markers, and patterns. (Two of the three even have previews in the lists.)
if Inkscape had a 'Format' menu, that would seem like a better place to put it. would it make more sense to have "Clean up defs" under 'Object' (objects are what defs apply to) rather than File?
No. "Object" is for per-object actions. "File" is for document-wide actions. Vacuuming defs is a document-wide action.