I guess what I'm trying to point out is that instead of focusing on
an optimized list structure,
The list structure is only a means to an end, which I outlined at the
beginning: to be able to iterate over references to an object, and to
iterate over references in an object. Those very generic capabilities will
make a lot of code simpler, and they're not really related to the structure
of the document. The concept of reference between objects will not go away
no matter how we change the SP tree.