Also, worth remembering... a compiled plugin CANNOT use the builder - at least for the most part. So what I've also got is a PluginStateData namespace, which maintains a key:value dictionary of module id's to module references. This runs on plugin instantiation, retrieves all of the actual object references, and stores them in a const dictionary.
You should only use the builder to build the module tree. You should not use it for retrieving or storing id's or object references.