@d-healey that's right, envelopes and other modulators are not processed on the channel level so you get a single modulation signal for every voice so even with polyphonic channel routing you would still be limited by this design.
You'll probably need to look for another way to achieve this, but usually once you apply different modulation sources to voices it's time to duplicate the samplers.