MIDI Processor Inconsistent/Processing Speed Issue?
-
Tested in a DAW (Reaper) with the HISE plugin armed for recording -> single threaded processing in Reaper. Happens in Standalone, too, but I have to keep hammering until I get the time interval correctly. I moved to the plugin version of HISE in order to track down this bug. So here it is.
I have a Play Logic script right underneath my Interface script, where I do all kinds of math to determine the correct group to play (the correct string of a guitar).
This Play Logic script ignores the physical note on, and creates artificial note ons accordingly. These then get passed to the container containing a few samplers. One is the guitar sampler, where all the articulations are stored, others are for noises, stops etc. Only the main guitar sampler cares about note ons. Others care about noteoffs.
So I play a note, it goes into my Play Logic script, I crunch some numbers, determine the correct string to play (the correct Active Group to be set), and execute my Synth.playNote methods.
The artificial note now goes into the container where it duplicates to all of its child samplers. All the others ignore it, and the Guitar sampler executes it. Now, there's a bit of number crunching in there, too, but I ran the test with a newly made empty script processor and the results are the same: if I play a chord and the difference between two notes is less than 2-3ms, the Play Logic will execute twice (for both notes) before the first note even gets to the Guitar sampler and its MIDI processor.
You can see that the last chord is the problematic one. The Play Logic NoteOn gets executed for both notes before the script processor in the sampler below even hears of the first NoteOn.
How or why is this happening? How is it possible that one MIDI event does not get to go through the entire tree before the other starts being processed?
I do not have any message delay method calls in the onNoteOn for Play Logic.
The obvious workaround is to split the Play Logic script properly and script the logic individually for each sampler. But help me understand this issue so I can work around it correctly in the future.
-
How is it possible that one MIDI event does not get to go through the entire tree before the other starts being processed?
That's in fact how it works. The reason for this is that containers in HISE are pretty "close to the metal" and process the entire audio buffer including the timestamped MIDI messages at once (that's also the reason why you can't use polyphonic effects on the container level) and depending on how you communicate downstream, it might be possible that a message with a future timestamp within the same audio buffer will be processed in the root container before its child modules.
I'll try to sketch the coarse system here (this is not my invention but basically how the JUCE synthesiser base class works):
Basically, you get an audio buffer from the DAW with a bunch of audio samples you need to fill (eg. 512). Alongside the audio buffer you get a list of MIDI messages that are timestamped within that buffer (so if you play a fast arpeggio, there might be a note with the timestamp 16 and a timestamp 384 and the scenario that you describe (2-3ms) is pretty much that. The containers will do these things
- Process first MIDI message at timestamp 16
- Process second message at timestamp 384
- Pass the audio buffer and the MIDI messages to the child sound generators
Now it's the job of the sound generator to allocate / stop voices based on the time information and the MIDI data. So the sound generator (sine wave-generator, sampler or whatever), splits up the buffer depending on the timestamps of the midi messages into a serialized list of actions. In our case:
- process 16 samples of silence
- start a new voice at timestamp 16
- process 368 samples
- start a new voice at timestamp 384
- process 128 samples
This ensures a sample accurate voice management. However as you can see, step 2 is executed before step 5 so the second message was processed in the root container before the first message in the sound generator.
Hope that helps. How to apply that newly-found knowledge to solve your particular problem is up to you :)
-
@Christoph-Hart Great write up, thanks a lot, exactly what I was looking for.
I think I'll try splitting the whole thing up and writing the logic separately for each sampler, then see how that goes.
The main issue I'm running into is the Play Logic (which gets executed before -- step 2 in your example) determines the string/group, and that needs to be on the level where it'll be executed directly instead of being passed downstream.
Will report back!
-
@Christoph-Hart I rewrote my play logic script into smaller individual scripts for each sampler. For the guitar sampler, active group logic is now done in the script processor there.
However, the issue persists. Although now I can't get a "console confirmation" of it being wrong.
Here's a chord. Timing is tight, but not perfectly on grid.
The console is printing the note to be played and the active group, both print methods are called right after Synth.playNoteWithOffset().
Except the first note doesn't play, because the second note's setting of active group to 4 overrides the active group for the first note. Since it's a guitar, the active group (string) is set to a higher number, and there's no sample for the lower string (2). I confirmed this by setting a static active group to a lower string (in that case the chord plays as expected, within the active group).
So what it looks like is that the noteOn callbacks for both notes get processed before either voice gets started, resulting in the wrong active group being set for the first voice (overridden by the second note/callback execution).
-
Here's a snippet. The sampler has two groups. Put a sample in group 1 on D1, and a sample on group 2 on A1. Then play the following MIDI file.
The console should output the correct groups as set in the sampler's script processor. But only the second group (A1) will play.
HiseSnippet 1191.3oc6Xs0aaaCElxNrsxqqXsqOrG1CBA6Aazh.6bqEqHXNWKBVchgUVwdqfQh1lHRjBjTYKas+m1Og9SZ+C1NTR1hx0oI0Ks6BleHvmajemK7bNN8kh.pRIjHG2StHghbtK1+Btd7tiILN5v8PN2C2inzToWNqctHgnTzPjiS8maX33tDJ6yu+c6PhH7.ZIKD5kBV.8ErXltja+teOKJ5.RH8DVrk1q28v.AeWQjHEvScbaTBI3LxH5QDiZ0vHmaseHSKj9ZhlpPNKsiH7B+whehmq+KYJ1oQTCQGjObP4rOPDEZPrgKZ2wrnv9S7aEB4f6WFEpmGEdHtGKjMkeYz3KxD3UZgc7voVU3UuB75XCu1VvaNPxwBRKkCo6i8CjrDcoDCd9L7gbH4Lj.gcanjqKx4s3cEfBb8JwjynGHAhoFzby1serG7mVOaXJOPyDbOA+HgldLuYqF+ZC2Fuog2rhFNbtxLWiTDEQkyUrISKeeF1jmFeJU9XuyIQozoJBte0XJ9xio1o7fbu1RQA+PNSebBkeYEBnhPE7se3v8HZhIQTvCzKgJ0LCDb1idNTUmmVbw6QUmoEIPc86jyfpEQXZDQWsDx7toP.DCpj2LIGthouv9c0MVc00Eh2G2moCFOeLVaNXDhTeLvXwqwOGu+vgz.cI.WBevO9w4om80+04W+Cv9ZIkDy3i7IwIPIdFFZfKn5bSzTL4Z2TrujFIHg9rewRs21cmTHFIqx825lcUaGKR4UtqBjOflPIZHMaYyc5NXvykhzjYsB0Mqn3DIzTFBE193wbJDqs0EsqTnTCA2I6rT1h5mJGYhUkbFPOmJUU4cTZLju4bZj4Uuii4UohZxurfdDsj8y1ZmcKsOgjkigIEYzclgd0YnWaF50mgdiYn2bF5mTR6fqV+c6qWapdjj9DvEszbGVQUFHKmi83qog0rq1Zz2bFhbq+EOW6KK3VhnNya9VsaWCCOOTZXrgzax6Qus7xVhYkQTcAulKOQ3xsdViIeeEJ2DFG.E5gCDmx3MGRhTzKcdHanWyd.ZfcRLmsQzQYisZ1xaqs7V6osZ3Bp4N87UT81vAcd9qflcf618MMbgZZp2UbXquwUbXqZeXueUaWnZCWXblRDQWIQx35K+9ej2xdu90dYV+sv2ezzXqQ07itnMQyVvg++6I7ep8Dp7N0cBF8YlRf84mCye.NFL9.vqFRRizS3V8UZOAWjLVvYAU61CcuGMhJsw9bcns0ZXXSImG1c.MhRTViq9ltufwoDY0gXePwhNev8rla95qv4v0yTC68WIusz0Ku89+YM2H65U+Zsq2MNdW7899jF5tWwFfdvsK0+COOeWbV6ZOyJl+cAUXPqFVcbx5aXXdiOrWa.0dSuZlNr4zsMzF.4S4gYD+A7oPXmIqEZD1YhPTvjiB7shum0i1od9v9reBSDrvYjcIsB1d1.IXUNjJKoFaV+xobMVRbLwNB8IwYhIARwqBxG8Ybo6jwAbFd1+SDWbOCsWGT13PaOJF1m6UAAUOp2wvUWTCWaQMb8E0vMVTC2bQM7IKpgO8pMzrt81oZQbdOWDpW+8yqSc1eZcZczehWjuE7Since we're talking buffers, this is 48k 256 samples.
-
@aaronventure wow, that is a real problem within the engine, wouldn't have thought :)
The MIDI file contains a few MIDI messages with the same timestamp, and the internal MIDI processing is processing those right after another, however this means that the
setActiveGroup()calls are being called immediately after another, so only the last group index is active when the voices are started a little bit later.There are a few options of solving this problem but I'm a bit surprised that this only pops up now after years of HISE usage :)
-
@Christoph-Hart said in MIDI Processor Inconsistent/Processing Speed Issue?:
but I'm a bit surprised that this only pops up now after years of HISE usage :)
Well yeah, me too :D
This fundamentally breaks the sampler RR group behavior in terms of selective group usage.
I didn't make the MIDI in the DAW, I cut this out out of a sequence of D5 chords, where 3 out of 15 or so behaved like this, so it's played behavior. There's all kinds of stuff that you can do with MIDI editors today that the user would never actually encounter, but this is one of the simplest things, just an imperfectly perfect chord.
-
@aaronventure yeah I guess there wasn't a project yet where the error of a RR group mismatch caused a "critical error" like this - usually all that happens is that it uses the same RR group for two different notes if they are played exactly at the same time (and only if you don't use the inbuilt RR counter as this calculated right before the voice start). Within a normal RR sample set this is hardly noticeable, but if you rely on it working with mathematical precision because the note ranges do not overlap, you're in trouble.
I'm currently working on a solution for this - I can query the note number of the current MIDI event during the onNoteOn callback and then store the active group into an array with 128 items so it will get picked up later by using the note number when the sample selection takes place, but this might cause some edge cases where you start transposing the MIDI message after calling
setActiveGroup().There's also the issue of being able to enable / disable multiple groups, which needs to be handled separately (and there it's not so easy because the data structure is already an array so I would have to make a 2D array which then grows to a non-trivial size.
-
@Christoph-Hart said in MIDI Processor Inconsistent/Processing Speed Issue?:
array so I would have to make a 2D array which then grows to a non-trivial size.
Rather than a 2D array could you not add a flag to the data stored in each element of the current array you're using?
-
@d-healey Or maybe I'll add a (short) queue of events with the required action attached to it. During the MIDI processing inside the buffering this gets populated and then used later by the voice allocation. At the end of the buffer the queue can be cleared because the problem only arises with multiple messages within a buffer.
-
@Christoph-Hart said in MIDI Processor Inconsistent/Processing Speed Issue?:
yeah I guess there wasn't a project yet where the error of a RR group mismatch caused a "critical error" like this - usually all that happens is that it uses the same RR group for two different notes if they are played exactly at the same time (and only if you don't use the inbuilt RR counter as this calculated right before the voice start). Within a normal RR sample set this is hardly noticeable, but if you rely on it working with mathematical precision because the note ranges do not overlap, you're in trouble.
I can see how this could go unnoticed. If you have different articulations in different groups, you trigger the keyswitch which sets the active group and it just works, you're not setting activeGroup on each noteOn. If you're using it for RR, it just skips a group and 3/15 for a group skip is still plenty of variance unless you have like 2 RRs.
Is it fine if I just call setActiveGroup() at the very end of the noteOn callback?
I'd like to point out that in my actual project, I'm ignoring the event in the guitar's noteOn, and then have a Synth.playNoteWithOffset because I need it to play up to 5 notes (based on multitracking settings), which are all neighbouring zones. So when I press a C3, it plays B2, A#2, C#3 and D3, and the pitch modulator checks for the IDs and applies correct pitch modulation to make them all sound at the correct pitch.
Back when I had this in the playLogic script which was above the samplers, these Synth.play method calls produced 5 artificial noteOn events in the sampler.
I don't know what that looks like when the Synth.play happens in the sampler itself.
I am storing the IDs of the Synth.play calls, though. Is it a good idea to maybe have another method which sets the active group for a given event ID?
-
hmm, that makes it even more complicated and I would have to change the API for the
setActiveGroup()call to take in an event ID.On the other hand this would leave the "default" path as it is right now without overhead but the slight inconsistency that the active group will get overriden for each event in the current audio buffer (and chances are great that if you go into this kind of advanced event modification & RR group control you want it to be tied to the event ID properly anyways).
So in this case I would add a new API method
Sampler.setActiveGroupForEventId(int eventId, int groupIndex).which will take the event ID and then check in the note on callback of the sampler which group to use for the current group index.
-
@Christoph-Hart exactly, yeah!
Great.
Don't forget warning for any clowning around with usage of both this and the current setActiveGroup method in the same callback (or will one simply override the other?)
-
@Christoph-Hart said in MIDI Processor Inconsistent/Processing Speed Issue?:
Sampler.setActiveGroupForEventId(int eventId, int groupIndex).
Also, any chance this bad boy could also be taking an array of indices in its second parameter? That'd be two flies with one strike.
-
@aaronventure I think you can just use a for loop for that, I would avoid branching inside the API call for your special use case :)
-
@Christoph-Hart Really? Wouldn't another call override the previous one?
I meant what if one wanted two different groups active for a note? I mean, sure, that can be another
Synth.playNoteand its ID passed into anotherSampler.setActiveGroupForEventId(int eventId, int groupIndex)call. -
@aaronventure Ah you mean the second parameter, I thought you were just too lazy to make a for loop for your 5 events that you want to spawn.
For this, I would rather add another API call that's similar to
setMultiGroupIndex()as this would be the non-event-ID-agnostic counterpart in the Sampler API. -
This post is deleted! -
@Christoph-Hart haha isn't looping the lazy way?
Just so we're on the same page, I play a note, I ignore the event, then I execute 5 notes via
Synth.playNotein a loop, storing their ID, and right in the next line (still within the loop) I just pass that ID into theSampler.setActiveGroupForEventId?So it would be possible to have multiple Sampler.setActiveGroupForEventId calls within a single noteOn callback, each directing its own eventId to the appropriate group?
-
@aaronventure yes, that should be possible.
