Phaselocking 2025 and other sampler related discussions

[Lindon]

@Christoph-Hart said in Phaselocking 2025 and other sampler related discussions:

So long story short: in order to continue my quest for a perfect legato engine, please let me know these three things:

1. What are you using for pitch/phase-locking your samples and with what success rate?
2. Do you need more parameters exposed in the default legato transition mode
3. Do you need more parameters exposed for the poly legato engine?

1. Well to be honest Im in a somewhat different relationship with the audio..but it may work as a model - in that I say to the client - "your problem" and like Dave's explanation they then use some external 3rd party tool set to get the files into "as good a state" as they can and I essentially do what you've outlined above, with a bunch of manual tweaking to get it playback positions synced up as best I can - either with some global or a per individual transition rule set. Given you have a very very attractive potential solution for that part then Im all good "as is"
2. Nope all good.
3. Nope again all good.

[Christoph Hart]

@d-healey said in Phaselocking 2025 and other sampler related discussions:

How will this system work with multi-mic samples where the phase is all over the place? Will it just sync individual mics together?

I would assume that the phase relation is consistent across mic positions - it might have a different phase but since it's the same waveform that phase difference will stay constant, no?

So you can just align the first mic position and the other ones should follow suit. Ideally you would use the closest position as "guide".

Also there's still a lot work to be done on my side, one of the tasks is to calculate a more reliable way to detect the phase position that just look for a zero crossing from negative to positive as most real waveforms have more than one zero crossing per wave cycle.

[tomekslesicki]

@Christoph-Hart will it also be possible to disable the crossfade after the transition and stay on the legato transition sample? I’m thinking about a scenario where the samples are edited so that the transition sample also contains the sustian portion to loop.

[Christoph Hart]

@tomekslesicki Ah yes, that should be an option.

It currently works by assigning the "Ignore" flag to the sustain samples (so that they are played regardless of which legato transition is active), but I can add an option to not start the sustain sample of the target not if the legato transition already includes the next sustain phase.

[tomekslesicki]

@Christoph-Hart that would be super cool and flexible I think!

[Simon]

@Christoph-Hart Oh boy my favorite subject :)

First off, automatically aligning the legatos to sustains is one of those coveted features that's impossible to do in Kontakt, at least without squashing your samples to a consistent pitch (and rendering them unlistenable in the process, which you then remedy by re-applying a pitch curve). What you are working on is beautiful and super exciting.

RE: a few of your points.

Sine waves

This alignment will be most noticeable on solo instruments that are already closer to sine waves than to the chaos of an ensemble. I imagine testing with sine wavs is a totally suitable stand in, and aligning based on rising/falling zero crossings should already give very good results.

Legato Parameters

I would like to have access to:

• Attack and release time, and curve, for source sustain, legato sample, target sustain.
• Volume for each legato sample (I don't see any reason this would be removed but just in case. Maybe it's possible to include this in a "legato adjustment" ui?)

Most of the time you will want the crossfades to be the same length on both sides, ie. sourceSustain > legato will have an xfade of ~180ms, and this value will be adjusted for both. This is what I eventually settled on for Poeesia. However, sometimes having the source sustain play a little longer can help preserve ambience, eg. sourceSustain.Release = 220ms, legato.Attack = 180ms.

There might also be situations where you want release samples to have yet another setting, so targetSustain.release would change depending on whether you're playing to a new legato sample, or to a release sample.

Whether it's worth exposing all the parameters separately depends on how annoying it is for you to implement.

sourceSustain > legato, crossfadeTime
sourceSustain > legato, crossfadeShape
legato > targetSustain, crossfadeTime
legato > targetSustain, crossfadeShape

is probably sufficient, and in fact probably the nicest dev experience.

Legato length

What I loved about building Poeesia in HISE was I wasn't forced to choose a "best fit" setting for the legato sample length like you're forced to in Kontakt.

I just got the length of the legato sample from the samplemap, and delayed playback of the target sustain by that amount. I was then free to set unique start end end points (and volume) for every legato sample in the library. This is the main reason the instrument sounds better than it ever could in Kontakt.

Sustain offsest

Legato sounds much more natural if you jump very far into the target sustain. I know this is tricky in HISE with the maximum offset being relatively small, I forget the name of the remedy you and David came up with.

Phase locking sustains

I do not pitch/phase lock my samples because it's never really been an option, and is only really beneficial on solo instruments with multiple dynamics. As usual, doing it in Kontakt is a pain, though I know some devs have done it.

However, I assume using loris to flatten the pitch, and re-applying the extracted pitch curve from one dynamic layer to all dynamic layers using the sample editor envelope in HISE, would be an extremely pleasant workflow.

Even on raw, non-phase-locked samples, aligning the phase of sustains to legatos is very noticeable, just like it is for loops.

Polylegato

You probably know this already, but just to recap. Two different conditions should trigger a legato sample:

• key released, and another one pressed soon after (jump legato)
• key pressed, and another one released soon after (overlap legato)

Either one of these on their own is also technically polylegato, but it is much less playable for the enduser, to the point where I wouldn't include it in a library.

The only parameter that needs to be exposed is the "legato window", which also represents the added latency. I experimented with different times for jump/overlap legato, but settled on adjusting them together and exposing that setting in the UI of the instrument.

[aaronventure]

@Christoph-Hart Phase-locking will only work on very, very dry mono samples.

Wet samples are inherently impossible to phase-lock in a crossfade because the phase differences are a result of the pitch fluctuations in the sustained note.

Some instruments don't have this behavior, but the ones where you wanna have legato samples do. Another factor is the noise in the recording, either produced directly by the instrument, or noise from the mic and pre-amp, but the last two you can easily reduce by simply opening your wallet. The noise will affect the detection of the pitch fluctuations if your goal is to iron it out using brute-force.

So when you put an instrument into a hall and record them, then try to phase-lock layers or even legato samples, there are a few wrecking balls coming at you full-speed. Already the first one is unavoidable, though:

• The pitch fluctuations (and noise) reverberate in the room, with their reverberation being recorded in the sustain of the sample. You can't phase match that, because at that point it's no longer a single voice with a single pitch; you have the original sound coming out of the instrument, and the reflected instances of that same sound with a slightly different pitch. If you move closer during the recording process to try and reduce the audible reverberation, you still have early reflections which are way more audible than the tail.

But even if you somehow survived the first wrecking ball (you won't), there's another:

• Only close mics are mono, but not always; orchestral sections are often close-miked LCR (strings and woodwinds) and some engineers even prefer two close mics on horns and two on trumpets/trombones. Pianos are always stereo-miked, and harps and mallets are as well more often than not. Even if you manage to eliminate the room issue, all you need for your now seemingly sure phase-locking idea to fail is for the instrument to move, even just a little bit. You're coming back from a break and will now record the legato samples... The player accidentally moves the chair as they settle into it, and they lean a little bit the other way. The reflections all shift, slightly but they do, and the direct sound and the reflections now all have different timings on each of the mics in the main configuration. You can try phase-locking individual mics in the configuration, the sense of direction that they provide to an instrument is precisely because of how the reflections interact with the spaced microphones.

That means your phase-locked legato will require bone-dry mono samples (or near-bone-dry), otherwise it's useless. This is the flaw with conventionally sampled instruments that every composer working with libraries like that accepts and understands (except me, which is why I made Infinite Brass and why I continue working on the concept). The main problem here is then putting these instruments into a convincing space. There are already some solutions on the market that claim to offer it, but all require extensive tweaking. So this may affect how many devs go for this approach at this time.

The legato groups automatically handling all the logic is a great idea and will surely attract a lot of people. I think one of the best advances in conventional sampled legato was made by Cinematic Studio Series where they recorded the slowest legato (gliss/port. is separate) and then use timestretching to shrink it down for higher velocities with some smart editing.

Therefore, one parameter you should definitely consider adding is the stretch %, and have the option to allow the legato samples to be timestretched.

[d.healey]

@aaronventure said in Phaselocking 2025 and other sampler related discussions:

Phase-locking will only work on very, very dry mono samples.

It works with stereo samples too, you just treat them like two mono samples - the same as with multi-mics.

@aaronventure said in Phaselocking 2025 and other sampler related discussions:

Wet samples are inherently impossible to phase-lock in a crossfade because the phase differences

I just wanted to test this. Here is a trumpet C3 crossfade between 3 dynamics, original and phase-aligned. These are hall samples. They both sound pretty good actually thanks to the smoothing of the CC modulator. I think the crossfade sounds a little better on the aligned one but that's subjective. The attack of the aligned one is a little chorusy.

Original:

Aligned:

@Christoph-Hart RR groups still have issues in the develop branch - when you import samples mapped to RR groups the little popup about creating groups (old vs new) has the wrong number and trying to adjust the RR group of a sample will move it to group 0 (which doesn't exist).

[aaronventure]

@d-healey said in Phaselocking 2025 and other sampler related discussions:

I just wanted to test this. Here is a trumpet C3 crossfade between 3 dynamics, original and phase-aligned. These are hall samples. They both sound pretty good actually thanks to the smoothing of the CC modulator. I think the crossfade sounds a little better on the aligned one but that's subjective. The attack of the aligned one is a little chorusy.

Both of these sound good. But that crossfade is very slow. How does it sound if you stay at the crossfade instead of driving the controller past it to fully fade in the other layer?

What about when you do a very fast crossfade? Because that's what'll happen when crossfading to a legato sample, and the room sound will be where it's the most obvious, and you can't remove it.

[d.healey]

@aaronventure said in Phaselocking 2025 and other sampler related discussions:

What about when you do a very fast crossfade?

original-fast.wav
aligned-fast.wav

[Simon]

@d-healey Audio embeds already work!

[d.healey]

@Simon Perfect, thanks, I didn't realise!

[clevername27]

@Christoph-Hart Are you sure you're solving the right problem?

[Simon]

@aaronventure Aligning legato samples to the sustains as Christoph is implementing certainly doesn't require bone dry nor pitch-locked samples. It's just like editing a loop, the crossfade happens over a few hundred ms, so the pitch variations don't have time to throw it more than a few degrees out of phase. Even on an ensemble the difference is subtle, but definitely audible.

As for pitch-squashing / phase-locking of different dynamic levels, you probably have the most experience with that of anyone here, and I agree with your analysis that it's practically useless on anything other than bone dry samples.

@clevername27 Depends what you mean by the "right" problem! Aligning legatos to sustains is a problem that every Kontakt developer who has ever done legato has encountered, and has no way of solving. Without switching to HISE, of course ;)

[Christoph Hart]

nice nice nice, good discussion guys.

This alignment will be most noticeable on solo instruments that are already closer to sine waves than to the chaos of an ensemble

Of course. In the end you probably enable this with solo instruments that are reasonable dry while deactivating it with big ensemble patches, but for the "sinesque" stuff it makes a drastic difference (hence my inquiry about phase-locking as this also is most useful for those kind of sounds.

Attack and release time, and curve,

Currently it's a single time for both the fade into and out of the legato sample, but I can easily make this two separate values. But do you really need to adjust the curve like in a real world scenario? I'm using the same internal technique as the usual Synth.addVolumeFade(), so in that case I would have to define a per-sampler curve for all volume fades.

There might also be situations where you want release samples to have yet another setting, so targetSustain.release would change depending on whether you're playing to a new legato sample, or to a release sample.

This is a subject for another discussion - the release transition will be handled by the specific release trigger layer that will come with its own set of properties (but these are much more clearer to me).

I just got the length of the legato sample from the samplemap, and delayed playback of the target sustain by that amount.

Yes that's precisely how it works at the moment and is hardcoded into the engine. It will not account for realtime-pitch modulation through modulators, but the sustain sample will be automatically delayed by the exact sample length of the legato sample.

Legato sounds much more natural if you jump very far into the target sustain. I know this is tricky in HISE with the maximum offset being relatively small, I forget the name of the remedy you and David came up with.

Yes that makes sense and the idea is an additional API method that can add an additional start offset to the sustain sample to make this work. The max offset is UINT16_MAX which would be 65536, if that isn't enough you can add a multiplier somehow, but I also forgot how to do that. David?

I think one of the best advances in conventional sampled legato was made by Cinematic Studio Series where they recorded the slowest legato (gliss/port. is separate) and then use timestretching to shrink it down for higher velocities with some smart editing.

Yeah, then we are back in CPU spike-land as my masterly deceitful trick of delaying the sample playback a few milliseconds until the timestretchers are initialised messes up the timing between legato & sustain samples.

I would assume that this process can be done offline and render a few versions to be layered across the velocity axis - I don't think that there's a need for more than eg. 5-6 prerendered legato speeds.

[d.healey]

@Christoph-Hart said in Phaselocking 2025 and other sampler related discussions:

but I also forgot how to do that. David?

Add start offset multiplier by davidhealey · Pull Request #680 · christophhart/HISE

As discussed - https://forum.hise.audio/topic/11921/how-to-get-around-max-start-offset-is-65536

GitHub (github.com)

[aaronventure]

@Christoph-Hart said in Phaselocking 2025 and other sampler related discussions:

I would assume that this process can be done offline and render a few versions to be layered across the velocity axis - I don't think that there's a need for more than eg. 5-6 prerendered legato speeds.

Is this something HISE could do? Could you make it less intensive on disk space?

Can you somehow precalculate 126 steps of timestretch without it being equal to rendering it out that many times?

If the answer is still rendering it to audio, this is one of those times having OPUS support would be very beneficial.

[Simon]

@Christoph-Hart Single time for fade into and fade out of legato sample is fine. Curve has been very important to adjust in the past, but I realize that's always been in the context of typical unaligned legato, where it might play in or out of phase.

So, going on the assumption that the phase-align-o-matic does its job, the crossfade should be linear as it is now, and probably doesn't need to be adjustable.