Spectral Morphing in HISE

Casmat

@d-healey with access to the FFT buckets for processing, couldn't one accomplish spectral morphing?

d.healey

@Casmat No idea, but if you figure it out let me know! Might also be possible using Loris: http://www.cerlsoundgroup.org/Kelly/soundmorphing.html

Casmat

@Christoph-Hart?

Casmat

bump!!!!

clevername27

@d-healey @Casmat Yes. Crossfade/interpolate between them. For a less manual approach, see Loris.

Christoph Hart

I've found that HISE supports spectral morphing

Where did you get that information? There are a few tools in HISE that are somewhat related to the subject (Loris, some FFT tools), but there's definitely not a readymade spectral morphing algorithm in HISE.

with access to the FFT buckets for processing, couldn't one accomplish spectral morphing?

What's "access to the FFT buckets"? Also an FFT won't get you all the way to spectral morphing it's like saying "Now that we've invented electricity, where are the self-driving cars"?

Casmat

@Christoph-Hart ahh I see, I guess I misread something!

DanH

Digging this up in case any advancements have been made in the spectral morphing area?!!

Could Max MSP do it?

HISEnberg

@DanH I don't think you will make it far in Max MSP if your intention is to bring it into HISE. RNBO's fft performs poorly, and Gen~ doesn't really have a viable fft option. Max itself has some good options for spectral processing but nothing exportable to C++

mmprod

@HISEnberg what would you use for fft processing if RNBO is not a viable option? and I’ve also heard that Faust is not optimal for this either

HISEnberg

@mmprod It's on my bucket list to do a deep dive into the Loris library, which I think has the solution. I have only scratched the surface here, but it is seriously impressive what it can do. My understanding with Loris is that it can't be used for live processing (though I could very well be mistaken on this).

RNBO, unless they have updated their FFT objects, would be my go to but from experience I know it is not useable. Otherwise the task probably involves some external C++ which is not a simple feat. I wonder if @griffinboy , the GOAT for external C++ implementation has looked into it?

griffinboy

@HISEnberg

spectral? Haha it's on my list but I've never done it before. I'm very interested in linking up neural networks to spectral synthesis in the future but I've not got an external node for that yet : )

Most of my finished nodes are for analog modelling, filters, compressors, etc.

clevername27

@Christoph-Hart said in Spectral Morphing in HISE:

"Now that we've invented electricity, where are the self-driving cars"?

Are you Elon Musk?

aaronventure

@clevername27 It's funny how even on this forum I cannot get away from hearing about this fucker.

clevername27

@aaronventure Right?

"OK, I found HISE on the internet. Why isn't my plugin finished?" — Phoney Stark

hisefilo

@Christoph-Hart

Got stuck on this:

lorisManager.process(file, "morph", ??????????

I'm implementing dilate without issues

lorisManager.process(file, "dilate", dilatePoints);

Is "morph" available as "dilate" and "shiftPitch" are?

Hope you can shed some light on this topic :)

clevername27

@hisefilo @aaronventure I'm thinking you can just use the FFT object in HISE, in real-time. Anything more than crossfading between the amplitudes may be too much, though. Use straight C or C++ for any sort of real DSP.

Take DFTs of the source and destination. Measure the amplitude (i.e., magnitude) difference between each of the buckets. Crossfade between them. There are some phase issues, but given your application, I they may not matter.

~~I haven't looked at HISE's object, but assuming it just takes in audio stream, and returns buckets, we're only talking about a few lines of code in HISEscript.~~ Just ensure that you start and finish the matrices at the same time. I can go into more detail.

I have taken a look—yes, it's just a few lines a code to do what you want. Preallocate your buffers; it looks like prepare will do that. The setMagnitudeFunction call looks like it lets you define a single function to process the bins, which is helpful in your case, because ostensibly, you want to do the same thing to all of them. I'm not clear what process does. Depending on what you want to do, the windowing function doesn't matter. You may wish to play around with the weighting of the bins, as the ear responds differently to different frequencies. Again, I'm not sure what you want to achieve, perceptually, but see if you perform your tasks without worrying about phase—you need to process the bins as quickly as possible, as HISEscript is not optimal for this. Use setEnableInverseFFT to reconstruct the signal.

All that being said, I would suggest listening to examples of spectral morphing. Much of the time, it's indistinguishable from simply crossfading the summed audio signals.

Start with what you want to achieve, perceptually, and work backwards from there.

ustk

@clevername27 said in Spectral Morphing in HISE:

All that being said, I would suggest listening to examples of spectral morphing. Much of the time, it's indistinguishable from simply crossfading the summed audio signals.

That's what I was thinking, if you crossfade between to FFTs, that'll be the same as crossfading gains in time domain.

Also, let say you do not apply the same weight to different bins. Wouldn't this be equivalent to filtering both signals before applying a crossfade in the time domain?

There are some phase issues...

Yeah I've seen this here and there when starting to play with magnitude ins... But in that case, wouldn't crossfading the phase bins at the same time as magnitude ones prevent any issues?

clevername27

@ustk said in Spectral Morphing in HISE:

That's what I was thinking, if you crossfade between to FFTs, that'll be the same as crossfading gains in time domain.

While perceptually, there may be little difference, I wouldn't quite characterise it as being the same. I'm still a little unclear about "spectral morphing", as I'm not aware of it as a formal term. In as much as I can surmise your purpose, one way to think of the distinction is that cross-fading amplitudes occurs in the time domain, while cross-fading FFT bins could be characterised as occurring the frequency domain.

Also, let say you do not apply the same weight to different bins. Wouldn't this be equivalent to filtering both signals before applying a crossfade in the time domain?

I would suggest applying different weighting, as auditory processing is not strictly linear in terms of frequency response.

Yeah I've seen this here and there when starting to play with magnitude ins... But in that case, wouldn't crossfading the phase bins at the same time as magnitude ones prevent any issues?

It's a good question, but I want to avoid wading into the phase issues of DFT as they are non-trivial. I would suggest starting with a simple interpolation of magnitudes between frequency bins.

hisefilo

@clevername27 hello professor! sorry for the delay. And for my lack of preparation!
I want to achieve this kind of morph https://www.cerlsoundgroup.org/Kelly/sounds/trumpetcry.wav

I actually want to start trying to use amplitudes from one audio with freqs from the other one to see what happens.
I can see Loris already prepares freqs and amps for morphing when I do

lorisManager.createEnvelopePaths(file, "frequency", 1);

Interface: distilling 24314 Partials
Interface: sifting 24314 Partials
Interface: collating 24330 Partials
Interface: distilling 24266 Partials
Interface: sifting 24266 Partials
Interface: collating 24276 Partials
Interface: channelizing 24330 Partials
Interface: channelizing 24276 Partials
Interface: Prepare partial list for morphing

And this is how it's made on C++ but can replicate on HISE.
Going the FFT way is much harder for my level I guess.

#include "loris.h"

#include <stdio.h>
#include <stdlib.h>
#include <strings.h>

int main( void )
{
   #define BUFSZ (3*44100)
   double samples[ BUFSZ ]; /* clarinet is about 3 seconds */
   unsigned int N = 0;
   
   PartialList * clar = createPartialList();
   PartialList * flut = createPartialList();
   LinearEnvelope * reference = 0;
   LinearEnvelope * pitchenv = createLinearEnvelope();

   LinearEnvelope * morphenv = createLinearEnvelope();
   PartialList * mrph = createPartialList();   

   double flute_times[] = {0.4, 1.};
   double clar_times[] = {0.2, 1.};
   double tgt_times[] = {0.3, 1.2};
   
   
   /* import the raw clarinet partials */
   printf( "importing clarinet partials\n" );
   importSdif( "clarinet.sdif", clar );
      
   /* channelize and distill */
   printf( "distilling\n" );
   reference = createF0Estimate( clar, 350, 450, 0.01 );
   channelize( clar, reference, 1 );
   distill( clar );
   destroyLinearEnvelope( reference );
   reference = 0;
      
   /* shift pitch of clarinet partials */
   printf( "shifting pitch of clarinet partials down by 600 cents\n" );
   linearEnvelope_insertBreakpoint( pitchenv, 0, -600 );
   shiftPitch( clar, pitchenv );
   destroyLinearEnvelope( pitchenv );
   pitchenv = 0;
   
   /* import the raw flute partials */
   printf( "importing flute partials\n" );
   importSdif( "flute.sdif", flut );
   
   /* channelize and distill */
   printf( "distilling\n" );
   reference = createF0Estimate( flut, 250, 320, 0.01 );
   channelize( flut, reference, 1 );
   distill( flut );
   destroyLinearEnvelope( reference );
   reference = 0;

   /* align onsets */
   printf( "dilating sounds to align onsets\n" );
   dilate( clar, clar_times, tgt_times, 2 );
   dilate( flut, flute_times, tgt_times, 2 );
   
   /* perform morph */
   printf( "morphing clarinet with flute\n" );
   linearEnvelope_insertBreakpoint( morphenv, 0.6, 0 );
   linearEnvelope_insertBreakpoint( morphenv, 2, 1 );
   morph( clar, flut, morphenv, morphenv, morphenv, mrph );
   
   /* synthesize and export samples */
   printf( "synthesizing %lu morphed partials\n", 
           partialList_size( mrph ) );
   N = synthesize( mrph, samples, BUFSZ, 44100 );
   exportAiff( "morph.aiff", samples, N, 44100, 16 );
   
   /* cleanup */
   destroyPartialList( mrph );
   destroyPartialList( clar );
   destroyPartialList( flut );
   destroyLinearEnvelope( morphenv );
   
   printf( "Done, bye.\n\n" );
   return 0;
}

Spectral Morphing in HISE

24

1.6k

11.1k

96.9k