The need for Spectral Processing and AI in HISE

[resonant]

I think this has been asked a few times before, but is there any progress on Spectral Processing?

Scriptnode really needs this. Many plugins currently on the market can perform Eq, Compressor and other operations at a much more advanced level using Spectral processing.

In addition to spectral processing, AI is also absolutely necessary.

It is absolutely necessary for Hise to add these systems in the near future in order to avoid remaining old technology and to be integrated with current technology.

[d.healey]

@resonant said in The need for Spectral Processing and AI in HISE:

AI is also absolutely necessary.

What do you mean by AI?

[Fortune]

@d-healey said in The need for Spectral Processing and AI in HISE:

@resonant said in The need for Spectral Processing and AI in HISE:

AI is also absolutely necessary.

What do you mean by AI?

I think AI is the Artificial Intelligence.

Yes nowadays we hear Artificial Intelligence and Spectral Processing Plugins more and more everyday. And Hise should integrate these tech asap.

And +1000 for this @Christoph-Hart

[Casmat]

@Fortune @resonant The thing is, for most people, the algorithm/data sets would need to be tailored since each plugin may have a different use for AI. It's just very broad unless there's a framework that could do something general (pytorch/tensorflow?), but that would take quite some effort from my understanding..

[Fortune]

@Casmat At least with machine learning integration, developers might have something to customize to their domain. Or as you said, a general approach.

Many things have been added to Hise lately. Unlike AI, Spectral Processing can at least be added, which can be integrated more easily.

[aaronventure]

Machine learning is mostly done in python. I'm not sure how well that would work in audio for real-time tasks.

You can translate it to C++, but then you can already do that without HISE, and simply import the code into HISE.

[Christoph Hart]

What do you want to do with AI? If you‘ve got a valid use case, then you can train your models in python and add whatever deep learning library you need yourself - this is the least of your problems.

My guess is that whenever you ask for AI to be added to HISE you have no clue how it works so you wouldn‘t be able to use whatever generic building blocks I might add to the framework anyway. At some point in the future I might add a few tools to execute neural networks but I don‘t see this a critical or mainstream feature at all.

I‘m happy to work with any developer who starts working on an AI feature and realizes that some stuff in HISE could be changed to make the process smoother, but a general request for „add AI to HISE in order to not get left behind“ is complete FOMO and rather naive.

With spectral processing it‘s similar - what do you want to do here? Make your own timestretching algorithm? Make a vocoder, simulate MP3 codec artifacts? In each of those examples the bulk of the work has to be done with raw number crunching from the FFT output so you need to do this in C++ which you can do already (and the complexity of understanding and creating these algorithms vastly exceed the complexity of writing C++) There‘s a limited amount of generic building blocks that will make things easier here (FFT functions, windowing etc) and they are already available on the C++ level.

[Dan Korneff]

@Christoph-Hart maybe a NAM player node? :beaming_face_with_smiling_eyes:
https://www.neuralampmodeler.com/the-code

[Christoph Hart]

@Dan-Korneff You should be able to use a custom C++ node with this code:

GitHub - sdatkinson/NeuralAmpModelerCore: Core DSP library for NAM plugins

Core DSP library for NAM plugins. Contribute to sdatkinson/NeuralAmpModelerCore development by creating an account on GitHub.

GitHub (github.com)

However I also stumbled across this neural network library which seems to be more versatile:

GitHub - jatinchowdhury18/RTNeural: Real-time neural network inferencing

Real-time neural network inferencing. Contribute to jatinchowdhury18/RTNeural development by creating an account on GitHub.

GitHub (github.com)

So whenever I find some idle time in my life (lol) I'll probably include the latter library to HISE.

[resonant]

@Christoph-Hart said in The need for Spectral Processing and AI in HISE:

You should be able to use a custom C++ node with this code:

It would be great if there was some examples for custom C++ node usage