Simple ML neural network

[Christoph Hart]

@hisefilo Just FYI, I've started working on adding a neural network API to HISE. It will take a while so don't expect immediate results, but my plans are the following:

1. add RTNeural as a "network player" that can run trained models. This will be available in the compiled plugin as well as in HISE. The use cases for this will be TBD (and I'm open to suggestions for possible applications), but I can imagine there will be both a scripting API for processing input data as well as a scriptnode node that will run the network on either the audio signal or cable level (a bit like cable_expr and core.expr). The advantage of this library is that it's fairly lightweight plus it has an emphasis on realtime performance (although there will be use cases like preset creation, etc).
2. Add a binding to one of the big ML libraries (the current favorites are either PyTorch or mlpack) to HISE in order to create and train neural networks. I know that the "industry standard" answer to this task is "use Python", however the integration of these libraries into HISE will yield a few advantages, plus I need to learn the API anyways so this will be a drive-by effect of the integration.

With the availability of training models within HISE we'll get these advantages over having to resort to Python:

• we need to convert the model data to be loaded into RTNeural anyways
• we can create training data from within HISE using a script that feeds into the training process
• no need to learn Python and it's weird whitespace syntax lol. If the tools are available without having to setup the entire Python ML toolchain, it might be used for mor simple tasks too.

I think in order to make this as non-bullshitty as possible (I have no interest in saying "HISE can now do AI too!"), we should talk about which use cases for neural networks occur in the development of HISE projects and then talk about the requirements.

From the top of my head there are a few applications:

• sound classification. Drop in a sample, the network categorizes it in whatever you want and processes this information => train a network with samples, then run it with user supplied samples to get the information out of it.
• preset creation. Drop in a sample, the network will analyse it and create preset data (a bit like this homie is doing it. => train a network with spectrograms of randomly created presets, then run the inverted process when the user drops a sample. A very powerful boost for this functionality could be the Loris library, I can imagine that having an array of highly precise time-varying gain values associated to the harmonic index is a much better input data than these few pixels from a spectrogram
• use a neural network to perform audio calculations (from amp sim and other stuff to changing the instrument like the ddsp plugins do). I'm not very picky about distortion and amp simulation (a simple tanh does the job for me lol), but apparently that's one of the prime applications of RTNeural.

The scope of the planned neural network support is currently limited to anything that boils down to "float numbers in, float numbers out" and I'm not deep enough in the ML rabbithole yet to decide how big of a step it would be to add language support (so you can finally make SynthGPT with HISE), plus my current intuition is that there's a rather limited and hype-focused use case for NLP in regards to audio plugins.

[Dan Korneff]

@Christoph-Hart This is pretty awesome news. I agree that RTNeural looks like a good choice to add as a player. I wouldn't be completely bummed if we had to train models outside of HISE, but the integration of PyTorch would be much welcomed to streamline the workflow.
I would most likely be training models of basic things (like guitar amps) but also component level models.

[A Former User]

i'll just leave this one here, the most impressive use-case i've seen so far

[A Former User]

@Christoph-Hart said in Simple ML neural network:

no need to learn Python and it's weird whitespace syntax lol

i like Python

[hisefilo]

@Christoph-Hart WOW!!! Nice surprise,
I can think of ML HISE playing the same role it did for non JUCE C++ experts in dsp developing arena. DDSP or RAVE also will be nice to have onboard.
We all are waiting for your news!!

[Christoph Hart]

Alright maybe I‘ll hold off with the HISE network training part and dust off my Python skillset, the pipeline is just to advanced to not use it for model building.

DDSP and stuff is nice but I think I need to add „conventional“ neural networks first, then we can expand on that (basically fast forwarding the last 30 years of development in this area lol).

[A Former User]

@Christoph-Hart Jatin (the guy who made RTNeural) is super friendly and knowledgeable, i'm sure he'd be happy to answer any questions you have about recurrent models and such if you haven't touched base with him already

i don't think training inside HISE would be particularly useful since it would be limited to the CPU anyway right?

[Christoph Hart]

@iamlamprey No libtorch has GPU support. But I just discovered TorchStudio, that's precisely the kind of GUI wrapper I needed to avoid the frustrating Python first steps stage...

[Christoph Hart]

@Christoph-Hart Alright, the first experimental integration is pushed. You can now load neural networks using RTNeural and either inference it using the scripting API or run it as realtime effect using the math.neural node. I've created an example project with some hello world stuff and a roundtrip tutorial for getting started with TorchStudio:

hise_tutorial/NeuralNetworkExample at master · christophhart/hise_tutorial

The Tutorial project for HISE. Contribute to christophhart/hise_tutorial development by creating an account on GitHub.

GitHub (github.com)

This is far from being production ready but it should be good enough for playing around and let me know what features might be interesting to add.

[Dan Korneff]

@Christoph-Hart oh shit! There goes my weekend plans. R.I.P my marriage

[Matt_SF]

@Christoph-Hart Awesome. My knowledge of Neural Networks is close to 0 but this is a good opportunity to learn something new. Thank you genius!

[A Former User]

checking this out now, very exciting stuff. I also appreciate you skipping over MNIST in the roundtrip example

[orange]

@Christoph-Hart Thanks for this

[hisefilo]

- added RTNeural & neural network API · christophhart/HISE@a295c6c

The open source framework for sample based instruments - - added RTNeural & neural network API · christophhart/HISE@a295c6c

GitHub (github.com)

:beaming_face_with_smiling_eyes:

[A Former User]

@Christoph-Hart said in Simple ML neural network:

A very powerful boost for this functionality could be the Loris library, I can imagine that having an array of highly precise time-varying gain values associated to the harmonic index is a much better input data than these few pixels from a spectrogram

Have you had a chance to play around with this particular use-case? Having the neural node handle the greyishbox modelling sounds a lot more streamlined than fine tuning an additive synth

[Christoph Hart]

@iamlamprey nope, I‘ve suspended my journey into ML until I have a real use case for it :)

If anyone is using this stuff I‘m happy to implement new features or fix issues, but now that the „hello world“ is implemented I expect it to grow with actual projects and their requirement.

[A Former User]

@Christoph-Hart yeah fair enough, i'll keep noodling on this additive synth but i'll definitely be trying the neural & loris pairing at some point

[A Former User]

@Christoph-Hart do you happen to have the training/dataloader for the sine model handy? the repo only has tanh

i'm currently migrating the example over to pure torch because torchstudio keeps uninstalling my local packages and in general is kinda gross

[A Former User]

@d-healey is there a Rhapsody update coming soon? I just tried loading the example as a rhapsody library and (duh) it broke everything, i assume the next version of Rhapsody will be built with the latest version of HISE?

[d.healey]

@iamlamprey said in Simple ML neural network:

is there a Rhapsody update coming soon?

Probably next month.

@iamlamprey said in Simple ML neural network:

i assume the next version of Rhapsody will be built with the latest version of HISE?

Yup