Simple ML neural network

Christoph Hart

@Christoph-Hart Alright, the first experimental integration is pushed. You can now load neural networks using RTNeural and either inference it using the scripting API or run it as realtime effect using the math.neural node. I've created an example project with some hello world stuff and a roundtrip tutorial for getting started with TorchStudio:

https://github.com/christophhart/hise_tutorial/tree/master/NeuralNetworkExample

This is far from being production ready but it should be good enough for playing around and let me know what features might be interesting to add.

Dan Korneff

@Christoph-Hart oh shit! There goes my weekend plans. R.I.P my marriage

Matt_SF

@Christoph-Hart Awesome. My knowledge of Neural Networks is close to 0 but this is a good opportunity to learn something new. Thank you genius!

iamlamprey

checking this out now, very exciting stuff. I also appreciate you skipping over MNIST in the roundtrip example

orange

@Christoph-Hart Thanks for this

hisefilo

https://github.com/christophhart/HISE/commit/a295c6c31d7a44e5323cbfde67395131b223a2b4

:beaming_face_with_smiling_eyes:

iamlamprey

@Christoph-Hart said in Simple ML neural network:

A very powerful boost for this functionality could be the Loris library, I can imagine that having an array of highly precise time-varying gain values associated to the harmonic index is a much better input data than these few pixels from a spectrogram

Have you had a chance to play around with this particular use-case? Having the neural node handle the greyishbox modelling sounds a lot more streamlined than fine tuning an additive synth

Christoph Hart

@iamlamprey nope, I‘ve suspended my journey into ML until I have a real use case for it :)

If anyone is using this stuff I‘m happy to implement new features or fix issues, but now that the „hello world“ is implemented I expect it to grow with actual projects and their requirement.

iamlamprey

@Christoph-Hart yeah fair enough, i'll keep noodling on this additive synth but i'll definitely be trying the neural & loris pairing at some point

iamlamprey

@Christoph-Hart do you happen to have the training/dataloader for the sine model handy? the repo only has tanh

i'm currently migrating the example over to pure torch because torchstudio keeps uninstalling my local packages and in general is kinda gross

iamlamprey

@d-healey is there a Rhapsody update coming soon? I just tried loading the example as a rhapsody library and (duh) it broke everything, i assume the next version of Rhapsody will be built with the latest version of HISE?

d.healey

@iamlamprey said in Simple ML neural network:

is there a Rhapsody update coming soon?

Probably next month.

@iamlamprey said in Simple ML neural network:

i assume the next version of Rhapsody will be built with the latest version of HISE?

Yup

iamlamprey

@d-healey i am excited to ignite everyone's CPUs

d.healey

@iamlamprey I thought the ML stuff is only in scriptnode?

iamlamprey

@d-healey uncompiled networks work in Rhapsody, the snex and expr stuff doesn't though

d.healey

@iamlamprey Aha ok that's good to know

iamlamprey

This post is deleted!

iamlamprey

This post is deleted!

Lindon

@Christoph-Hart said in Simple ML neural network:

@iamlamprey nope, I‘ve suspended my journey into ML until I have a real use case for it :)

If anyone is using this stuff I‘m happy to implement new features or fix issues, but now that the „hello world“ is implemented I expect it to grow with actual projects and their requirement.

Well speak of the devil and he will appear.... I now have a customer who would like to make Amp sims using Neural Net ML....

So I've spent some time with the research papers, the youTube Videos and the blogs. It would seem I might need a statetful LSTM model - but I could easily be wrong.

My understanding of how I(we) might approach this is very very poor, but I think there any number of ways to get to the json code that would be needed to "make it work", but wow do I have a bunch of questions.... here's some of them:

is this even possible in HISE? (I mean the playback not the modelling or learning)
Can I "just" use (say) this stuff : https://www.youtube.com/watch?v=xkrqF0D8pfQn and take the .json output and load it into RTNeural and expect it to "sort of" work?
If not what's the best way to get from a bunch of guitar recordings to a set of .json files that I can then load into math.neural?
I assume all these Learned models are in fact static snapshots of dynamic processes, so every time the user changes the distortion control on the plugin I will need to load another model - if so how practical is that?

iamlamprey

@Lindon said in Simple ML neural network:

So I've spent some time with the research papers, the youTube Videos and the blogs. It would seem I might need a statetful LSTM model - but I could easily be wrong.

Yeh any recurrent network is fine for a guitar amp, Jatin prefers GRU's but all of the guitarML stuff is LSTM if i remember correctly, i think GRUs are a bit easier on the CPU

My understanding of how I(we) might approach this is very very poor, but I think there any number of ways to get to the json code that would be needed to "make it work", but wow do I have a bunch of questions.... here's some of them:

is this even possible in HISE? (I mean the playback not the modelling or learning)

if the entire RTNeural framework is implemented, recurrent models should be supported

Can I "just" use (say) this stuff : https://www.youtube.com/watch?v=xkrqF0D8pfQn and take the .json output and load it into RTNeural and expect it to "sort of" work?

if he's just using regular old pytorch and building simple models, you should be able to load the state_dict and run the export script, then bring the JSON into HISE

If not what's the best way to get from a bunch of guitar recordings to a set of .json files that I can then load into math.neural?

this is where training a model comes in, you'd need to learn some basic Python and familiarize yourself with packages like Torch or Tensorflow, as well as some audio-handling ones like Librosa

the short version is:

load and preprocess/augment the audio data
convert it into a dataset that can be read by a model
create the model & train it on the data
export the model's state_dict using the RTNeural export script
load it into a Math.Neural node

I assume all these Learned models are in fact static snapshots of dynamic processes, so every time the user changes the distortion control on the plugin I will need to load another model - if so how practical is that?

recurrent models aren't really "static", they can only estimate a single function at a given time, but that function can change depending on the "state" of the network, it basically has a short memory (literally in the name of LSTMs) so the function it's estimating is dependant on that memory

that being said, for a guitar amp the typical process is to train several models on various settings of the amp, then crossfade between them

Simple ML neural network

32

1.9k

10.6k

92.8k