Just dreaming. Any plans for adding Tensorflow lite?

[hisefilo]

I was just playing around with tflite on JUCE and was wondering if there’s any plans to add it. I want to load .tflite models. Maybe it’s an absurd question due my almost zero background in ML

[aaronventure]

@hisefilo said in Just dreaming. Any plans for adding Tensorflow lite?:

.tflite

to do what?

Rule of thumb is that for realtime, you either try to get Chris to implement it or try to do it yourself as a custom c++ node.

For non-realtime, you can do it right away with BackgroundTask as long as you can ship everything along with your plugin. Or download it on the spot first, but I guess that's a bit less reliable.

[hisefilo]

@aaronventure I want to implement any text-to-audio model. Non real time will work.
Will investigate BackgroundTask. thanks for the tip!

[Christoph Hart]

@hisefilo check out ONNX for this, I‘ve been using it to run non realtime models in HISE for stuff like spectrogram analysis etc.

[d.healey]

Can't you already load tensor flow with the neural node? https://docs.hise.dev/scripting/scripting-api/neuralnetwork/index.html#loadtensorflowmodel

[orange]

@Christoph-Hart Is the neural node now running CPU-effectively?

[Christoph Hart]

@d-healey yeah but it has a very limited set of layers so you can‘t expect it to load any model. The ONNX runtime in HISE is basically feature complete and should be able to run any model that you can export into this format.

[hisefilo]

@Christoph-Hart Wow, you were keeping that under wraps! I thought ONNX wasn’t working yet.

Any simple way to load it? NeuralNetwork.loadONNXModel or something like that?

Why am I so lazy?
https://docs.hise.dev/scripting/scripting-api/neuralnetwork/index.html#loadonnxmodel

[tomekslesicki]

@Christoph-Hart is ONNX more CPU-friendly than the older tensor flow method?

[Christoph Hart]

@tomekslesicki what's the old tensor-flow method?

There are two neural engines available in HISE:

1. RTNeural which is focused on realtime processing of audio data. It comes with a limited set of layer types predominantly used by neural networks that process realtime audio (so eg. there's no use of adding a transformer or LLM type neural network support as this would not run in realtime. You can load neural networks into this engine using two formats (PyTorch & TensorFlow) with the respective API methods.
2. ONNX runtime is a more general purpose network inference engine that can be used to run almost any kind of network as the framework is basically feature complete. Of course the integration into HISE is very spotty (I just used it for spectrogram analysis so far), plus you need to build a separate DLL and ship it with your project because the framework is very heavyweight so I wouldn't want to include it in the default compilation process.

[tomekslesicki]

@Christoph-Hart thanks, that's great info!

[hisefilo]

@Christoph-Hart said in Just dreaming. Any plans for adding Tensorflow lite?:

I just used it for spectrogram analysis so far

Do you have a snippet or tip??
I managed to load a simple network that infers the double of a number LOL. But cannot connect it.

const var onnxRoot = FileSystem.getFolder(FileSystem.UserPresets).getParentDirectory().getChildFile("Scripts/Models");

const var on = Engine.createNeuralNetwork("onnxNetwork");
const var onnxModel = onnxRoot.getChildFile("double_model_v2.onnx").loadAsBase64String();


const var out = on.loadOnnxModel(onnxModel, [4.0]);

Console.print(out);

[Christoph Hart]

@hisefilo it's currently only used by the processFFTSpectrum() method which directly grabs the spectrum from the FFT object. The workflow for this is:

1. Create the FFT object, set the spectrogram properties suitable to your task
2. Run the FFT on your training data, then dump the spectrums as images
3. Train your model with these spectrograms (I've been using TorchStudio for that).
4. Export the model as ONNX.
5. Load it back into HISE
6. Use the FFT with the same properties to create spectrograms of the user input and feed that into the processFFTSpectrum() method.
7. It outputs an array of float numbers that you can use for classification or something else.

This is a very narrow use case but I've been using that to train on samples to detect the release trigger point and some basic classification of drum samples with moderate success.

[hisefilo]

@Christoph-Hart Well, I was aiming to implement something simpler than that to begin with. I’ll go for this later, I guess. Thanks, mate, for the tutorial!

[Christoph Hart]

@hisefilo I mean you're kind of limited to this exact pipeline at the moment so if you want to play around with some hello world type stuff I would recommend a simple classification system that detects different waveforms (like static sine / saw / noise) spectrograms.

Once you get that going, we can think about other use cases and how to expand the integration towards other input / output scenarios.

[hisefilo]

@Christoph-Hart meaning NeuralNetwork.process wont work yet, with a simple 1in 1out float32 network?
Just NeuralNetwork.processFFTSpectrum is working?

[HISEnberg]

@hisefilo Just wanted to chime in about this work here:

Building AI Enhanced Audio Plugins by Matthew John Yee-King

GitHub - yeeking/ai-enhanced-audio-book: A repository of AI-enhanced audio plugins written using C++, JUCE, libtorch, RTNeural and other libraries. Supports my book

A repository of AI-enhanced audio plugins written using C++, JUCE, libtorch, RTNeural and other libraries. Supports my book - yeeking/ai-enhanced-audio-book

GitHub (github.com)

I am just finishing the work and its a great starter for training in Torch then building in the JUCE framework. I haven't tried HISE integration yet (I didn't know the API was updated either). I am happy to share the pdf if you are interested!