@marcrex Ship ollama or a similar layer with your plugin, and a model of your choice. For this you probably want a good tool calling model, like Devstral. You will need the super light version. Unsloth has a super small quant https://huggingface.co/unsloth/Devstral-Small-2505-GGUF/blob/main/Devstral-Small-2505-UD-IQ1_S.gguf but I have no idea how well it runs. Check Gemma 1b as well.

Your system prompt needs to contain the functions that it has available for use and what each of them does.

Define these functions in HISE.

You'll run it by mounting the model, if not mounted, on plugin start, using BackgroundTask. These small models should be fast to mount anyway, so best to have it unload itself automatically after an hour of not using it.

The user can then write an instruction and the model is fed system prompt + instruction via BackgroundTask. The system prompt details what it can do, the instruction tells it what to do. You want to explicitly tell it to return a "function call" in a clearly specified format (for example doThis(arg1, arg2)) and for that to be the only return, no other text.

You should then validate the return of the BackgroundTask call, by checking that it contains a function by using string operations, and if it validates, you would call eval(return). eval() will attempt to execute the string within the code. In this case it's a function that you already defined, so it will run.

In the end, the brunt of the work is you designing the tools (functions) that the model can call.

This is a very simple example, if you want to chain tool calls, you'll need to feed the output of the function back into the model along with the previous prompt so that it can continue doing the work if needed.

If you want an example of how to use eval(), I did a cool experiment here https://forum.hise.audio/topic/9086/hise-s-best-kept-secret-dynamic-plugins

These are the limits of my current knowledge on the subject, some of these workflows may already be outdated, but hopefully I gave you a good starting point to go and explore.

Before you download ollama and start fiddling with local inference, you can try it out instantly via OpenRouter. https://openrouter.ai/mistralai/devstral-small:free

This free endpoint will of course use any data you send for training, but you're not gonna be sending any sensitive info, just your system prompt any whatever the user types in. For a simpler implementation (though obviously not as rock-solid) you can offer the user to enter their OpenRouter API and use the free endpoint that way, or provide your own via cloud functions, at which point it becomes a service. However, this way everyone can run the proper 24B model; on macBooks with 32+ GB of RAM it runs fine, but on Windows, VRAM isn't super easy to come by.

@marcrex Looks like you're just hardcoding everything currently. You need to instead use the expansion handler to give you a list of installed expansions, loop over those and grab the bits of data you need in order to populate the combo box.

@marcrex You can do this with LAF

The granulator is a sound generator, it has no effect on incoming signals. For that kind of thing, best look to FAUST.

To get the Scriptnode Granulator playing polyphonically, you have two options.

Increasing the density and reducing grain size will allow you to play chords, keep in mind though, anything after the granulator will still be monophonic.

For true polyphony, implement multiple granulators in different groups, one for each voice of polyphony. Then use a round robin script to handle the voice allocation.

@d-healey Ohh ok. If you have a youtube video tutorial about it I'll track that down then.

@dannytaurus
No problem! :-) Thanks for the clarification! :-)

@marcrex definitely do-able

Link Preview Image Elitist Loop | Modern Vocal Hook Generator | Music Software Instruments | Sound Effects | Professional Audio | Ava Music Group

Elitist Loop is a modern vocal engine that features 480+ unique vocal chops. Choose from over 250 premade loops. Change rhythm and pitch in just one click. With highly customizable features and endless combinations to choose from, ELITIST gives you vocals you won’t hear on anyone else’s track.

favicon

AVA MUSIC GROUP (avamusicgroup.com)

@marcrex This is the pitch shifter@DanH is referring to, but its delay based so the "no decent" assessment is quite accurate. You could easily put this in between the feedback routing of delay network inside Scriptnode.

I would recommend taking Lindon's recommendation though. Do some research on Routing in HISE.

@DabDab
Great! Thanks so much

@d-healey Thanks a lot! :)