Spatial Audio support?
-
Maybe this is already available and I'm not aware of it, but thought I'd ask anyway.... Is there support for spatial audio in HISE or any plans to include it in the future? Also what about support for OpenGL like they have in JUCE or ability to import 3D models and materials to then animate them as part of a UI? I'd love to create an interactive animated 3D UI for an instrument. I'm coming from a background in Unity development so I'm just thinking towards the future of spatial computing and working in 3D. There is already a spatial VST on the market now that I was reading about in magazine. I'd love to be able to design sample presets and synth presets with a graphical UI that makes it easy to position the sound within a 3D space. I think Apple's spatial audio is based of of Dolby Atmos, but I could be wrong.
-
@ElectroBlue for the UI I would recommend using a webview - OpenGL is deprecated and the OpenGL support in HISE is limited to writing fragment shaders without texture support, but with the webview you can go completely crazy with WebGL knowing that the frameworks you’ll use will adapt to any new API that Apple will throw at us.
You might even export your Unity project as web app and run it in HISE (I haven‘t digged into Unity yet, but I managed to load the hello world project of a small web-only game engine called Canvas playground or so). Finding a way to communicate over a JS layer with your HISE instrument might be annoying to setup first but doable.
As for spatial audio, if you happen to find a C++ library that does this well you can wrap it into a node and use it in scriptnode.
-
@Christoph-Hart said in Spatial Audio support?:
You might even export your Unity project as web app and run it in HISE
That is an interesting option! Do you mean exporting my Unity project as WebGL and then loading that into HISE in a web view?
-
Here is the link to the spatial audio virtual synth I mentioned. link text
-
@ElectroBlue yup. You might have to play around with some configs and check how to assign javascript callbacks to unity events but it should be doable.
-
I'm just throwing this out there, but I also had the idea of using particle effects as a UI element that would get triggered when you play notes on your midi keyboard. You could assign different particle effects for different keyboard zones. This also could be achieved with Unity WebGL if I could find a way to communicate between Unity WebGL and HISE. That part is kind of beyond my skillset but if there was a way to get the two to communicate it would open up a world of possibilities and not just for UI stuff because Unity has it's own sound engine as well and supports spatial audio.
Like if there was a way to route HISE audio into Unity's WebGL output? I have researched all sorts of options for using JUCE with Unity since JUCE supports creating a Unity Native Plugin, but that is really more geared for using your plugin as an insert into a mixer channel inside Unity and not so much for displaying the GUI inside Unity and on top of that I'd have to learn C++ to use JUCE so that option is not really an option for me.
And on the strictly Unity side of things it is possible to create virtual instruments that work with MIDI. Here is a great example of that: Unity Midi Virtual Instrument
But Unity's built-in audio engine features are pale in comparison to what HISE is capable of. Imagine if there was a way to combine the two! -
@Christoph-Hart said in Spatial Audio support?:
As for spatial audio, if you happen to find a C++ library that does this well you can wrap it into a node and use it in scriptnode.
Reply Quote 0
Is this difficult to do? If I find a C++ library for spatial audio or some other cool thing? Would I need to be a seasoned C++ programmer to do that? Or is it just a matter of putting that library in the correct folder inside HISE?
-
@ElectroBlue said in Spatial Audio support?:
it would open up a world of possibilities and not just for UI stuff because Unity has it's own sound engine as well and supports spatial audio.
nope the webview is for UI stuff only - once you close the plugin window your entire state is gone, so you need to keep the sound and data models in HISE.
-
@ElectroBlue said in Spatial Audio support?:
Would I need to be a seasoned C++ programmer to do that? Or is it just a matter of putting that library in the correct folder inside HISE?
Seasoned, no, but you need some C++ skills to include the library and write the wrapper glue code that loads it as third party node.
