ScriptNode Compilation Workflow and What to Do About It

Chazrox

@DabDab Im currently experiencing problems compiling dsp's also causing immediate crash upon clicking compile. I've resorted to only using script fx nodes as it allows me to still use certain features but im still not able to work at full capacity for some reason that I cant figure out. This thread seems like its right on point!

Chazrox

@Christoph-Hart Where do I get the latest version of HISE as you guys are updating? Sorry if im redundant, im still new to this. You're greatly appreciated!

DabDab

@Chazrox said in ScriptNode Compilation Workflow and What to Do About It:

Where do I get the latest version of HISE as you guys are updating?

GitHub - christophhart/HISE: The open source framework for sample based instruments

The open source framework for sample based instruments - christophhart/HISE

GitHub (github.com)

Chazrox

@DabDab Thank you.

Christoph Hart

Not working. Still crashing.

Oops, forgot to push a simple typo... should be fixed now.

DabDab

@Christoph-Hart Please add currentGitHash.txt and currentGit.h files.

Christoph Hart

@DabDab yup, done.

DabDab

@Christoph-Hart Yup.. Working fine now !!

@griffinboy thanks for the LPF. Can we get HPF ?

Christoph Hart

@DabDab HPF = signal - LPF?

DabDab

@Christoph-Hart Well I don't know much C++ DSP How High pass filter is structured in C++.

griffinboy

@Christoph-Hart

From my understanding, that'll kind of work but not completely.
I'll release a highpass version at somepoint

Christoph Hart

From my understanding, that'll kind of work but not completely.

It does with the only filter I've been writing myself - a one pole. Thanks for coming to my TED Talk.

DabDab

I have tried to make it a High Pass Filter from the Griffin_LadderFilter.h provided by @griffinboy
I am not a good C++ DSP programmer. Test it whether it is working or not.

/*
Copyright (c) 2024 griffinboy

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/

#pragma once
#include <JuceHeader.h>

namespace project
{
    using namespace juce;
    using namespace hise;
    using namespace scriptnode;

    template <typename T>
    class ScopedValue
    {
    public:
        explicit ScopedValue(T& val) : value(val), ref(val) {}
        ~ScopedValue() { ref = value; }
        T& get() { return value; }

    private:
        T value;
        T& ref;
        ScopedValue(const ScopedValue&) = delete;
        ScopedValue& operator=(const ScopedValue&) = delete;
    };

    template <int NV>
    struct Griffin_HighPassFilter : public data::base
    {
        SNEX_NODE(Griffin_HighPassFilter);

        struct MetadataClass
        {
            SN_NODE_ID("Griffin_HighPassFilter");
        };

        static constexpr bool isModNode() { return false; }
        static constexpr bool isPolyphonic() { return NV > 1; }
        static constexpr bool hasTail() { return true; }
        static constexpr bool isSuspendedOnSilence() { return false; }
        static constexpr int getFixChannelAmount() { return 2; }

        static constexpr int NumTables = 0;
        static constexpr int NumSliderPacks = 0;
        static constexpr int NumAudioFiles = 0;
        static constexpr int NumFilters = 0;
        static constexpr int NumDisplayBuffers = 0;

        void prepare(PrepareSpecs specs)
        {
            filtersLeft.prepare(specs);
            filtersRight.prepare(specs);

            for (auto& filter : filtersLeft)
                filter.prepare(specs.sampleRate);

            for (auto& filter : filtersRight)
                filter.prepare(specs.sampleRate);
        }

        void reset()
        {
            for (auto& filter : filtersLeft)
                filter.reset();

            for (auto& filter : filtersRight)
                filter.reset();
        }

        template <typename ProcessDataType>
        void process(ProcessDataType& data)
        {
            auto& fixData = data.template as<ProcessData<getFixChannelAmount()>>();
            auto audioBlock = fixData.toAudioBlock();

            auto* leftChannelData = audioBlock.getChannelPointer(0);
            auto* rightChannelData = audioBlock.getChannelPointer(1);
            int numSamples = (int)data.getNumSamples();

            for (auto& leftFilter : filtersLeft)
            {
                leftFilter.process(leftChannelData, numSamples);
            }

            for (auto& rightFilter : filtersRight)
            {
                rightFilter.process(rightChannelData, numSamples);
            }
        }

        class AudioEffect
        {
        public:
            AudioEffect() = default;

            void prepare(float sampleRate)
            {
                fs = sampleRate;
                reset();
            }

            void reset()
            {
                x1 = x2 = y1 = y2 = 0.0f;
            }

            void updateCoefficients(float fc, float q)
            {
                float omega = 2.0f * MathConstants<float>::pi * fc / fs;
                float alpha = std::sin(omega) / (2.0f * q);
                float cosOmega = std::cos(omega);

                float a0 = 1.0f + alpha;
                b0 = (1.0f + cosOmega) / (2.0f * a0);
                b1 = -(1.0f + cosOmega) / a0;
                b2 = (1.0f + cosOmega) / (2.0f * a0);
                a1 = (-2.0f * cosOmega) / a0;
                a2 = (1.0f - alpha) / a0;
            }

            void process(float* samples, int numSamples)
            {
                for (int i = 0; i < numSamples; ++i)
                {
                    samples[i] = processSample(samples[i]);
                }
            }

        private:
            float fs = 44100.0f;
            float b0 = 0.0f, b1 = 0.0f, b2 = 0.0f;
            float a1 = 0.0f, a2 = 0.0f;
            float x1 = 0.0f, x2 = 0.0f;
            float y1 = 0.0f, y2 = 0.0f;

            inline float processSample(float input)
            {
                float output = b0 * input + b1 * x1 + b2 * x2 - a1 * y1 - a2 * y2;
                
                x2 = x1;
                x1 = input;
                y2 = y1;
                y1 = output;
                
                return output;
            }
        };

        template <int P>
        void setParameter(double value)
        {
            if (P == 0) // Cutoff Frequency
            {
                cutoffFrequency = static_cast<float>(value);
                for (auto& filter : filtersLeft)
                    filter.updateCoefficients(cutoffFrequency, q);

                for (auto& filter : filtersRight)
                    filter.updateCoefficients(cutoffFrequency, q);
            }
            else if (P == 1) // Q Factor
            {
                q = static_cast<float>(value);
                for (auto& filter : filtersLeft)
                    filter.updateCoefficients(cutoffFrequency, q);

                for (auto& filter : filtersRight)
                    filter.updateCoefficients(cutoffFrequency, q);
            }
        }

        void createParameters(ParameterDataList& data)
        {
            {
                parameter::data p("Cutoff Frequency", { 20.0, 20000.0, 1.0 });
                registerCallback<0>(p);
                p.setDefaultValue(1000.0);
                data.add(std::move(p));
            }
            {
                parameter::data p("Q Factor", { 0.1, 10.0, 0.01 });
                registerCallback<1>(p);
                p.setDefaultValue(0.707);
                data.add(std::move(p));
            }
        }

        void setExternalData(const ExternalData& data, int index) {}
        void handleHiseEvent(HiseEvent& e) {}

        template <typename FrameDataType>
        void processFrame(FrameDataType& data) {}

    private:
        PolyData<AudioEffect, NV> filtersLeft;
        PolyData<AudioEffect, NV> filtersRight;
        float cutoffFrequency = 1000.0f;
        float q = 0.707f;
    };
}

griffinboy

@Christoph-Hart

surely there would be residue? I need to research this now haha. I've never thought to do that

aaronventure

@Christoph-Hart said in ScriptNode Compilation Workflow and What to Do About It:

@aaronventure There's already a special tool for this: Export -> Clean DSP Network files.

this doesn't seem to remove the files from ThirdParty folder (the ones generated for faust).

It also doesn't seem to clean the Binaries directory in DSPNetworks.

ustk

@Christoph-Hart A few things that I think are belonging here:

Replace Script FX modules messes up the Module tree order, and...
...the drag and drop to move the modules seems very buggy. We need to move them one by one to the bottom until we can recreate the right order. Trying to move a module between two others doesn't work.
Replace Script FX modules doesn't recreate the connections of the original ScriptFX Routing Matrix to the new HardcodedFX (this can lead to a crash when trying to remake the connections, and a real mess in the DSP anyway until you understand what just happened...)

aaronventure

@ustk said in ScriptNode Compilation Workflow and What to Do About It:

..the drag and drop to move the modules seems very buggy. We need to move them one by one to the bottom until we can recreate the right order. Trying to move a module between two others doesn't work.

this was never properly implemented in the first place. or if it was, it was broken within a few commits.

I eventually opened an issue about it.

[UX] Module Browser Drag&Drop Needs Some Love · Issue #566 · christophhart/HISE

The only way to consistently get the correct order is to move modules to be the last, and create the desired order that way. Moving them above doesn't really work. Moving processors is another story, as it's unclear whether a dragged pro...

GitHub (github.com)

aaronventure

@Christoph-Hart

Here's the big test of the new compilation workflow that involves Faust. Sorry I couldn't get to it sooner, I had to refactor a lot of the networks to not rely on nested networks, and then got caught up in other stuff.

TL:DR: it's buggy as fuck. Faust is now Schrodinger's Cat of HISE for anyone who hasn't gone through the trials or reads the forum religiously: the only thing that works is manually replacing Faust nodes with their hardcoded versions and everything else has it either producing clicks, no sound at all or being completely bypassed in the final exported plugin, all the while working in HISE and compiling successfully.

Here's a snippet. It's just a Scriptnode Synth with a Faust node that has Osc.dsp loaded, and it's just a sine oscillator.

I'm on macOS 14.5.

HiseSnippet 1257.3oc4W0sSibCE1SBF1jR61sZun8tQq5EPEaHSR3utpp.gjtQcAhHTTuC4LiCwhYrGY6.jV02s9HzGk9Fzd73LISfHZHpc2V0fDI97i827c9YNldmlxCnAWp3r3XpFgbx2VJ7oJkPhbJd9vXJxYUbmgbc+58ILNp0QHmOAeLQooRWqnCGFSTJZ.xwI+2YD3TXITxme+aOjDR39zIhPnKDLe56XQL8Dos2+6YggMIAzyYQYrt19s7E75hPw..O4wkQwD+qIWQOgXLKGF8VhpOx4qvU500uRucptq2dd6sSs8771Z2t0JWwOXq8pVc6pA8prKY6cAmVtQ.SKjczDMUAa5ghfgc5KtkaOfKXJV2PpYgGpCbxVwn58YgAiIGEPU3LTUdKU8R7wr.1X4SnrOMQg6DOxRZN4dLH48DfjSFHsjERu.2wWxh0SzXvyGgawgHXOBDaxBEqsHmeEWW.Fv0khHWSaJgEicXssKWdCW3eq+ldC39Zlf6J3mHzzS4qsdwetXgh+RQ26qpWuYpybLRQXHUNS0lzA4i43Z7AQcoxMbugDNfN1P3weZNc44iS8sO0YLTvawY5SioiV2TDFX3JyueXD.Mh1fe8CsNhnIlfxHYfcwTolYfiyQzafx.aHp.9Hp5ZsHFJDdP7CxbDACBI5oSmLEZiT.7wTwPSfhqX5gYKDeB4Xkezbr4Ehu.2lo86OaLlaFXDXp+Iv3nJyOF2nWOpudB.WB27GWzxv4+3Kjd71JqjNoIG+WLRBWDPcSDSULEU58H8S+s4seZ7b2OMsfeC7So35d8PWZ9JtlNmMi8ynS1xe.Zt92Sh+GvZyo5+WHEicXQwgzF7angPCnDL9YPGmdjAg5ToSm0crfKh6K3L+rIZmQ0R1UWQkYw9LefNPqgWSOQxK2+LZHknxjI9k6+NFmRj.OQWPt3o+tvYFu9brEttlRE2+61SM++a5o9XTzyvMtSKId+KL5MBZUdeBsyDCzL9UGSfR26fQVOYPTGn2uOEfEmSCgMG6jyL2hccYyZCH5.WLHYwe.eFozyr1YjRuTkYF44Dp9Vg75jpuQ+F4rh4YeU74Tk9PIk.u96fvPws0EQwrQYw.OjHqsHbXZeGmbVKRQ5AQhAvgLBtvX+mSXgFy5LPACmEbJuCXbx6FcxeAUpR13UvkKA+AkAmjznY4lDefXG11fCndxLuGDenxR9iiSSg0rwJ6kNlRsYWSecpk3.TGYXCykSRtMU5Le1NRlUPrNm8fZotvn0mDhtv7JVyySxdZoPKjMltBFZWqznof+p.7kzRVMODno97jvXt6gwB35gv91JHEfX7oJ+jb9rl8bSlUahDNX3VBpwF6Xud2DEPxysldXmSjWQ0p6q7Pg35HRRJzhM.86iD9HhuTboucFICo8rDI.UxSH9Bv8ig0tdnarzvj1wQvfJW56O8V8.GqrnNVcQcr1h53VKpiaunNtyh53t+0NZli7fAZQjsuDBcb6FIIgNNM3DnkaR4EhBSEG.2BzN7rIWcrDN8tlPenj52dLSO5blurH441JxMgJnRAp3zaahb9I7la51zny0ls5Z1ju1ELrHnotoYCMvEpb5691VcZ.in6Vobksdc4pu1qRQX.OgTu1qT5fjCnTHq6qV+MEisuJv8abUrRciIx0prgqPURn7WqVsxqClfP+IfedQuD

The Faust node in the Scriptnode Synth works in HISE
It doesn't work when the plugin is exported using the new Compile Project exporter
If I compile the dynamic library first and replace the Faust node with the compiled osc node, it makes no sound in the Scriptnode Synth, or in the compiled plugin
The compiled osc node works in ScriptFX in HISE, and if the plugin is compiled that way, it works, too
The RAW Faust node with osc.dsp works in HISE, but doesn't work when the project is exported that way using the new exporter - it's as if i doesn't exist, e.g. if it's put into ScriptFX in a Waveform generator, in HISE you'll just hear the sine (because it overrides the waveform generator output), but the exported plugin will play the sound of the waveform generator on midi input
if using the raw Faust node with osc.dsp in a ScriptFX and setting the network to AllowCompilation, the ScriptFX goes grey once the plugin export starts, remains bricked afterwards and in the exported plugin it's as if it's not there, e.g. waveform synth output still comes through
if instead of exporting the plugin we start the DLL compilation and set the flag to replace ScriptFX with hardcoded, it fails to do so and the network is not visible at all in the hardcoded FX dropdown, either in the wavetable synth FX section or in the master fx section, no matter if polyphonic or master fx

Can anyone else confirm this? If I got this right, this means that:

You cannot use Faust in a Scriptnode Synth, yet that particular Scriptnode environment still lets you load in a Faust node
You still need to run the DLL compilation, then manually replace your nodes in your network if you're not flagging networks as to be compiled (due to usage of global cables, for example)
there's no way to just enable AllowCompilation for a network and expect it to work

The Culprit?

So it looks like the issue is with hardcoded FX replacement is not working properly, as it invalidates the network from being able to be located in the hardcoded FX dropdown, Faust or no Faust

Takeaways

There should be no need for developers to have to manually swap Faust nodes all over the place with compiled versions. Faust nodes should compile anyway even if no AllowCompilation flag is set, and then be replaced with their hardcoded counterparts in the network (would this mean the new exporter runs the DLL compilation twice?).

Additionally, as mentioned earlier, editing a .dsp file and hitting F5 should refresh it all across the project(not just the current network) and regenerate the parameters. If that's somehow problematic, it should at least be doable with Shift+F5.

aaronventure

@aaronventure @Christoph-Hart the recent commit seems to have made some changes, so here's an update

HiseSnippet 1466.3oc6X0sTaaDEdElkDbnMMcRmzK0joWX5PLVleBzLcBfAm3oAvCxkz6XVKsFuCx6pQ6ZHNc588wpOB8QnOB8Mn8bjjskHNDpaKIYZMy.dO+n8aO64mOA+UFtzm6ehVJBC4FBwpPyHkGWqUQDqhsFDxIVKPcGHMcq0kIjjF6RrliZ3ZCYmAgLsl6SrrJ7LTm07yRh+76OcGV.S5wGKhPNVI73uPzSXFKs4VemHHnNym2RzKi0qtUCOkrlJP0GvQAZERHy6L1o7CXnYyPIOmo6Rr9ZZ0Ns8p14wqrgylNa93U2zwYsMZuZkpd9qs4Jqr9J9cptAa8M.mlaOegQE4ZX.5IVytixefaW0ExjM3XgVzNfiKbHtvNmHttJvGOhnTRsth.+QAHMDtnYBWERBW2mtuvWLR93v1mEqvdrGYCfVyjGdExAOmrvqRF3MAHYkARyl.o6Qc8hDglwZP7bGZCogG0gA2SYgRhsDqegVSAFHMk6wNiWOBVLxgRqWoxR1vuV7Ic5K8LBkzVIOPY3GJKsXwer37E+oh1WVUmNSTGtMQpf.dzDUioFQWkikj860lGsj84rf97QFBG+7wT5aOll8J2K4TmwPkrgTXNLjKeaIBjzPE7suuwtLCCuHRkA1ExiLBDBV6xOGJCRtVlmtKWelQEBEBuwcFjsn76GvL4SgvBsTEPLH28Fd4H0ByfrEh+ikWccg38nMEFutSFiyLALBQp+MvXZ03mP2qSGtmYL.mkV+Gt4J8JlV5A6YRM2vkNSrl6moiyuCi3grHdKUy.1fRZVuv.9Q.DWxtcfx6LWwq4uYQQXBB1AsnjWWlTxCzSSsybW6fhyUembjpuQHOcelIR7JnH7f98bg15d7ZonCjYMCVBkrtBtFSKbgYSwK9C3SpRGbsUpRmgJyT8c.2bgJ5r3aizuSrtEF5+Tp.6dIYAm3qCIGyizX1o0snUJC+P1NHPcQMUuPQZdKbsDKqoJXPXWkT3ghRrXH32tmpOruom.XnTKlH.Szc6qg1E9GJcAiimDBIZVGn7wIQ0YdPncPSloKVvfMbfrSdTYuQYoWBsiyUvnbxTvKYB9rGNwLI.BPsGFUv4kVEP8w0HrHvcvwzBlg8lRRewUPw4LHFVf1PeLp0iEPNFyOvS1e6FbOHQ4cnujcNOlZQ7I9KhW2QE0y9YbHXfsUbtBRF+50kjQ30ljwgdF.BshXRcnRycx9jc48DsTRtNmzK4Q0I5QNo6xM8k4ezIhxYEFJpCghL1c6sFJrZVgMYxbOLXctmz9XQ2vk+1S2SxfxVWNbx8OT6AAFLNmchQy9AZ9KE9ltNYcbr3pYE+bVjObG5kqedgqhpkyTR0ZtO3nZ8QvT6bijleHFcE3rj8jmyCfJ2XL94PoZGV+.yPo4K81WIUC6AN9h9HNzS+zS44xel3AZaiAHvOVx825Hd.moyTN9Ua8BnGHKZ+3djSUrv4u734Ide8kzD3Zi8Kr+3ksUg+qy1Zgzk1+Ocqab5VKPaw0lch3HImqAYK7f89itUNzNIxV4L3ZR0ZD.vc3VTn8p1PxAlE.vDwKmnYRa6PudW6XNdbybIdbySqE.O4F9CIwQovn+Qj+FZ1cwbkLrCSMNNOKGoQ2KvdNsXQmxMePvn7pG9BLf9vtI9Cvqip1wcxse+zJ+lnqwMwdzi4EoNwKoYKlAd6XIv4VFWOMOcebssC47jruwAyd.iwS77x+ndCGqNsNtxz53pSqiqMsNt9z53imVG23c6HRne69FUujZC3EaZtWbWCKqj2oItLgvgwq9vqQjLgGSsGIQxeUcX5PbeoNBrrXF7OIH4tIMZWFpDKiuMc5+DPh0qoKurccTmcR1pM9P9FavvhflZ3DAtuMzRrq8ya3tGLq2tZkpq8nJq7HmMJBLsUQlROTa7i2fxAh1ObwmTLk3f82ZqEkaCDOJUcIaktrR6UZ0UqrHXBg7mDbE1sE

This is a snippet that:

Has a waveform generator with a single ScriptFX
The ScriptFX has a single node, which is a Faust node, with a osc.dsp file which is just si.bpar(2, os.osc(440);
The network has Allow Compilation enabled

Export > Compile project results in a VST3 plugin that either pretends the node or the network doesn't exist, i.e. the saw synth plays on MIDI input

Export > Compile bricks the script FX inside the HISE project, which makes it go gray and the 440 hz sine changes to 200hz (???). If you click on the scriptFX and see that the network has been selected and instead select the compiled Faust file ("osc"), the sine will be at 440Hz correctly.

Exporting with Export > Compile project if the network was selected in the hardcoded dropdown results in it being bypassed and a saw synth plays on MIDI. If the Faust dsp file, in this case "osc" was selected, then it doesn't work either.

Faust nodes being in the networks that don't have Allow Compilation enabled still don't work when compiled with Export > Compile project

Triggering project compilation should not automatically replace ScriptFX instances with the hardcoded versions.

aaronventure

@aaronventure

Some further testing: I created a new project with the same setup

FaustTest.zip

Commit is 6bc83f40afce511cc97efe5390ead05538eb3bb6 from 9 hours ago.

Setting the network to Allow Compilation and attempting to Compile Networks as DLL results in

BUILD FAILED **

CompileC /Users/testuser/HISE/Projects/FaustTest/DspNetworks/Binaries/Builds/MacOSX/build/FaustTest.build/Release/FaustTest\ -\ Dynamic\ Library.build/Objects-normal/x86_64/include_hi_tools_02.o /Users/testuser/HISE/Projects/FaustTest/DspNetworks/Binaries/JuceLibraryCode/include_hi_tools_02.cpp normal x86_64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'FaustTest - Dynamic Library' from project 'FaustTest')
(1 failure)

ScriptNode Compilation Workflow and What to Do About It

GitHub - christophhart/HISE: The open source framework for sample based instruments

[UX] Module Browser Drag&Drop Needs Some Love · Issue #566 · christophhart/HISE

The Culprit?

Takeaways

41

1.7k

11.7k

101.9k