Agentic coding workflows

[Christoph Hart]

The XML/node structure of HISE is ripe for AI collaboration I believe.

Actually, the XML structure is not the best target for this as this is a stale file that doesn't reflect the actual state in the HISE instance. The MCP server can query the entire component tree and fetch the properties, so you can select (!) some buttons in HISE just tell him:

"Move the button row to the top"

The agent will then call hise_runtime_get_selected_components, which returns a list of JSON objects for each selected component (like the actual selection (!) you did in the interface designer), apply the modification and send it back using hise_runtime_set_component_properties. then call hise_runtime_get_component_properties to fetch the current state again and verify it against it's expected position. This is a trivial example and the verification loop is unnecessary here, but you can imagine how that scales up to be a real workflow boost.

[dannytaurus]

@Christoph-Hart I find ChatGPT to be surprisingly good at generating HISE scripts, but that might be the accumulation of dozens of prompts and my resulting corrections.

Also, I use Claude in a Cursor project of the full HISE repo. I ask it things like "In HISE script, how would I do XYZ?" and it usually gives a very good answer.

I believe that asking a model that has access to BOTH the full HISE source AND your current project would produce high-quality results.

[dannytaurus]

@Christoph-Hart One of the reasons that AI coding works so well with Ruby on Rails web apps is because Rails has a huge amount of convention. It's heavily opinionated, uses standard libraries for lots of stuff and has a clearly defined directory structure.

This means AI spends way less time (and tokens) deciding HOW and WHERE to implement a feature than in something like a vanilla Node project, etc.

HISE has a similarly opinionated and well-defined structure, so I think that can be used to our advantage.

[dannytaurus]

I'll also say that although I've seen many horror stories about how much people are spending on AI coding subscriptions, I'm working very happily within a $20/month Cursor plan.

I use Cursor for 2-4 hours a day, pretty much every day, and I haven't hit the plan max since I started paying 18 months ago (Aug 2024).

I do think the fact that I'm mostly using it to work on a Rails app (see post above) helps moderate token usage, and also I keep conversations fairly short, moving to a new chat when the context changes.

Thorough rules files help too, although I'm still fighting to get it to follow certain rules. Seems a bit picky!

[Christoph Hart]

@dannytaurus lol I burned through the 100 bucks Claude Max subscription in 3 days.

[dannytaurus]

@Christoph-Hart Haha, you're in the honeymoon period!

And I suspect you're diving way deeper with it than most of us are.

[HISEnberg]

@Christoph-Hart Woah I am gonna have to try this out

[Christoph Hart]

@HISEnberg sure I'm holding back about 50-60 commits with that stuff, but as soon as the MVP is solid I'll let you all play with this. The current feature I'm toying around with is the ability to create a list of semantic events that will create a automated mouse interaction:

{
  { type: "click", id: "Button1", delay: 0 }
  { type: "drag", id: "Knob1", delta: [50, 60] , delay: 500 },
  { type: "screenshot", crop: "Knob1" }
}

AI agents sends this list to HISE, HISE opens a new window with an interface, simulates the mouse interaction that clicks on button1 ( had to hack the low-level mouse handling in JUCE to simulate a real input device for that), then drags knob1 and makes a screenshot. Evaluates that the expected screenshot uses the code path of the LAF it has send to HISE previously, then realizes that the path bounds need to be scaled differently and retries it until the screenshot matches its expected outcome. If it just needs to know whether the knob drag updated the reverb parameter, it fetches the module list from HISE and looks at the SimpleReverb's attribute directly - this won't waste context window space for the image analysis.

It's that feedback loop that allows it to autonomously iterate to the "correct" solution (aka ralph loop), so even if it only gets it 90% of the time right, it will approximate to 99.9% after three rounds of this.

[iamlamprey]

This was all a big surprise to wake up to.

Altar had quite a bit of vibe-coding involved, specifically on the modular drag-n-drop FX panels, and some of the third-party DSP nodes. I used various models available on Github Copilot (mostly Claude) because it seems to handle multiple files / larger context window better than the standalone models.

That being said, it still required quite a bit of manual fixing up. A common one mentioned above was using var instead of local with inline functions etc, and it would default to using JUCE buffers instead of the templated SNEX methods which I assume comes with a performance hit.

I think a good idea would be to establish some sort of global guiding prompt which includes some of the most important best practices for HISE and appending that to individual case-specific prompts eg:

"Use this bestPractices.md to familiarize yourself with the HISE codebase, remember to make use of the existing templated functions in the auto-generated files and only use JUCE methods as a fallback" etc etc

[Lindon]

@iamlamprey said in Agentic coding workflows:

"Use this bestPractices.md to familiarize yourself with the HISE codebase, remember to make use of the existing templated functions in the auto-generated files and only use JUCE methods as a fallback" etc etc

this.

Perhaps it might be worth thinking about how this is integrated in to the agents approach for every prompt it gets issued, a little like the MCP server interaction above...

Big warning: Im even more of a noob at this stuff than I think anyone here - I havent used AI for any coding work - so watch out I may be talking bullsh*t