@HISEnberg sure I'm holding back about 50-60 commits with that stuff, but as soon as the MVP is solid I'll let you all play with this. The current feature I'm toying around with is the ability to create a list of semantic events that will create a automated mouse interaction:
{
{ type: "click", id: "Button1", delay: 0 }
{ type: "drag", id: "Knob1", delta: [50, 60] , delay: 500 },
{ type: "screenshot", crop: "Knob1" }
}
AI agents sends this list to HISE, HISE opens a new window with an interface, simulates the mouse interaction that clicks on button1 ( had to hack the low-level mouse handling in JUCE to simulate a real input device for that), then drags knob1 and makes a screenshot. Evaluates that the expected screenshot uses the code path of the LAF it has send to HISE previously, then realizes that the path bounds need to be scaled differently and retries it until the screenshot matches its expected outcome. If it just needs to know whether the knob drag updated the reverb parameter, it fetches the module list from HISE and looks at the SimpleReverb's attribute directly - this won't waste context window space for the image analysis.
It's that feedback loop that allows it to autonomously iterate to the "correct" solution (aka ralph loop), so even if it only gets it 90% of the time right, it will approximate to 99.9% after three rounds of this.
