My setup for building HISE plugins with AI (YMMV)

[dannytaurus]

Rewind a couple of years ago and I had no idea what I was doing with any of this. I was pasting HISE scripts into a ChatGPT window, getting back something 80% right, pasting it into HISE, pasting the error back, and so on. It worked, sort of, but it was slow and the model never really knew what my project looked like. Later I moved to Cursor, so the model could at least see all my code. But since then I've moved to doing everything in the terminal instead of a full IDE.

This post is the setup I've ended up with for building Sublime, the plugin I released this month. And it's the setup I use for all other coding tasks too - building static websites, writing Ruby on Rails web applications, making Mac apps, etc.

This post is based on Claude Code because that is what I use, but pretty much everything here has an equivalent in the other tools. If you already run an agent locally you won't learn much here. If you're very new to AI, or still in a Claude/Codex/Gemini browser tab, read on.

As with all setup posts like this, your mileage may (very likely will) vary.

1. The layers, because the names are confusing

The single most confusing thing when you start is that "AI" is at least four separate things stacked on top of each other, and people use the names interchangeably.

The models and their overlords

The model is the thing that actually thinks. Each overlord company makes their own model(s). Anthropic (company) makes Claude models (Opus, Sonnet, Haiku, and now the Fable/Mythos tier), OpenAI (company) makes GPT (model), Google makes Gemini. The model is just text generator, lovingly known in some circles as a 'bullshit generator' . It takes text in and gives text out - that's it.

Christoph's rule of thumb in the agentic thread was that Opus 4.6 or an equivalent was the floor for HISE work, and anything smaller is a false economy. I agree, but that was a while ago, so adjust your baseline accordingly. For my work, I only ever use the best model available from the provider I use. Right now, that's Anthropic's Fable 5.1.

The harness

The harness (also called the agent, or the CLI) makes the model feel conversational and personal. It reads your files, runs shell commands, edits code, calls tools, and loops until the job is done. It's really only passing text to and from the model but it's also adding all the context needed for a continual, useful conversation.

Claude Code is Anthropic's agent/harness. Codex is OpenAI's. opencode is the open-source one Christoph uses. Cursor, Zed, VS Code with Copilot, and Google's Antigravity are editors with a harness built in. The harness is a choice independent of the model: opencode and Cursor can run Claude and other models, Claude Code can only run Claude.

I've been very happy with Claude Code in the terminal for many months now, only tweaking the actual terminal app that I use - currently using herdr.

The context files

These files are plain markdown you leave in your folders for the harness to read at the start of every session. Claude Code reads CLAUDE.md; most of the others read AGENTS.md, and the two are converging (Christoph said his hise-cli templates will be agents.md). This is the layer that turns a clever generalist into something that knows YOUR project. Most of this post is about these.

The tools

The things the agent/harness can call out to beyond the shell: MCP servers (a protocol for plugging a tool into any harness), language servers, and plain command-line programs. For HISE this means Christoph's HISE MCP server, hise-cli, and the HISE REST server inside the editor.

Show me the money!

A subscription (Claude Pro or Max, ChatGPT Plus, a Cursor plan) gives you a monthly allowance with rate limits. An API key is pay per token, no ceiling, but it's usually more expensive for a comparable amount of access. Beginners should start with a subscription so a runaway loop just means you run out of tokens rather than facing a massive API bill at the end of the month.

I'm currently on the Claude Max plan at £90/mo I was on the Pro plan at £20/mo for quite a while, but when I stepped up production I started running out of tokens too early. I do get close to the token limit now, on £90/mo, so the next step up for me is £180/mo. But in terms of productivity gains, and compared to how much human help you could hire for £180/mo, the cost is tiny.

Usage on Claude is rate-limited by 5-hour and weekly windows. Supposedly so the overlords can manage bandwidth of their mega-cloud-thinking-machines, but the relaity is usually fine for regular work.

The highest-end Claude Fable model has its own rate limit because it uses a lot more resources than the lower Opus/Sonnet models. As I write this on a Saturday afternoon, I'm at 95% Fable usage for the week, which resets on Tuesday, so I'll definitely hit the limit before then.

2. Why local beats the browser

The browser chat is the model with limited abilites and environment.

In a regular AI chat session, you play the agent. You copy files in, copy answers out, run the code and report back. Every round trip loses detail, and the model only ever sees the fragments you chose to paste.

In a Claude Code browser session at least you can connect to your project repo on GitHub. But you're still working in an isolated environment. The model can only see your repo code. You migth think that's all you need, but the real power in agentic coding comes from all the things and actions around the project repo.

Running the agent in the project folder changes the job completely:

• It reads the full project. The whole script tree, the UI XML, the presets, project_info.xml, samples, untracked files, everything.
• It reads the HISE source. ~/Code/HISE is on the same local disk. When the docs are vague about a LookAndFeel obj field or what setIntensity expects, the agent opens the .cpp and reads the implementation. This is the single biggest quality jump I've had. The majority of the "HiseScript hallucination" problem goes away when the model can check the HISE source.
• It runs things. Shell scripts, git, node checkers, sox for audio analysis, the build. It can write a script to migrate 125 presets and then run it, instead of giving me a script to run.
• It compiles and test the UI for real. With the HISE running and the REST server on, it can recompile the project and take a screenshot of the interface, and iterate until it's right.
• The conversation is attached to the files. Decisions get written into the repo (more on the ai/ folder below) instead of staying isolated in a chat tab.

One correction to something I wrote in Christoph's thread back in March: I said terminal agents lose the chat history when you close the window. I since learned that Claude Code keeps every session and the /resume command lets you pick an old one up with its full context. Cursor's tabbed history is still friendlier, but the capability is there. And at the end of the day, this is all just text files on your machine. They're shuffled away in hidden Claude folders, but they're there. You can back them up, make a searchable archive of all your agent sessions from day one (I just did that), and move them between machines.

3. The context files, and why there are three of them

This is the heart of the setup. The harness reads context files from several places and stacks them, so each file can hold the rules that belong at that level. I have three, plus one import.

NOTE: If you don't use Claude, your file name and location will be different, for example ~/.codex/AGENTS.md for Codex. Also, any paths mentioned are Mac - adjust for Windows/Linux.

3a. The global file: ~/.claude/CLAUDE.md

Rules about ME and this MACHINE, which apply to every project. Nothing about HISE lives here, and nothing about inidividual projects. Examples from mine:

• Never commit unless I say "commit" in the current message. Finishing a task is not permission to commit.
• Never run a build unless I asked for one in this message. A build overwrites the plugin I may be auditioning.
• Never touch files outside the repo.
• Never give me a multi-line terminal command. When I paste them they often break.
• No em-dashes or en-dashes, anywhere, ever. Hyphens only. (I only use characters I can type.)
• Assume I know nothing about Windows. Give every Windows step click by click.
• Audio analysis is pure Python, after sox.

Notice these are mostly "never do X without asking". Each one was added the day the agent did X. That is how the file grows, not by sitting down and writing all the rules at the start, but by writing one line each time something bites.

Take the MERGE_HEAD issue I had recently. When that was resolved I asked Claude to add a note to ~/.claude/CLAUDE.md so that every new coding session is aware of the problem and the solution.

3b. The plugin repo: my-project/CLAUDE.md and my-project/HISE.md

This is a separate CLAUDE.md file with specific context for this project.

The HISE knowledge lives in a second file, HISE.md, pulled in with one line at the top of CLAUDE.md:

@HISE.md

HISE.md is about HOW TO WORK ON A HISE-BASED PROJECT in general (not how to work on the HISE source code itself - that's below). What the MCP server is for and when to use it, where the HISE source is and which directories answer which questions, the audio-thread timer pattern, the HiseScript gotchas I've hit (inline locals shadowing namespace members, getModulator being onInit-only, scriptnode Smoothing being in seconds whatever the display says). Nothing in it names the plugin, so I can copy it into the next plugin repo unchanged.

CLAUDE.md itself is about THIS plugin and the decisions that are not derivable from the code: the naming convention for component ids and plugin parameters, why the top-bar panel is declared after the main sections (VST3 parameter order), which switches are fronts for hidden sliders, which features are hidden for 1.0 and how to ship them later, the pre-release checklist, how the two XML files stay in sync. It is long (7500 words) and every paragraph has a date and a reason, because "why" is what the agent cannot reconstruct from the code.

The split matters. If you mix the HISE mechanics in with the project decisions you cannot reuse them. Keeping them in their own file means the next plugin starts with everything I've learned so far.

3c. The HISE source clone: ~/Code/HISE/CLAUDE.md

A third file, for when I am working ON the HISE source rather than WITH it in a project. It says which branch is mine, which tracks upstream, that builds are mine to run, that the agent may commit only when asked. Then the C++ conventions (unity builds, tabs, Allman braces, the USE_BACKEND / USE_FRONTEND split, the lock-free threading rules), and a pointer to the guidelines/ folder Christoph added for exactly this purpose.

The reason this is a separate file and not a section of the plugin repo's file is that a session started in the plugin repo reads the plugin rules and a session started in the HISE clone reads the HISE rules. Each one gets its own proper set of rules.

4. Giving the agent reach beyond the project folder

By default the agent/harness works in the folder you started it in and asks before reading outside it. For HISE work you want it to reach:

• The HISE source (~/Code/HISE). The agent greps hi_scripting/scripting/api/ for the real signature of a scripting call, hi_core/.../floating_layout/ for the FloatingTile content types, and so on. My repo file tells it which directory answers which question so it doesn't repeatedly search the whole thing.
• The HISE MCP server (Christoph's hise_mcp_server, cloned and built with npm run build). Registered in the repo's .mcp.json so any harness that opens the folder gets it. The docs tools are always available and are much cheaper in tokens than fetching the HISE docs website; the runtime tools (recompile, screenshot, read/set component properties) work when HISE is open with Tools > Enable REST Server.
• hise-cli, which is absorbing the MCP server's runtime side. Compile over REST works from the CLI since 0.8.6 / HISE 4.9.3.
• The installed plugin's AppData folder, read-only, for checking what the shipped plugin actually wrote.

In Claude Code the permission for an outside path goes in the project's .claude/settings.local.json (or claude --add-dir ~/Code/HISE at launch). Mine has the AppData folder and the MCP tools allow-listed; the HISE clone itself I just approve when asked, which in practice is once per session. Other harnesses have the same idea under a different name.

Two habits worth adopting:

1. Tell it which world it is in. The HISE editor and the exported plugin are different binaries with different preprocessor flags. A define you turned on in the editor's jucer does nothing in the export unless project_info.xml says it too. I shipped three months of linear filter modulation that was logarithmic in the editor before I learned this. My repo file now has a paragraph on it, and the agent catches it for me.
2. Decide who edits what. The CLI and MCP edit the project in HISE's memory; disk edits need a reload. Mixing both in one session loses work. I do script edits on disk and let HISE reload, and use the runtime tools for UI inspection and screenshots.

5. The project "brain" in an ai/ folder

This is just a folder of markdown files in the repo, committed like code.

ai/ holds durable write-ups of things that took real digging. Why plugin parameters appear in the order they do across five DAWs, what the 1.0 preset format froze and how to add controls after it, experiments that were tried and dropped (with the reasons), extensive guides for setting up and using test machines. Each file is one topic, dated, with the decision AND the mechanical reason. ai/README.md indexes them in one line each.

The HISE clone has the same folder with its own conventions: specs/ for how a subsystem works anchored to file:line, discovery/ for things found out that are not in the docs, todo/ for diagnosed bugs with a proposed diff that has not landed yet.

The agent reads these when a question touches the topic, and it writes them when we finish an investigation. Two rules I enforce: the notes are the agent's wording, not my feelings, so they are quoted as "the note says", never as "you decided"; and when a note contradicts what I say now, I win and the note gets fixed.

Claude Code also has a per-project memory of its own (short files it writes for itself about how I like to work: "never commit unasked", "flag anything that breaks saved DAW sessions"). That's a harness feature and the files live outside the repo. Don't allow the agent to rely too much on this 'memory'. It's useful day to day, but the repo ai/ folder is the one that survives switching tools.

6. The pipeline

Everything between "the project is done" and "there is a working installer" is scripts I had the agent write and I never have to think about. A pre-release checker (version bumped in all three required places, no TEMP code left, UI XML and project XML in sync, factory preset count matches the user guide, etc.), the export via HISE's batch compile, pluginval with strictness setting, codesign and notarise the exported plugin, build the pkg, render the user guide PDF, upload it to file storage. One command. The Windows build has a twin script that runs the same checks.

None of that is HISE knowledge. It is the kind of tedious shell work the models are genuinely excellent at, and it is a good first project if you are nervous about letting an agent near your script code - let it automate your build first.

7. Random stuff

Planning. When chatting with the agent, I use phrasing like "Let's discuss how to ..." or "Let's plan a new feature called XYZ". This not only prevents the agent from writing code right away, it also frames the task as discovery/discussion rather than unplanned action.

Code IDE. You may notice I haven't mentioned using an IDE. That's because I don't use one anymore. I read the code written by Claude right in the terminal, either with git diff or in the chat output. On the rare occassion that I want to hand edit code I open that single file in CotEditor, a basic free code editor on Mac. I do regularly edit the plan files and feature specs - they're plain Markdown and form the contracts between me and the agent. But editing code, hardly ever these days.

8. If you are starting today

1. Pick a harness and a subscription. Claude Code with a Claude plan if you want the path of least setup; opencode if you want to swap models; Cursor or Zed if you want to stay in an IDE.
2. Clone HISE next to your project - you should have it anyway for building HISE.
3. Clone and build the HISE MCP server, add it to the project's .mcp.json.
4. Write a ten-line CLAUDE.md (or AGENTS.md) in the project. Include what the plugin is, where HISE is, "look up every API call before writing it", and your two or three personal rules. Add a line every time it does something you did not want.
5. Keep the global rules (about you, your machine) in the global file and the project rules in the project. Don't mix them and don't repeat details in both files - that drifts real fast and starts to contradict.
6. Start with the pipeline or a simple function change, not the DSP.
7. Commit often. The diff is your safety net. And if you're like me, insist that the commit command comes only from you.

Happy to answer questions, and I would love to hear how other people have arranged theirs, especially the opencode and Zed crowd. @iamlamprey's custom harness setup sound cool too.

[Christoph Hart]

@dannytaurus nice post buddy! I would add the Pi agent as a viable harness option as I found this to be the most token efficient one - although it might not work with a claude sub as they restrict sub usage to claude code last time I checked (and subsequently cancelled the sub and went over the the street to ChatGPT.

One detail: if you use the hise-cli to edit external files instead of the native edit tool you get a diagnostic back which uses the shadow parser I‘m about to rewrite :) I found that just giving direct feedback for most obvious coding issues saves you from spamming your md files with HISE gotchas.

[iamlamprey]

@dannytaurus great writeup :)

I can't share the source for the harness because it contains a web-exploitation module & toolkit I use for bug-bounties and could very easily be used inappropriately / illegally if a bad actor got hold of it.

In general it's a simple harness written in Python using the textual package for the UI. It contains all of the standard tools like read_file, write_file, run_command etc, plus some extra goodies:

Git Tools

Includes things like diff and status to verify the code it wrote, as well as reset in case it breaks something. It can't commit or push or otherwise modify the repo directly.

Browser

It includes the camofox browser, which uses various techniques to bypass turnstiles / captchas and bot-detection. It's not 100% successful but it's able to consistently browse sites like reddit, run web-searches etc.

ComfyUI

The harness contains additional custom tools for controlling a local ComfyUI instance directly, so I can have the harness create images, videos, edit things etc. Probably one of my favourite features as I get access to a variety of local models for product images, videos etc. I've been using it to make ad creatives for meta.

Ponytail

It leverages the popular ponytail skill in the system prompt, basically a checklist to help improve the quality and consistency of the code:

Before adding code, walk this ladder and stop at the first solution that fully holds:

1. Does this need to exist? Do not build speculative work.
2. Does the standard library already solve it? Use that.
3. Does the platform or framework solve it natively? Use that.
4. Does an already-installed dependency solve it cleanly? Reuse it before adding machinery.
5. What is the smallest readable implementation that fits the codebase?

And later:

## Editing Files

Make file changes with `write_file` — it creates or overwrites in a single call, so there is no separate "create" step, and parent directories are created automatically. Prefer `write_file` over shell redirection for anything that creates or replaces a file.

For surgical edits to an existing file, use a Python script via `run_command`: read the file, transform the text, write it back — preserving the file's existing line endings (e.g. CRLF). Read the file before editing it; never write blind.

This combination of Ponytail and surgical Python edits noticeably improved the quality and consistency of the code.

Verification

The system prompt includes things like:

Verify your changes work, but scale the verification to the size of the change — verification should never become the bottleneck of a session.

- Small, localized fixes: run the smallest check that exercises the changed behaviour — compile the affected target, run a single filtered test, or do a quick manual check. Do not run the full test suite or a full rebuild for a small fix.
- Major fixes, rewrites, and new features: run the full build and relevant test suite. This is the only case that justifies building a broader verification harness.
- Prefer the project's existing test/lint/build commands. Do not build verification harnesses (new tools, pipelines, or persistent test files) unless the change is major, a rewrite, or a new feature.

This sort of pessimistic critic-gating is a combination of GAN-style adversarial gating, and some of the stuff in hermes-agent.

Deepseek Harness

I (the agent) also ripped some of the guts out of DSH: https://github.com/deepseek-ai/deepseek-harness to improve the cache hit rate and context compression (doesn't work 100% but I haven't bothered to go in and fix it yet).

Models

It's hardcoded to run Deepseek Flash via API, which costs cents on the dollar at 80-90% of the quality of the "western" models. For my personal uses it has outperformed basically everything else, including Gemini.

It can also use Ollama, either locally or one of their cloud models. As well as strata: https://github.com/niko1221/strata

Unfortunately my PC is not amazing for local models, 12gb VRAM and 32gb system RAM can only just run strata with nothing else open, the instant I open a web-browser or anything it OOMs.

Security / Safety

The harness is initialized by cd'ing to a directory and launching it. Only the read tools work on files outside of that directory (and sub-directories), if the agent tries to write files or do anything sketchy to anything outside the workspace - it gets blocked.

Example Project

I'm noodling at a harmonic + stochastic model trying to recreate my sample libraries without any actual samples. I put some of the original samples in the workspace and asked it to do some web-research, write up a plan for whichever method it thinks is the best and start working on the model.

It researched things like DDSP and Loris, wrote up a plan and used Python to continuously analyze the generated synthetic output against the original samples until it was happy. It took 1-2 extra rounds but it was able to accurately recreate my guitar harmonics instrument as a synth with infinite round-robins and an extended keyrange, all for just 100kb of data (no torch / DDSP used, it just researched it to understand the harmonic + stochastic decomposition).

I think the main takeaways are:

• Secure the harness to prevent external writes
• Be intentional with tooling, too few restricts the agent, too many tools confuses it.
• Iterate on the system prompt so the agent understands the available tools and how to use them
• Test it on real work and incrementally improve it to fit your specific use-cases

[Christoph Hart]

@iamlamprey have you tried the pi agent? It‘s also highly customizable and embeddable - I actually integrated it into hise-cli for fun and hardwired the tool usage to directly call the Rest API of HISE.

[iamlamprey]

@Christoph-Hart Nah, I did look at it among some other ones and settled on hermes before rolling my own. embedding use-cases like that sound very useful though

[dannytaurus]

@Christoph-Hart said:

I would add the Pi agent as a viable harness option

I've tried a handful of other harnesses but ended up feeling most comfortable in Claude Code.

The beauty for CC for me is the subscription. The amount of token usage I get compared to paying API pricing in other harnesses is substantial.

I do want to try some of the cheaper models in another harness though, to check what the quality is like compared to Fable. I'll check out Pi, looks cool.

[David Healey]

@dannytaurus Thanks for the detailed post. I'm setting up herdr now, looks like Christoph was right about Pi, you pay extra to access Claude through third-parties.