File system allow for binary write/load

Lindon

@ustk said in File system allow for binary write/load:

@Christoph-Hart On top of objectand string, Is it possible to add write/read methods for binary files, please?

yeah then we can all write our own expansion loaders to move ch1 files around...

d.healey

@Lindon +1

Christoph Hart

On top of objectand string, Is it possible to add write/read methods for binary files, please?

What's the reason for this request?

yeah then we can all write our own expansion loaders to move ch1 files around...

I'd rather add a special API for the decompressor for the .hrx archive format that you just feed the downloaded sample archive and it automatically extracts it into the sample directory that is chosen by the end user with a nice callback indication about the process.

d.healey

@Christoph-Hart said in File system allow for binary write/load:

I'd rather add a special API for the decompressor for the .hrx archive format that you just feed the downloaded sample archive and it automatically extracts it into the sample directory that is chosen by the end user with a nice callback indication about the process.

I retract my previous +1 and move it to here instead!!

Lindon

@Christoph-Hart said in File system allow for binary write/load:

I'd rather add a special API for the decompressor for the .hrx archive format that you just feed the downloaded sample archive and it automatically extracts it into the sample directory that is chosen by the end user with a nice callback indication about the process.

Yep me too... now just to be an ar*e - when can we expect this?

-oh and can we pre-fill the destination with the current sample location - users forget where they've put stuff so this would help them...

Christoph Hart

Yep me too... now just to be an ar*e - when can we expect this?

IIRC I started implementing something similar already for this API call:
https://docs.hise.audio/scripting/scripting-api/expansionhandler/index.html#installexpansionfrompackage

But I would suggest a dedicated method:

// Extracts the given file to the global sample folder with a callback about the progress
File.extractSampleArchive(callback)

// Example usage:

// Use this function to display the progress to the user
function extractProgress(data)
{
    // The data parameter will be a JSON object with the status
    // information (the exact structure is TBD, but it most likely will
    // contain all information from the internal extraction object
    // seen here:
    // https://github.com/christophhart/HISE/blob/8ef678ec2fcae0973cc269163404b1f9df967733/hi_lac/hlac/CompressionHelpers.h#L393
    Console.print(data.currentProgress / data.totalProgress)
     
     // returning false here will cancel the extraction
     // so you can give the user the ability to abort the process.
     return true;
}

// Let the user point to a .hr1 file
FileSystem.browse(FileSystem.Downloads, false, "*.hr1", function(f)
{
    // Call it with the file
    f.extractSampleArchive(extractProgress);
}

oh and can we pre-fill the destination with the current sample location

You can get the sample path with FileSystem.getFolder(FileSystem.Samples). Or just use FileSystem.Downloads as starting point.

ustk

@Christoph-Hart said in File system allow for binary write/load:

What's the reason for this request?

Write audio wav files. But I know Juce already has what's needed for that so it would be better to add those methods?

Lindon

@Christoph-Hart looks good. Cant wait (well I can obviously...)

So this is my plan, let me know if any of this assumes too much:

create an expansion
create my sample maps and samples for my expansion.
Compress the sample maps into a .hrx file set - I'm not sure here if I need to do this in a separate HISE project or I can use the "encode expansion" - which currently does nothing for me (I might be using it wrong)
Ship an installer and some .hrx files as an expansion
The installer will install the expansion (but not the samples) in the appData structure.
It will also put a JSON file describing the expansion in the AppData folder structure, this will include a flag to say the samples haven't yet been installed...along with (for my purposes) a list of all the presets in the expansion (and their tag data) and all the sample maps in the expansion (along with their tag data)
On start the app will get a list of all the available expansion JSON flies - so named: <myExpansion>_expansion.JSON.

using:

FileSystem.findFiles(FileSystem.getFolder(AppData),"*_expansion.JSON", false );

check in the JSON file to see if its samples have been installed, if not do this stuff you outlined above.

At this point I will have an app that knows about expansions, knows all about the presets and the sample maps and will have the samples in the global samples folder...

Which should be enough I think...all I need do is the {EXP}:: trick to load the sample maps when the user asks for them.

d.healey

@Christoph-Hart said in File system allow for binary write/load:

FileSystem.getFolder(FileSystem.Samples)

Does this update when changing expansion?

Christoph Hart

@Lindon you can also embed the expansion data (samplemaps, etc) as metadata into a HRX archive. Then all you need to do is to call ExpansionHandler.installExpansionFromPackage(var packageFile) and it will do all of the steps you mentioned without having to write a temporary JSON file, which looks a bit hacky to me. The suggested workflow then would be:

Create expansions like a sub HISE project
Encode the expansion, then run "Archive samples" and choose the expansion you want to encode. This step collects all samples that are referenced by any samplemap in the expansion and writes it into a hr1 file, then prepends the (encoded) expansion as header.

The end user's procedure looks like this:

Your installer just needs to install the main plugin and don't bother about the expansions. If you want to show a list of all available expansions, you might query your server to fetch a list, but I would suggest hardcoding or including any information about expansions (because it renders the whole concept useless since you then again rely on the state of your main plugin as it was shipped).
You can ship each expansion separately as one monolith file. The end user can then download it externally and just somehow point to the file and it will copy the embedded expansions to the AppData folder and extract the sample data to the right location.

@d-healey Why should it? It'll point to the global sample folder. If you need the expansion's subfolder, call Expansion.getRootFolder().getChildFile("Samples"). Do you want to actually store samples in there instead of the global sample folder?

ustk

Making a new thread since the original has been unscrupulously thieved by coders who desperately need to EXPAND to the rest of the world...

d.healey

@Christoph-Hart

Expansion.getRootFolder().getChildFile("Samples"). Do you want to actually store samples in there instead of the global sample folder?

That's a good solution. I'm thinking that users will want to store different expansion on different drives. Especially for large sample sets.

Christoph Hart

@ustk I think you can write Buffers to a audio file already. I‘ll look up the API tomorrow

Lindon

@Christoph-Hart said in File system allow for binary write/load:

@Lindon you can also embed the expansion data (samplemaps, etc) as metadata into a HRX archive. Then all you need to do is to call ExpansionHandler.installExpansionFromPackage(var packageFile) and it will do all of the steps you mentioned without having to write a temporary JSON file, which looks a bit hacky to me. The suggested workflow then would be:

yes - except....

I need to be able to write to this meta data for presets and for samplemaps, as the user can set samplemaps as favourites and set presets as favourites as well as add new presets that have a meta data tag called "User Preset".

So a rough example of (part of) this meta data JSON file is:

[
  [
    "Acoustic",
    "Factory",
    "Analogue"
  ],
  [
    "Attarkus",
    "Factory",
    "Analogue",
    "Metalic",
    "Guitar",
    "Favourite"
  ],
  [
    "Bassica",
    "Substrate",
    "Analogue",
    "Guitar"
  ],
  [
    "CleanCriminal",
    "User Preset",
    "Analogue",
    "Guitar"
  ]
]

So as you can see there are 4 presets here, the first array value is the preset name, the second its expansion ("Factory" is a virtual expansion - its what ships as the default product, and "User Preset" is a virtual expansion that only shows up if there are some user defined presets), then a set of N tags
which includes a tag called "Favourite"

So my preset browser looks like this ( the sample map selector looks similar):

So the user can select an expansion from the list on the left(including "All") then narrow their results with the Filters(tags) in the middle....

When a user selects a preset (from the list on the right) they can add it to their favourites, or remove it from their favourites if its already tagged.

When a user adds a preset, they get to name it and set any number of tags, and it will add the "User Preset expansion" automatically...

If you have a better way of adding in this functionality and writing it somewhere, I'm all ears..

Christoph Hart

Yeah, just create the JSON file on your server and query it on load (or if you let the user click a refresh button if you don't want it to call home automatically). This way you will always have an up to date list of expansions (otherwise you need to update the main plugin for each expansion that comes out after release). Then merge it (or use a separate JSON file) with the favorite property (or anything that comes from the user).

Lindon

@Christoph-Hart said in File system allow for binary write/load:

Yeah, just create the JSON file on your server and query it on load (or if you let the user click a refresh button if you don't want it to call home automatically). This way you will always have an up to date list of expansions (otherwise you need to update the main plugin for each expansion that comes out after release). Then merge it (or use a separate JSON file) with the favorite property (or anything that comes from the user).

well--- this is a bit confusing. I'm pretty sure I dont need a new plug-in each time I have a new expansion.

All I need do is get the plug-in to read the list of installed expansions with:

const var appDataFolder = FileSystem.getFolder(FileSystem.AppData);
const var metaDataFolder = appDataFolder.getChildFile("ExpansionMetaData");

const var fileList = FileSystem.findFiles(metaDataFolder, "*_expansion.json", true);

This gives me a list of all the currently installed expansions, which is actually what I care about, and each *_expansion.json file tells me all about the "stuff" in the expansion e.g:

{
	"EName":"Substrate",
	"ELoadState": 0,
	"EPresetData" : 
	[
		{"PName" : "AccBass",
		 "PTags" : ["Bass", "Analogue"]
    	},
    	{"PName" : "ArtBass",
    	 "PTags" : ["Bass","Analogue","Metalic","Guitar"]
    	}
	],
	"ESampleMapData" :
	[
		{"SName" : "Acoustic",
		 "STags" : ["Analogue","Smooth"]
		},
		{"SName" : "Attakus",
		 "STags" : ["Metalic","Grind"]
		}

	]

};

so to read this file (which is substrate_expansion.json) I just need to say:

const var myData = fileList[0].loadAsObject();

I admit I will nee dot ship this *_expansion.json file with each expansion, but I will be storing it in a pre-known folder (the AppData folder)

or am I missing something ?

d.healey

@Christoph-Hart said in File system allow for binary write/load:

@d-healey Why should it? It'll point to the global sample folder. If you need the expansion's subfolder, call Expansion.getRootFolder().getChildFile("Samples"). Do you want to actually store samples in there instead of the global sample folder?

Bump Bump - how do we handle the common situation when users store samples across different drives?

d.healey

@Christoph-Hart said in File system allow for binary write/load:

Encode the expansion, then run "Archive samples"

Is expansion encoding supposed to be implemented? I tried the button in the expansion toolbar but it always tells me I need to set an encryption key (which I have done).

I also had a look in the source and noticed that the expansion handler encode function looks like a placeholder, but there are other encoding functions in ScriptExpansion.cpp which look more complete.

File system allow for binary write/load

26

1.8k

10.4k

90.7k