HEADS UP: Server.downloadFile() issue on macOS Sequoia - and a solution

d.healey

Yeah it's the crude method HISE uses for checking if there is an active internet connection. It's also a big issue for users in China because the great firewall doesn't always allow the requests through.

I made a feature request back in January to add proper network detection in HISE by getting the status from the operating system. Someone else has already done the work to implement it as a JUCE module, it just needs integrating into HISE.

[feature request] Network connectivity checker

Currently we can use Server.isOnline() to see if there is an internet connection. This works by pinging google's server and seeing if it gets a result. It's ...

Forum (forum.hise.audio)

tomekslesicki

@d-healey that would be cool, but actually just changing the check from http to https would make it more mac-friendly. @Christoph-Hart

There’s also a question of whether to base url is passing the https part properly, but that’s too complicated for me to check. As I mentioned, the download callback wasn’t working for some Sequoia users way before I added the online check, so there’s definitely something going on.

Anyway - changing the build settings in xcode is a way around it.

d.healey

@tomekslesicki This reminds me actually. I added this to my custom plists for MacOS builds ages ago which bypasses the http/https security restriction

<plist><dict><key>NSAppTransportSecurity</key><dict><key>NSAllowsArbitraryLoads</key><true/></dict></dict></plist>

Christoph Hart

but actually just changing the check from http to https would make it more mac-friendly.

Done. I also agree, the other suggestion from Dave is overkill, plus it's a GPL licensed module so I would have to "get loosely inspired" by it to not mess up the licensing situation.

d.healey

@Christoph-Hart said in HEADS UP: Server.downloadFile() issue on macOS Sequoia - and a solution:

the other suggestion from Dave is overkill

It would solve the issue for Chinese users getting timeouts. It also would allow us to differentiate between mobile data and wifi/ethernet so that the user can choose to only download on unmetered networks.

But yeah I forgot about the license thing.

aaronventure

@d-healey said in HEADS UP: Server.downloadFile() issue on macOS Sequoia - and a solution:

I made a feature request back in January to add proper network detection in HISE by getting the status from the operating system. Someone else has already done the work to implement it as a JUCE module, it just needs integrating into HISE.

Background Task

ping -c 1 1.1.1.1 >/dev/null
echo $?

If it returns 0, you're online.

d.healey

@aaronventure How is that different from HISE pinging google's server?

aaronventure

@d-healey It's not really, but how would you check if you're online if not by connecting to another server? 1.1.1.1 is a CloudFlare IP which uses AnyCast, so it automatically connects to the nearest operational server, vastly reducing a chance that you're making a call when the server appears to be down.

I Imagine that's why Google was picked: the chance of downtime being so rare.

This is supposed to be even more robust, with the benefit of the latency being almost non-existent in most countries where people buy audio plugins.

d.healey

@aaronventure said in HEADS UP: Server.downloadFile() issue on macOS Sequoia - and a solution:

but how would you check if you're online if not by connecting to another server

Ask the OS if the system is currently connected to a network.

aaronventure

@d-healey that's true, but how do you think the OS does the check? I would bet it's something like this. Maybe macOS is calling www.apple.com and Windows is calling microsoft.com, as that also confirms DNS resolution and HTTPS working.

d.healey

@aaronventure How it does it doesn't matter, the problem is that every time you ping from in HISE and you don't reach the server you have to wait for the timeout time. I had a user in China who kept getting hit by 20 second delays every time I was calling Server.isOnline() - which I was doing in a few places in my login/download chain.

If I could just ask the OS are you online, I get an instant yes/no. The checking, however it happens has already been done - it's probably reported by the router or access point but I'm not sure.

My solution at the moment is I call Server.isOnline() once when the plugin loads and store the result. However if the user goes offline during the session then this will no longer be true, but I can handle that more easily than the 20 second delay.

It's also possible to reduce the server timeout time, but you can only take it so low before you start giving users on slow connections the wrong result.

This method would also allow for a broadcaster that reports when the system goes online/offline - I think I said this in my feature request post too.

aaronventure

@d-healey Ah I see your point. Also I figured if you're making a channel strip plugin or something and a user just happens to have 100 instances in a project, it could get interesting on load.

I did some more digging. On macOS, it appears that

/usr/sbin/scutil -r $?

will check your internet connection without sending any packets. It returns "Reachable" or "Not Reachable".

If you call

/usr/sbin/scutil -r -W $?

the task will run until terminated and will respond with Reachable or Not Reachable whenever the network status changes (I just tested it), which is pretty much like a broadcaster.

@Christoph-Hart you can implement this instead of having to clean room that module. I'm sure Windows has a similar mechanism.

d.healey

@aaronventure said in HEADS UP: Server.downloadFile() issue on macOS Sequoia - and a solution:

I did some more digging. On macOS, it appears that

Oh that's nice and simple.

I bet ChatGPT knows the magic spell for Windows too, I'll ask it tomorrow if you don't get there first :)

aaronventure

@d-healey said in HEADS UP: Server.downloadFile() issue on macOS Sequoia - and a solution:

I bet ChatGPT knows the magic spell for Windows too, I'll ask it tomorrow if you don't get there first :)

Hah, sure, but I'm too lazy to go test it right now.

So here's the reply :

One-shot check (returns Boolean, no packets sent)

$nlm = [Activator]::CreateInstance([type]'Microsoft.Windows.NetworkList.NetworkListManager')
$nlm.IsConnectedToInternet   # $true = online, $false = offline

Continuous monitoring (no polling)

$nlm  = [Activator]::CreateInstance([type]'Microsoft.Windows.NetworkList.NetworkListManager')

Register-ObjectEvent -InputObject $nlm `
                     -EventName   'ConnectivityChanged' `
                     -SourceIdentifier NLM `
                     -Action { 
                         if ($Event.SourceEventArgs.IsConnectedToInternet) {
                             Write-Host 'online'
                         } else {
                             Write-Host 'offline'
                         }
                     }

# keep the session alive
while ($true) { Start-Sleep -Seconds 3600 }

HISE-native integration of this might be better than manually launching Background Task as then it can just start a process and every instance of the plugin can read from the process instead of having every plugin instance start its own monitoring process through BackgroundTask (as I don't think BackgroundTask can connect to an existing one?)

tomekslesicki

@Christoph-Hart is this a part of the latest develop branch commint?

tomekslesicki

Ok, a quick update on the subject - the XCode transport solution only works for some users. For others, the download callback returns this.data.finished = true && this.data.sucess = false even before anything can be downloaded. This is happening both on Windows now, and macOS, for about 0.5-1% of users, for everyone else things are working fine.

Here's a very simplified overview of my download logic:

At the beginning of the scrip, I call:

Server.setBaseURL("https://");

which is used for POST calls, and then this download callback, with the url aprt being the rest of the url, after the https part.

inline function startDownload()
{
	// Download samples and content
	for (i = 0; i < plugincontent.length; i++)
	{
		local files = workingFolder.getChildFile(plugincontent[i]);
		downloads[i] = Server.downloadFile(url + plugincontent[i], p, files, downloadCallback);
	}	

	// There's just one zip file and it's always the first in the plugincontent array
	zips[0] = workingFolder.getChildFile(plugincontent[0].replace(subfoldernamefull).replace(subfoldernamedemo));
	
	// Show update progress panel
}


function downloadCallback(obj)
{
	if (this.data.finished != true)
	{
		// Show and update download progress
	}

	// Connection lost
	if (this.data.finished && this.data.success == false)
	{
		// Show error message
	}

	if (this.data.finished && this.data.success == true)
	{
		// Update downloaded files count and extract zips
	}
}

As said, this works 100% for POST calls for all users. The problem is for 1% of users with the download callback. I remember having a similar issue with GET when I started this project.

Save me, please!

tomekslesicki

The p part is the authorization that’s getting passed to the server. I was having an issue using that with GET - again, very inconsistent and on some systems only, and I can’t reproduce it on any fo the 3 macs I have around - as though the parameter was not being passed to the server properly. So maybe it’s about that, but then I have no clue on how to troubleshoot this further.

d.healey

@tomekslesicki Nothing in particular stands out to me. Here's my downloader if you want to compare.

tomekslesicki

Is there any way to print the final url with the parameter part? The docs https://docs.hise.dev/scripting/scripting-api/server/index.html#callwithget say that "The parameters have to be a non-nested JSON object and will automatically change the URL to embed these parameters". What does the final url look like after the change?

d.healey

@tomekslesicki said in HEADS UP: Server.downloadFile() issue on macOS Sequoia - and a solution:

The issue was only present on macOS Sequoia,

Have the three customers tested other versions of MacOS?

HEADS UP: Server.downloadFile() issue on macOS Sequoia - and a solution

[feature request] Network connectivity checker

28

1.7k

11.6k

101.4k