Server.downloadFile with signed Amazon S3 URL

[David Healey]

@bendurso I'm guessing that link doesn't go directly to a file but redirects and maybe that is causing the issue.

[bendurso]

@d-healey The link is the final one. But your comment led me to think the link was different, so I checked it and realized that the download link in Chrome transforms the " into %22 and the spaces into %20.

I can replace the spaces. But how do I transform the " into %22?
Because this doesnt work

String.replace(""", "%22");

update: ohh, like this:

String.replace("\"", "%22");

Now the link is identical.. but still doesnt work :(

[bendurso]

@bendurso The link contains 1359 characters. Maybe that could be the issue? Is there max limit for the link length?

[David Healey]

@bendurso said in Server.downloadFile with signed Amazon S3 URL:

The link contains 1359 characters. Maybe that could be the issue? Is there max limit for the link length?

Could be, I'm not sure. I believe HISE is using cURL to do the download so you can test that by running it manually in a terminal.

[bendurso]

@d-healey Thanks, yes it works with curl in terminal.

In HISE, I'm clearing the HttpHeader before starting the download, as I read in the forum, and I'm making sure not to repeat the base URL (just using https://).

It works with links from other servers, but it doesn’t seem to work with signed S3 URLs.

[David Healey]

@bendurso Does it work in a private browser session?

[bendurso]

@d-healey yeap :)

[David Healey]

@bendurso I'm stumped then... You could use cURL directly from in HISE by calling it as a process - @aaronventure posted about this a while back I think.

[bendurso]

@David-Healey Thanks. It worked with cURL as a background process. I read that if the user has a Windows version prior to 2019 wont work. Should I use powershell for Windows for better compatibility right?

[David Healey]

@bendurso said in Server.downloadFile with signed Amazon S3 URL:

Thanks. It worked with cURL as a background process.

I'd really like to know why it doesn't work with the standard downloadFile call.

@bendurso said in Server.downloadFile with signed Amazon S3 URL:

Should I use powershell for Windows for better compatibility right?

I don't know, I think powershell needs permissions doesn't it?

[bendurso]

@David-Healey I found a post where Dan was using the same amazon s3 links as me.
https://forum.hise.audio/topic/4635/question-about-queued-downloads-from-amazon-s3/26

I'm actually trying to use moonbase, but I got the same links for download.

@Dan-Korneff How did you solve this?

[Dan Korneff]

@bendurso I can't quite remember. Reading the post it seems to be related to how my webserver was redirecting the traffic. I'll take a look in the AM to see where I left off

[bendurso]

@David-Healey @Dan-Korneff Great news! I was able to fix the issue with Amazon S3 pre-signed URLs and the HISE Server.downloadFile function.

Thanks to Dan's explanation of the source code (on the other post), the problem was identified:

HISE original logic sees the ? in the S3 URL and assumes it needs to separate and re-parse the query string into a separate parameter object. This process destroys the specific encoding required for the X-Amz-Signature, invalidating the S3 link.

So, I modified the ScriptingApi::Server::downloadFile function in the HISE source code to skip the parameter parsing if the URL contains the S3-specific parameter "X-Amz-Algorithm=".

I think this should be changed in the HISE source code because AWS S3 are pretty common.

[David Healey]

@bendurso said in Server.downloadFile with signed Amazon S3 URL:

I think this should be changed in the HISE source code because AWS S3 are pretty common.

Is parameter splitting ever needed with a file download? Shouldn't we leave that up to the scripter to pass parameters via the parameter argument of the downloadFile function?

We could add a convenience function that converts a URL query string to a parameters object.

[bendurso]

@David-Healey Haha that make sense too :)