Server.downloadFile with signed Amazon S3 URL

[bendurso]

I'm trying to download a signed Amazon S3 URL using Server.downloadFile(), but the request always fails. The same URL downloads correctly in the browser, so it seems an issue on HISE. The link is extremely long because it contains X-Amz-Security-Token, X-Amz-Date, X-Amz-SignedHeaders, X-Amz-Signature, etc.

[David Healey]

@bendurso I'm guessing that link doesn't go directly to a file but redirects and maybe that is causing the issue.

[bendurso]

@d-healey The link is the final one. But your comment led me to think the link was different, so I checked it and realized that the download link in Chrome transforms the " into %22 and the spaces into %20.

I can replace the spaces. But how do I transform the " into %22?
Because this doesnt work

String.replace(""", "%22");

update: ohh, like this:

String.replace("\"", "%22");

Now the link is identical.. but still doesnt work :(

[bendurso]

@bendurso The link contains 1359 characters. Maybe that could be the issue? Is there max limit for the link length?

[David Healey]

@bendurso said in Server.downloadFile with signed Amazon S3 URL:

The link contains 1359 characters. Maybe that could be the issue? Is there max limit for the link length?

Could be, I'm not sure. I believe HISE is using cURL to do the download so you can test that by running it manually in a terminal.

[bendurso]

@d-healey Thanks, yes it works with curl in terminal.

In HISE, I'm clearing the HttpHeader before starting the download, as I read in the forum, and I'm making sure not to repeat the base URL (just using https://).

It works with links from other servers, but it doesn’t seem to work with signed S3 URLs.

[David Healey]

@bendurso Does it work in a private browser session?

[bendurso]

@d-healey yeap :)

[David Healey]

@bendurso I'm stumped then... You could use cURL directly from in HISE by calling it as a process - @aaronventure posted about this a while back I think.

[bendurso]

@David-Healey Thanks. It worked with cURL as a background process. I read that if the user has a Windows version prior to 2019 wont work. Should I use powershell for Windows for better compatibility right?

[David Healey]

@bendurso said in Server.downloadFile with signed Amazon S3 URL:

Thanks. It worked with cURL as a background process.

I'd really like to know why it doesn't work with the standard downloadFile call.

@bendurso said in Server.downloadFile with signed Amazon S3 URL:

Should I use powershell for Windows for better compatibility right?

I don't know, I think powershell needs permissions doesn't it?

[bendurso]

@David-Healey I found a post where Dan was using the same amazon s3 links as me.
https://forum.hise.audio/topic/4635/question-about-queued-downloads-from-amazon-s3/26

I'm actually trying to use moonbase, but I got the same links for download.

@Dan-Korneff How did you solve this?

[Dan Korneff]

@bendurso I can't quite remember. Reading the post it seems to be related to how my webserver was redirecting the traffic. I'll take a look in the AM to see where I left off

[bendurso]

@David-Healey @Dan-Korneff Great news! I was able to fix the issue with Amazon S3 pre-signed URLs and the HISE Server.downloadFile function.

Thanks to Dan's explanation of the source code (on the other post), the problem was identified:

HISE original logic sees the ? in the S3 URL and assumes it needs to separate and re-parse the query string into a separate parameter object. This process destroys the specific encoding required for the X-Amz-Signature, invalidating the S3 link.

So, I modified the ScriptingApi::Server::downloadFile function in the HISE source code to skip the parameter parsing if the URL contains the S3-specific parameter "X-Amz-Algorithm=".

I think this should be changed in the HISE source code because AWS S3 are pretty common.

[David Healey]

@bendurso said in Server.downloadFile with signed Amazon S3 URL:

I think this should be changed in the HISE source code because AWS S3 are pretty common.

Is parameter splitting ever needed with a file download? Shouldn't we leave that up to the scripter to pass parameters via the parameter argument of the downloadFile function?

We could add a convenience function that converts a URL query string to a parameters object.

[bendurso]

@David-Healey Haha that make sense too :)