Question about queued downloads from Amazon S3

[d.healey]

@Dan-Korneff That first link is a direct link to the resource, doesn't look like it's a link that expires to me so if someone gets hold of it and shares it your wallet would hurt.

How are you getting the link? Are you using the WooCommerce S3 plugin?

[Dan Korneff]

@d-healey Yeah, I'm using the S3 plugin. I'm gonna check out the S3 API since this might just be an amazon issue.

[d.healey]

@Dan-Korneff I'm using a custom wordpress plugin that calls $item->get_item_downloads(); https://woocommerce.github.io/code-reference/classes/WC-Order-Item-Product.html#method_get_item_downloads

[Dan Korneff]

@d-healey Thanks for this! I have half of the issue worked out. I had to set my server to "force download" so it doesn't issue a redirect.
The other half of my issue has to do with extracting the download link from JSON.
If I paste the exact link into Server.downloadFile(), it works perfectly.
If I store the download link in a variable, the link seems to add "" to the string.

for example the response from my server looks something like this:

  {
    "download_url": "https://mywebsite.com/?download_file=productnumber&order=wc_order_ordernmber&email=me@myemail.com&key=some-long-key-1234",
}

I store it in a variable like this:

downloadLink = trace(response.download_url);

if I print downloadLink to the console, it says:

"https://mywebsite.com/?download_file=productnumber&order=wc_order_ordernmber&email=me@myemail.com&key=some-long-key-1234"

So far, so good.
Then I user Server.downloadFile() to start the download.

When I look at the data from the Server Controller, the download link seems to add a " in front of the link and then times out.
So it looks like:

https://"mywebsite.com/

instead of

https://mywebsite.com/

What am I doing wrong?

[d.healey]

@Dan-Korneff said in Question about queued downloads from Amazon S3:

@d-healey Thanks for this! I have half of the issue worked out. I had to set my server to "force download" so it doesn't issue a redirect.

Set it to redirect otherwise all of the bandwidth runs through your server. Ignore the warning WooCommerce gives you about this, it doesn't apply when using S3 links - ask WooCommerce if you want to be certain, I did ;)

What have you set your base URL to? You might need to set it to https:// and then strip that from the download URL (I think that's what I did). You can see my function here https://github.com/davidhealey/HiseDownloadManagerToolkit/blob/master/Downloader.js#L40

[Dan Korneff]

@d-healey said in Question about queued downloads from Amazon S3:

Set it to redirect otherwise all of the bandwidth runs through your server. Ignore the warning WooCommerce gives you about this, it doesn't apply when using S3 links - ask WooCommerce if you want to be certain, I did ;)

If I set like this, downloads won't start no matter what I do.

[d.healey]

@Dan-Korneff Are you using an NGINX server?

[Dan Korneff]

@d-healey Yes

[d.healey]

@Dan-Korneff Did you follow this link and do what it says?

[Dan Korneff]

@d-healey Those settings only seem to apply to files hosted on your server, not S3

[d.healey]

@Dan-Korneff Ah well in that case I'm not sure why it's not working for you. Are you using something like hoppscotch for testing?

[Dan Korneff]

@d-healey I'm completely reckless. Testing on my live site

[d.healey]

@Dan-Korneff Living on the edge :) You can use hoppscotch with your live site though, it is often helpful for debugging problems as you can skip the whole UI stuff in HISE and just get straight to the web requests and responses.

[Dan Korneff]

@d-healey Did you ever find a way to use the JWT token for authentication so you can enable "downloads require login"?

[d.healey]

@Dan-Korneff Unfortunately the JWT doesn't work with the require login thing, but the links are time limited anyway so it's not a huge security problem.

[Dan Korneff]

@d-healey said in Question about queued downloads from Amazon S3:

Set it to redirect otherwise all of the bandwidth runs through your server.

Can you point me to some links about this? The Amazon S3 plugin documentation says:

We do not recommend using the redirect method

I just wanna make sure I'm doing the right thing here.

[d.healey]

@Dan-Korneff Open a support ticket with WooCommerce and ask for clarification. If you don't redirect then (as far as I know) the bandwidth will go through your server which is not good (unless you have an unlimited bandwidth deal). Let me know what they say.

[Dan Korneff]

@d-healey You're right. I haven't heard from them yet, but I can definitely see my bandwidth usage go up when Redirect isn't used.

[Dan Korneff]

In the end, I just needed to run HISE as Administrator for the download to work properly.
Thanks to @d-healey for the help!

[Dan Korneff]

Hey Guy,
I'm getting back to my downloader and wanted to dig back in to downloading files hosted on Amazon S3 without routing through my website. To keep from using server bandwidth, I need to use the redirect method so the file is fetched directly from Amazon.

My website supplies a link that looks like:

"https://MyBucketName.s3.bucketRegion.amazonaws.com/myfile.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=SOMELONGSTRING%2BucketSTufff%2Fs3%2Faws4_request&X-Amz-Date=20220924T040013Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=longstringwithnumbersandletters"

The source code processes the link like this:

var ScriptingApi::Server::downloadFile(String subURL, var parameters, var targetFile, var callback)
{
	if (auto sf = dynamic_cast<ScriptingObjects::ScriptFile*>(targetFile.getObject()))
	{
		if (subURL.contains("?") && parameters.getDynamicObject() != nullptr && parameters.getDynamicObject()->getProperties().isEmpty())
		{
			auto parameterObject = new DynamicObject();
			auto realSubURL = subURL.upToFirstOccurrenceOf("?", false, false);
			auto parameterString = subURL.fromFirstOccurrenceOf("?", false, false);
			auto parameterObjects = StringArray::fromTokens(parameterString, "&", "");

			for (auto po : parameterObjects)
			{
				auto key = po.upToFirstOccurrenceOf("=", false, false);
				auto value = po.fromFirstOccurrenceOf("=", false, false);

				if (!key.isEmpty() && !value.isEmpty())
				{
					parameterObject->setProperty(Identifier(key), var(value));
				}
			}

			parameters = var(parameterObject);
			subURL = realSubURL;
		}

		if (sf->f.isDirectory())
		{
			reportScriptError("target file is a directory");
			return var();
		}

		auto urlToUse = getWithParameters(subURL, parameters);

		if(urlToUse.isWellFormed())
		{
			ScriptingObjects::ScriptDownloadObject::Ptr p = new ScriptingObjects::ScriptDownloadObject(getScriptProcessor(), urlToUse, globalServer.getExtraHeader(), sf->f, callback);
			return globalServer.addDownload(p);
		}
	}
	else
	{
		reportScriptError("target file is not a file object");
	}

	return var();
}

if the URL contains a "?", it stores the info after that as parameterString.

parameterString = X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=SOMELONGSTRING%2BucketSTufff%2Fs3%2Faws4_request&X-Amz-Date=20220924T040013Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=longstringwithnumbersandletters;

it then separates the string by "&" and stores the keys in parameterObjects:

parameterObjects = [  X-Amz-Algorithm=AWS4-HMAC-SHA256,
                      X-Amz-Credential=SOMELONGSTRING%2BucketSTufff%2Fs3%2Faws4_request,
                      X-Amz-Date=20220924T040013Z,
                      X-Amz-Expires=60,
                      X-Amz-SignedHeaders=host,
                      X-Amz-Signature=longstringwithnumbersandletters ]

it then replaces "parameters" passed in the server call with parameterObjects and the sever call is actually paced with the extracted url and parameter object.

urlToUse = getWithParameters(extracted url , extraced parameter Objects);

If I enter the link directly in a browser, or if I separate all of the parameters and enter them in postman, the file downloads. No luck in HISE though. What's my next step to debug this?