Question about queued downloads from Amazon S3

[d.healey]

@Dan-Korneff said in Question about queued downloads from Amazon S3:

@d-healey Thanks for this! I have half of the issue worked out. I had to set my server to "force download" so it doesn't issue a redirect.

Set it to redirect otherwise all of the bandwidth runs through your server. Ignore the warning WooCommerce gives you about this, it doesn't apply when using S3 links - ask WooCommerce if you want to be certain, I did ;)

What have you set your base URL to? You might need to set it to https:// and then strip that from the download URL (I think that's what I did). You can see my function here https://github.com/davidhealey/HiseDownloadManagerToolkit/blob/master/Downloader.js#L40

[Dan Korneff]

@d-healey said in Question about queued downloads from Amazon S3:

Set it to redirect otherwise all of the bandwidth runs through your server. Ignore the warning WooCommerce gives you about this, it doesn't apply when using S3 links - ask WooCommerce if you want to be certain, I did ;)

If I set like this, downloads won't start no matter what I do.

[d.healey]

@Dan-Korneff Are you using an NGINX server?

[Dan Korneff]

@d-healey Yes

[d.healey]

@Dan-Korneff Did you follow this link and do what it says?

[Dan Korneff]

@d-healey Those settings only seem to apply to files hosted on your server, not S3

[d.healey]

@Dan-Korneff Ah well in that case I'm not sure why it's not working for you. Are you using something like hoppscotch for testing?

[Dan Korneff]

@d-healey I'm completely reckless. Testing on my live site

[d.healey]

@Dan-Korneff Living on the edge :) You can use hoppscotch with your live site though, it is often helpful for debugging problems as you can skip the whole UI stuff in HISE and just get straight to the web requests and responses.

[Dan Korneff]

@d-healey Did you ever find a way to use the JWT token for authentication so you can enable "downloads require login"?

[d.healey]

@Dan-Korneff Unfortunately the JWT doesn't work with the require login thing, but the links are time limited anyway so it's not a huge security problem.

[Dan Korneff]

@d-healey said in Question about queued downloads from Amazon S3:

Set it to redirect otherwise all of the bandwidth runs through your server.

Can you point me to some links about this? The Amazon S3 plugin documentation says:

We do not recommend using the redirect method

I just wanna make sure I'm doing the right thing here.

[d.healey]

@Dan-Korneff Open a support ticket with WooCommerce and ask for clarification. If you don't redirect then (as far as I know) the bandwidth will go through your server which is not good (unless you have an unlimited bandwidth deal). Let me know what they say.

[Dan Korneff]

@d-healey You're right. I haven't heard from them yet, but I can definitely see my bandwidth usage go up when Redirect isn't used.

[Dan Korneff]

In the end, I just needed to run HISE as Administrator for the download to work properly.
Thanks to @d-healey for the help!

[Dan Korneff]

Hey Guy,
I'm getting back to my downloader and wanted to dig back in to downloading files hosted on Amazon S3 without routing through my website. To keep from using server bandwidth, I need to use the redirect method so the file is fetched directly from Amazon.

My website supplies a link that looks like:

"https://MyBucketName.s3.bucketRegion.amazonaws.com/myfile.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=SOMELONGSTRING%2BucketSTufff%2Fs3%2Faws4_request&X-Amz-Date=20220924T040013Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=longstringwithnumbersandletters"

The source code processes the link like this:

var ScriptingApi::Server::downloadFile(String subURL, var parameters, var targetFile, var callback)
{
	if (auto sf = dynamic_cast<ScriptingObjects::ScriptFile*>(targetFile.getObject()))
	{
		if (subURL.contains("?") && parameters.getDynamicObject() != nullptr && parameters.getDynamicObject()->getProperties().isEmpty())
		{
			auto parameterObject = new DynamicObject();
			auto realSubURL = subURL.upToFirstOccurrenceOf("?", false, false);
			auto parameterString = subURL.fromFirstOccurrenceOf("?", false, false);
			auto parameterObjects = StringArray::fromTokens(parameterString, "&", "");

			for (auto po : parameterObjects)
			{
				auto key = po.upToFirstOccurrenceOf("=", false, false);
				auto value = po.fromFirstOccurrenceOf("=", false, false);

				if (!key.isEmpty() && !value.isEmpty())
				{
					parameterObject->setProperty(Identifier(key), var(value));
				}
			}

			parameters = var(parameterObject);
			subURL = realSubURL;
		}

		if (sf->f.isDirectory())
		{
			reportScriptError("target file is a directory");
			return var();
		}

		auto urlToUse = getWithParameters(subURL, parameters);

		if(urlToUse.isWellFormed())
		{
			ScriptingObjects::ScriptDownloadObject::Ptr p = new ScriptingObjects::ScriptDownloadObject(getScriptProcessor(), urlToUse, globalServer.getExtraHeader(), sf->f, callback);
			return globalServer.addDownload(p);
		}
	}
	else
	{
		reportScriptError("target file is not a file object");
	}

	return var();
}

if the URL contains a "?", it stores the info after that as parameterString.

parameterString = X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=SOMELONGSTRING%2BucketSTufff%2Fs3%2Faws4_request&X-Amz-Date=20220924T040013Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=longstringwithnumbersandletters;

it then separates the string by "&" and stores the keys in parameterObjects:

parameterObjects = [  X-Amz-Algorithm=AWS4-HMAC-SHA256,
                      X-Amz-Credential=SOMELONGSTRING%2BucketSTufff%2Fs3%2Faws4_request,
                      X-Amz-Date=20220924T040013Z,
                      X-Amz-Expires=60,
                      X-Amz-SignedHeaders=host,
                      X-Amz-Signature=longstringwithnumbersandletters ]

it then replaces "parameters" passed in the server call with parameterObjects and the sever call is actually paced with the extracted url and parameter object.

urlToUse = getWithParameters(extracted url , extraced parameter Objects);

If I enter the link directly in a browser, or if I separate all of the parameters and enter them in postman, the file downloads. No luck in HISE though. What's my next step to debug this?

[d.healey]

@Dan-Korneff I just use the URL WooCommerce provides with the Server.downloadFile() function. https://codeberg.org/LibreWave/Rhapsody/src/branch/main/Scripts/Downloader.js

[Dan Korneff]

@d-healey that does work on my end too, but the link provided by woocommerce funnels the transfer thru your site, which uses bandwidth.
Your download links look like this, right?
:
https://mywebsite.com/?download_file=productnumber&order=wc_order_ordernmber&email=me@myemail.com&key=some-long-key-1234

[d.healey]

@Dan-Korneff Yes, but I use the redirect method so my site redirects it to S3

Edit1: Actually, give me a moment and I'll check what the link looks like...

Edit 2: Yep that is exactly the same format.

[Dan Korneff]

@d-healey said in Question about queued downloads from Amazon S3:

Edit 2: Yep that is exactly the same format.

In that case, the file is stored on S3, but funnels through your site to deliver the file. Using bandwidth.