Question about queued downloads from Amazon S3

[Dan Korneff]

Hey Guy,
I'm getting back to my downloader and wanted to dig back in to downloading files hosted on Amazon S3 without routing through my website. To keep from using server bandwidth, I need to use the redirect method so the file is fetched directly from Amazon.

My website supplies a link that looks like:

"https://MyBucketName.s3.bucketRegion.amazonaws.com/myfile.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=SOMELONGSTRING%2BucketSTufff%2Fs3%2Faws4_request&X-Amz-Date=20220924T040013Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=longstringwithnumbersandletters"

The source code processes the link like this:

var ScriptingApi::Server::downloadFile(String subURL, var parameters, var targetFile, var callback)
{
	if (auto sf = dynamic_cast<ScriptingObjects::ScriptFile*>(targetFile.getObject()))
	{
		if (subURL.contains("?") && parameters.getDynamicObject() != nullptr && parameters.getDynamicObject()->getProperties().isEmpty())
		{
			auto parameterObject = new DynamicObject();
			auto realSubURL = subURL.upToFirstOccurrenceOf("?", false, false);
			auto parameterString = subURL.fromFirstOccurrenceOf("?", false, false);
			auto parameterObjects = StringArray::fromTokens(parameterString, "&", "");

			for (auto po : parameterObjects)
			{
				auto key = po.upToFirstOccurrenceOf("=", false, false);
				auto value = po.fromFirstOccurrenceOf("=", false, false);

				if (!key.isEmpty() && !value.isEmpty())
				{
					parameterObject->setProperty(Identifier(key), var(value));
				}
			}

			parameters = var(parameterObject);
			subURL = realSubURL;
		}

		if (sf->f.isDirectory())
		{
			reportScriptError("target file is a directory");
			return var();
		}

		auto urlToUse = getWithParameters(subURL, parameters);

		if(urlToUse.isWellFormed())
		{
			ScriptingObjects::ScriptDownloadObject::Ptr p = new ScriptingObjects::ScriptDownloadObject(getScriptProcessor(), urlToUse, globalServer.getExtraHeader(), sf->f, callback);
			return globalServer.addDownload(p);
		}
	}
	else
	{
		reportScriptError("target file is not a file object");
	}

	return var();
}

if the URL contains a "?", it stores the info after that as parameterString.

parameterString = X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=SOMELONGSTRING%2BucketSTufff%2Fs3%2Faws4_request&X-Amz-Date=20220924T040013Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=longstringwithnumbersandletters;

it then separates the string by "&" and stores the keys in parameterObjects:

parameterObjects = [  X-Amz-Algorithm=AWS4-HMAC-SHA256,
                      X-Amz-Credential=SOMELONGSTRING%2BucketSTufff%2Fs3%2Faws4_request,
                      X-Amz-Date=20220924T040013Z,
                      X-Amz-Expires=60,
                      X-Amz-SignedHeaders=host,
                      X-Amz-Signature=longstringwithnumbersandletters ]

it then replaces "parameters" passed in the server call with parameterObjects and the sever call is actually paced with the extracted url and parameter object.

urlToUse = getWithParameters(extracted url , extraced parameter Objects);

If I enter the link directly in a browser, or if I separate all of the parameters and enter them in postman, the file downloads. No luck in HISE though. What's my next step to debug this?

[d.healey]

@Dan-Korneff I just use the URL WooCommerce provides with the Server.downloadFile() function. https://codeberg.org/LibreWave/Rhapsody/src/branch/main/Scripts/Downloader.js

[Dan Korneff]

@d-healey that does work on my end too, but the link provided by woocommerce funnels the transfer thru your site, which uses bandwidth.
Your download links look like this, right?
:
https://mywebsite.com/?download_file=productnumber&order=wc_order_ordernmber&email=me@myemail.com&key=some-long-key-1234

[d.healey]

@Dan-Korneff Yes, but I use the redirect method so my site redirects it to S3

Edit1: Actually, give me a moment and I'll check what the link looks like...

Edit 2: Yep that is exactly the same format.

[Dan Korneff]

@d-healey said in Question about queued downloads from Amazon S3:

Edit 2: Yep that is exactly the same format.

In that case, the file is stored on S3, but funnels through your site to deliver the file. Using bandwidth.

[d.healey]

@Dan-Korneff Oh I didn't realise!

How did you get this URL?

"https://MyBucketName.s3.bucketRegion.amazonaws.com/myfile.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=SOMELONGSTRING%2BucketSTufff%2Fs3%2Faws4_request&X-Amz-Date=20220924T040013Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=longstringwithnumbersandletters"

Isn't it risky to use a direct URL like that?

[DanH]

@Dan-Korneff

In that case, the file is stored on S3, but funnels through your site to deliver the file. Using bandwidth.

is there a way around this in Woocomerce?

[d.healey]

@DanH Not directly it seems, that's what Dan's trying to solve (and so am I now!).

[Dan Korneff]

@d-healey I'm still new at this, but this is a Pre-Signed URL I'm generating with the S3 SDK which contains an expiration associated with the signature.

[d.healey]

@Dan-Korneff Have you asked WooCommerce about a solution for this?

Also what about this - https://wordpress.org/support/topic/bandwidth-for-downloads/

[Dan Korneff]

@d-healey gonna check this out now

[Dan Korneff]

@d-healey Here's how it works with woocommcerce.
https://www.download-monitor.com/kb/how-to-use-download-monitor-with-woocommerce/
I'm gonna give it a try, but once you change your download method to "Redirect Only" as suggested, it's going to supply an amazon link which HISE can't download, and I'm back to square one.

[d.healey]

@Dan-Korneff

I'm gonna give it a try, but once you change your download method to "Redirect Only" as suggested, it's going to supply an amazon link which HISE can't download, and I'm back to square one

What you talking bout Willis? I've been using Redirect Only with Rhapsody.

[Dan Korneff]

@d-healey like this?


[d.healey]

@Dan-Korneff Scroll up - https://forum.hise.audio/topic/4635/question-about-queued-downloads-from-amazon-s3/12?_=1698944591828

[Dan Korneff]

@d-healey Redirect Only or X-Accel?

[d.healey]

@Dan-Korneff Redirect Only

[Dan Korneff]

@d-healey Are you using WooCommerce Amazon S3 Storage plugin to load your files?

https://woo.com/products/amazon-s3-storage/

[d.healey]

@Dan-Korneff Yup

[Dan Korneff]

@d-healey what about the redirect method for the S3 plugin?