https://s3.eu-central-1.amazonaws.com/data/avatars/s/6/6816.jpg?1613728101
<Error>
<Code>PermanentRedirect</Code>
<Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
<Endpoint>data.s3-ap-northeast-1.amazonaws.com</Endpoint>
<Bucket>data</Bucket>
<RequestId>3233044C22A99172</RequestId>
<HostId>BiZe3oWFWudDZmK/hHa7dL9My0BpkJm2/uGlS20TXX5UboQDx8QHswhvK4JazBYyJnkakxRq78E=</HostId>
</Error>
$config['externalDataUrl'] = function($externalPath, $canonical)
{
return 'https://xftest.s3.eu-west-2.amazonaws.com/data/' . $externalPath;
};
$config['externalDataUrl'] = function($externalPath, $canonical)
{
return 'https://s3.eu-central-1.amazonaws.com/data/' . $externalPath;
};
$config['externalDataUrl'] = function($externalPath, $canonical)
{
return 'https://data.s3-ap-northeast-1.amazonaws.com/data/' . $externalPath;
};
That doesn't have to be the case. You can create a pre-signed URL from PHP and redirect to said URL. It gives you a 36-hour valid URL to a private object. I would much prefer if this was possible in XenForo / adapters. One would need to extend XF\Pub\View\Attachment\View to cause a redirect instead of streaming the file.Note that the attachment URL for your attachments will remain the same - it will still be you’re URL. But we stream this from the remote location.
If we didn’t do this then we wouldn’t be able to still handle permissions.
Chris, is this something that has been looked into further? If the attachments from /internal_data/ are streamed through the hosting server, wouldn't this then cause double the bandwidth usage?There is nointernalDataPath
and due to various approaches - mainly permission checks - we have to serve the attachments from a URL where we can do those checks.
The offloading issue is indeed an issue that we're conscious of but we've not take steps to address that yet.
internal_data/attachments
folder that internal_data/filecheck
and internal_data/sitemaps
are also the ones being uploaded to the s3 server? Just to be safe I uploaded the whole internal_data folder to the s3 storage server and I see that those 3 sub-folders get regularly updated by new files. And there is nothing mentioned about that so just to be safe, wanted to ask it here.The download case is something we're aware of and ideally we'd like to support in the future, though it would be optional as it would effectively make potentially private attachments accessible to anyone who happens to have the URL and no longer directly protected by node / conversation permissions.
Yeah pretty much.So offloading the images to S3 buckets would double this if the images are streamed through the hosting server.
The abstracted file system in XF works on the basis of "mounts". We actually have three.I also have one question. I am running this for 2 months now.
Is it normal that besidesinternal_data/attachments
folder thatinternal_data/filecheck
andinternal_data/sitemaps
are also the ones being uploaded to the s3 server? Just to be safe I uploaded the whole internal_data folder to the s3 storage server and I see that those 3 sub-folders get regularly updated by new files. And there is nothing mentioned about that so just to be safe, wanted to ask it here.
data
, internal-data
and code-cache
. This resource goes through the steps of changing the data
and internal-data
mounts to use the S3 adapter rather than the local file system adapter.code-cache
is its own mount, we still write out files to the local directory internal_data/code_cache
. We have to do this because this contains things like compiled templates which need to be executed directly on the server. You don't need to maintain a copy of this remotely if you copied it over.internal_data/temp
. There isn't a separate mount for this, and it cannot be changed, it's just always written to as internal_data/temp
. Again, you don't need to maintain a copy of this remotely.Yeah I think roughly that might have been the approach we have discussed in the past.I think there's a simple per-node setting here that could simply allow admins to choose their privacy concerns.
eg, I probably only care about my admin forum being permission checked and streamed through the server. all the rest can be served direct through the cdn (eg, cloudfront on top of s3), regardless of banned permissions, ability to see the forum the attachment is in, etc.
something like
if(nodeAttachmentPermsOpen() { url=cdn.domain.com/attachment ...' }
else { url = domain.com/attachment... ; ]
let the url dictate needing a permission check.
$s3 = function()
{
return new \Aws\S3\S3Client([
'credentials' => [
'key' => 'xxx',
'secret' => 'xxx'
],
'region' => 'sfo3',
'version' => 'latest',
'endpoint' => 'https://sfo3.digitaloceanspaces.com'
]);
};
$config['fsAdapters']['data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'name', 'data');
};
$config['externalDataUrl'] = function($externalPath, $canonical)
{
return 'https://name.sfo3.digitaloceanspaces.com/data/' . $externalPath;
};
$config['fsAdapters']['internal-data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'name', 'internal_data');
};
I'm having the exact same result via S3. I've read through this thread a few times and I see where others have reported the same but didn't see a resolution note. Can someone confirm that full-size attachments should show the S3 or DO URL instead of an internal URL?@Chris D
i am using XF 2.2 and try to upload my files on DO,and i installed the plugin (v.2.1), put the following code into my config.php
Code:$s3 = function() { return new \Aws\S3\S3Client([ 'credentials' => [ 'key' => 'xxx', 'secret' => 'xxx' ], 'region' => 'sfo3', 'version' => 'latest', 'endpoint' => 'https://sfo3.digitaloceanspaces.com' ]); }; $config['fsAdapters']['data'] = function() use($s3) { return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'name', 'data'); }; $config['externalDataUrl'] = function($externalPath, $canonical) { return 'https://name.sfo3.digitaloceanspaces.com/data/' . $externalPath; }; $config['fsAdapters']['internal-data'] = function() use($s3) { return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'name', 'internal_data'); };
looks like it work , but i noticed that only thumb,avatar and profile banner image shows DO URL, the full size image still shows internal URL and i can see them in the XF admin panel, i logged into my ftp accounts, they still in my local server (data folder), i don't know what's going on here
We use essential cookies to make this site work, and optional cookies to enhance your experience.