Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

No permission to download
Overview Updates (4) Reviews (7) Discussion
Guys!

How can I relocate my avatars? I use
[Xen-Soluce] Avatar Gallery 2.1.4 - and I would replace my avatars (im on a shared server so cant use s3cmd ... so I would replace them manually but I dont know how can the add-on fint it's way.
 
Do I need to edit the "internal_data" line or leave it alone?
If I have to edit it where can I get the line for it or location?
 
I recommend it. I made 2 folders in 1 bucket on s3, one for public (data) and one for private (pdata)

Code: 
//public data
$config['fsAdapters']['data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'mybucket', 'data');
};

//private data
$config['fsAdapters']['internal-data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'mybucket', 'pdata');
};
 
briansol said:
I recommend it. I made 2 folders in 1 bucket on s3, one for public (data) and one for private (pdata)

Code: 
//public data
$config['fsAdapters']['data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'mybucket', 'data');
};

//private data
$config['fsAdapters']['internal-data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'mybucket', 'pdata');
};
Click to expand...
Thanks.

So what I need to replace is
"internal_data" to my DO's url name so if my url is https://EXAMPLE.fra1.digitaloceanspaces.com /
then it will be "example"

I also need to edit the "return" part?

Unfortunatelly Im not an IT guy, I can only replace parts if its in a tutoral or the description specify it....
 
briansol said:
I recommend it. I made 2 folders in 1 bucket on s3, one for public (data) and one for private (pdata)

Code: 
//public data
$config['fsAdapters']['data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'mybucket', 'data');
};

//private data
$config['fsAdapters']['internal-data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'mybucket', 'pdata');
};
Click to expand...
Also what's the difference between public and private data?
All the data is accessable via the Forum, no? So if a forum is private the content will be as well even if its "public", no?
So what does private data do?
 
The data (public) directory is for avatars.

The internal_data (private) directory is for attachments (which are permission controlled).
 
NandorHUN said:
Thanks.

So what I need to replace is
"internal_data" to my DO's url name so if my url is https://EXAMPLE.fra1.digitaloceanspaces.com /
then it will be "example"

I also need to edit the "return" part?

Unfortunatelly Im not an IT guy, I can only replace parts if its in a tutoral or the description specify it....
Click to expand...
Sorry, i never used DO so i can't help there. I assume it would work that way if EXAMPLE is your bucket's name.

can you set

https://EXAMPLE.fra1.digitaloceanspaces.com/data
and
https://EXAMPLE.fra1.digitaloceanspaces.com/pdata

folders in the EXAMPLE space?


Try it quickly - upload an avatar and see that it's in the DO space. if it's not, revert your config file.

Remember, the old stuff won't move automatically - you'll have to ftp or use an s3cmd type command line tool to do the asset migration. Setting the paths only updates sourcing locations and go-forward store location. So, you need to get the local stuff to the droplet.
 
i just checked my old config file from when i was using do spaces. i have the same format:

return 'https://bucketname.sgp1.cdn.digitaloceanspaces.com/data/' . $externalPath;

this is for externalDataUrl.

for internal-data, it shows:

return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'bucketname', 'internal_data');
 
briansol said:
Sorry, i never used DO so i can't help there. I assume it would work that way if EXAMPLE is your bucket's name.

can you set

https://EXAMPLE.fra1.digitaloceanspaces.com/data
and
https://EXAMPLE.fra1.digitaloceanspaces.com/pdata

folders in the EXAMPLE space?


Try it quickly - upload an avatar and see that it's in the DO space. if it's not, revert your config file.

Remember, the old stuff won't move automatically - you'll have to ftp or use an s3cmd type command line tool to do the asset migration. Setting the paths only updates sourcing locations and go-forward store location. So, you need to get the local stuff to the droplet.
Click to expand...
I did this tutoral:

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x

And after that when I upload avatar they did appear on DO space... I did everything exacly like in the tutoral, works.


Chromaniac said:
i just checked my old config file from when i was using do spaces. i have the same format:

return 'https://bucketname.sgp1.cdn.digitaloceanspaces.com/data/' . $externalPath;

this is for externalDataUrl.

for internal-data, it shows:

return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'bucketname', 'internal_data');
Click to expand...

Well, I still have some questions, basically about the two tutorals, they dont cover each other. Let me explain:

In the S3 tutoral it is said:

ext we need to set up the actual Flysystem adapter to use the S3 client:
PHP:
$config['fsAdapters']['data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xftest', 'data');
};

Finally, we need to ensure that avatar and attachment thumbnail URLs are prepended with the correct URL. This requires the endpoint URL you noted down earlier, again:
PHP:
$config['externalDataUrl'] = function($externalPath, $canonical)
{
return 'https://xftest.ams3.digitaloceanspaces.com/data/' . $externalPath;
};

And in the Attachment Improved tutoral it is said:

"Additional configuration is required to enable X-Accel-Redirect when using DigitalOcean Spaces or Amazon S3

A internalDataUrl stanza l must be added to config.php like the externalDataUrl described in the tutorial

PHP:
$config['internalDataUrl'] = function($externalPath, $canonical)
{
return 'internal_data_s3/..../internal_data/' . $externalPath;
};
nginx config must be updated with a new internal_data_s3 block;
PHP:
location ~* /internal_data_s3/(.?)://(.?)/(.*) {
internal;
set $xfEtag $upstream_http_etag;
set $download_protocol $1;
set $download_host $2;
set $download_path $3;
set $download_url $download_protocol://$download_host/$download_path;

resolver 127.0.0.1 ipv6=off;
proxy_set_header Host $download_host;
proxy_set_header Authorization '';
proxy_set_header Cookie '';
proxy_max_temp_file_size 0;
proxy_intercept_errors on;
error_page 301 302 307 = @handle_redirect;

proxy_pass $download_url$is_args$args;

proxy_hide_header Content-Disposition;
proxy_hide_header Content-Type;
proxy_hide_header Etag;
proxy_hide_header x-amz-request-id;

add_header Etag $xfEtag;
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
}

location @handle_redirect {
resolver 127.0.0.1 ipv6=off;
set $saved_redirect_location '$upstream_http_location';
proxy_pass $saved_redirect_location;
}


So the main difference:

In S3 it is said:


PHP:
$config['fsAdapters']['data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xftest', 'data');
};

PHP:
$config['externalDataUrl'] = function($externalPath, $canonical)
{
return 'https://xftest.ams3.digitaloceanspaces.com/data/' . $externalPath;
};

In case for Attachments Imrpved add on:

PHP:
$config['internalDataUrl'] = function($externalPath, $canonical)
{
return 'internal_data_s3/..../internal_data/' . $externalPath;
};


So now....
  • is it internal or external?
  • or both? or neither? or I have to add a new one as internal?
  • what is this "nginx config" ? Do I have to add that part at the end of the config file or is there a nginx config file?

 
I also noticed something strange about even external data:
I can upload files, works great, but through Xenforo I cant delete files. So even if I delet an attachment in Xenforo (as the user or as admin in ACP/ Attackments, the files still remains in DO.
Is that a bug?
 
Chromaniac said:
Files are deleted through a cron job so periodically and not immediately iirc.
Click to expand...
Thanks.

I copy here my example "myexamplesite" S3 part of the config file.... could you guys have a look, is it good or do I have to edit anything else? I marked with red the parts that I edited and with green the parts that I dont know if I have to edit or leave it like this.


$config['fsAdapters']['data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'myexamplesite', 'data');
};

$config['externalDataUrl'] = function($externalPath, $canonical)
{
return 'https://myexamplesite.fra1.digitaloceanspaces.com/data/' . $externalPath;
};

$config['fsAdapters']['internal-data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'myexamplesite', 'internal_data');
};

$config['internalDataUrl'] = function($externalPath, $canonical)
{
return ' ' . $externalPath;
};


Is this OK?
Thanks for the help!
 
that looks correct.

I think some of the confusion is that you are seeing 2.0 instructions/older version mixed in the thread. It was streamlined quite a bit in the latest version.
 
briansol said:
that looks correct.

I think some of the confusion is that you are seeing 2.0 instructions/older version mixed in the thread. It was streamlined quite a bit in the latest version.
Click to expand...
Which part of it is older?
My XF is 2.2.

Also I still dont knowwhere is this "ngix config file is...
 
if you're on apache, you won't have nginx config.... that's a competing webserver that some people use instead of apache. It doesn't support htaccess so those rules need to be added to it's config file.

This thread started before xf2.2 was out, and even the first version of this mod... so, the thread has a lot of old/outdated info in the first several pages before newer releases came out so don't confuse the old instructions with modern ones
 
briansol said:
if you're on apache, you won't have nginx config.... that's a competing webserver that some people use instead of apache. It doesn't support htaccess so those rules need to be added to it's config file.

This thread started before xf2.2 was out, and even the first version of this mod... so, the thread has a lot of old/outdated info in the first several pages before newer releases came out so don't confuse the old instructions with modern ones
Click to expand...

So do I have to add this part simply to the config file?

Im asking because at the moment I dont have eny errors...
 
You must log in or register to reply here.
Top Bottom