MetaMirror 1.6.1

[Mick West]

That would be wonderful and maybe simpler compared with my fringe suggestion

I had a think about this, it's essentially the same as disabling instant rehosting, and having the cron job running. So I'd suggest you see if that fixes the problem.

 

[Mick West]

MetaMirror and XF 1.3 Proxy System

MetaMirror appears to work without any problems in XF 1.3.0 b1, both with and without Image Proxying turned on. There is no reason to use both at the same time though.

You could use the Image Proxy system to replace all the functionality of MetaMirror by caching images indefinitely, and I would suggest that is probably the best for most people. However there are still advantages to MetaMirror, in particular that it creates direct links for images, which is faster, and may make external CDN caching simpler.

Take an external image like:
http://classroomclipart.com/images/gallery/Anatomy/TN_anatomy_heart.jpg

With MM, this is rehosted as a local image, like:
https://www.metabunk.org/data/MetaMirrorCache/643596448e2bfee23035c1f5b14c8864.jpg

Or as an attachment, which translates to an image URL of:
https://www.metabunk.org/attachments/1a978dfb5b9d8dc578338a2219b08b7e-jpg.5842/

With XF 1.3 Proxy, the image URL becomes:

http://local.metabunk.org/proxy.php?image=http%3A%2F%2Fclassroomclipart.com%2Fimages%2Fgallery%2FAnatomy%2FTN_anatomy_heart.jpg&hash=25fe48ec245c0de2180142c637102d69

(link on my local setup, I've not deployed 1.3 yet)

So this is executing proxy.php for every image, essentially a dynamic image generator. This is going to be quite a bit slower than directly serving the file. It also might not be cached by CDNs like CloudFront, as it's dynamic (although I'm not sure about that).

The Proxy system is much safer to use for the average user, as it's reversible (just turn it off, and your image links are back the way they were). However I think MM will be the better choice if performance is an issue, and you don't mind the direct and irreversible re-writing of the image URLs.

 

[arms]

Thanks for taking the time to explain @Mick West .

 

[yavuz]

Thanks for explaining @Mick West . I wonder if there would be any conflict if I turned MM of for certain forums (Exclude Forums) and turned on the XenForo Proxy feature.

I have a small request, for your consideration; could you maybe make the "Exclude Forums" option a selectable forum list. Users could easily select them from the list. Currently If one had lots of forums to exlude it would be hard to track them from their ID.

 

[Mick West]

Thanks for explaining @Mick West . I wonder if there would be any conflict if I turned MM of for certain forums (Exclude Forums) and turned on the XenForo Proxy feature.

I have a small request, for your consideration; could you maybe make the "Exclude Forums" option a selectable forum list. Users could easily select them from the list. Currently If one had lots of forums to exlude it would be hard to track them from their ID.

I'm afraid you'd have to try it and see. It will certainly work, however It's not clear if the proxy system will also attempt to proxy local files.

I'm not really planning any updates to MM, but if I do I'll look at the forum exclusion. Can't promise anything though.

 

[cmeinck]

I'm planning to move my forum from http://www.mysite.com/forum/ to http://forums.mysite.com.

I use a CDN, which is configured to http://mycdn.com/forum/data and http://mycdn.com/forum/js

I have ignore strongs for http://mycdn.com and http://mysite.com

I guess my question is, what changes need to be made to either my CDN and/or MetaMirror to ensure all the images/data move over smoothly to the subdomain. I plan on moving all my files from /forum/ to the new suddomain folder and changing the base URL.

Apologies if there are some questions outside of MetaMirror, but with the crossover, I thought it best to include a full description of my plan.

Thanks.

 

[Mick West]

Metamirror alters the actual posts to change the URL, so if you want the older mirrored files to point to the external CDN, then you'll want to search and replace if it's a plain CDN.

MM will uses the config variable;
$config['externalDataUrl'] = 'http://example.com/data';
via
XenForo_Application::$externalDataUrl

To let you set a url path to the data directory.

and uses the function:
$root = XenForo_Helper_File::getExternalDataPath();
To get the file path to the data directory.

I'm assuming your CDN is pulling files from /data.

Personally I'm using Cloudflare, which requires no changing of paths etc. I had a thread with lots of images get 300,000 views last week, and Cloudflare handled it just fine, basically caching all the image, JS, and CSS (with [bd] cache) requests (well, 99.9% of them, as it does occasional refreshes). I'd recommend that over a traditional CDN as it's super easy to configure.

 

[Mick West]

Here's my cloudflare stats for the last 30 days

 

[cmeinck]

When I switched to my subdomain, all of my reshosted images on the CDN broke. I've updated my config.php and origin URL, but no luck.

My understanding was that if I disabled my CDN (config.php), the images would revert to the local data folder?

What's the fastest and easiest way to fix all of the broken image URLs?

 

[Mick West]

When I switched to my subdomain, all of my reshosted images on the CDN broke. I've updated my config.php and origin URL, but no luck.

My understanding was that if I disabled my CDN (config.php), the images would revert to the local data folder?

What's the fastest and easiest way to fix all of the broken image URLs?

MetaMirror edits the url in the post when it rehosts. So if you change where the images are you need to update the urls in the posts.

Probably the best way to do this is to use the Post Content Find / Replace

http://xenforo.com/community/resources/post-content-find-replace.1549/

 

[Mick West]

The replace tool is fairly straightforward, you simply need to surround the string with / and escape any / or . characters with \

Like: to replace http://www.metabunk.org/ with https://www.metabunk.org I do:



So the regular expression is: /http:\/\/www\.metabunk\.org\//i

 

[HWS]

MetaMirror edits the url in the post when it rehosts. So if you change where the images are you need to update the urls in the posts.

Probably the best way to do this is to use the Post Content Find / Replace

http://xenforo.com/community/resources/post-content-find-replace.1549/

THis is a little bit bad. It should only store the relative URLs!

 

[Mick West]

THis is a little bit bad. It should only store the relative URLs!

Then it would have to dynamically update them at run time, more complex, extra DB query. You can use the 1.3 Proxy system for that.

 

[HWS]

Then it would have to dynamically update them at run time, more complex, extra DB query. You can use the 1.3 Proxy system for that.

You simply could store it as "data/metamirror/pic.jpg".

Since in Xenforo all content comes from index.php in the root directory, this will always be a correct, non-broken URL and would allow the forum owner to switch the domain, the protocol and/or even the install directory without having to "search and replace" each single post message in his database.

Storing the images as "http://www.domain.com/installdirectory/data/metamirror/jpg" is only necessary if the forum uses a CDN, which also never can be changed easily. Maybe some kind of BB-Media code would also work? Like IMG-META working like IMG, but translates the path to the current CDN (or local storage)? If someone decides to deinstall your add-on he simply has to have a simple BB media code which instructs XF to handle "IMG-META" like "IMG".

At least it would be perfect to include a (deferred) rename function which allows to update all posts, if someone decides to switch the CDN, domain name, install directory or to/from SSL.

 

[Mick West]

You simply could store it as "data/metamirror/pic.jpg".

Since in Xenforo all content comes from index.php in the root directory, this will always be a correct, non-broken URL and would allow the forum owner to switch the domain, the protocol and/or even the install directory without having to "search and replace" each single post message in his database.

Storing the images as "http://www.domain.com/installdirectory/data/metamirror/jpg" is only necessary if the forum uses a CDN, which also never can be changed easily. Maybe some kind of BB-Media code would also work? Like IMG-META working like IMG, but translates the path to the current CDN (or local storage)? If someone decides to deinstall your add-on he simply has to have a simple BB media code which instructs XF to handle "IMG-META" like "IMG".

At least it would be perfect to include a (deferred) rename function which allows to update all posts, if someone decides to switch the CDN, domain name, install directory or to/from SSL.

I wanted to avoid going via index.php for speed. The add-on grew out of my own needs (a combination wanting to keep images from vanishing, and occasionally getting huge spike in traffic from all new visitors) , and I've been adding other features by request.

If people want a cleaner way, then I suggest they simply use the "Use Attachments" options - or ideally the new Image Proxy system in 1.3 - both of which are slower, but safer.

 

[Paul]

Mick,

Good to hear that this is still supported by you - Works perfectly as it is, and don't like the potential impacts on things like viglink when this does a perfect job

I shall be turning off the cache and carrying on using MM, at least for the foreseeable future...

Paul

 

[sweetpotato]

I cannot browse MetaMirror Test Tool. Page not found?
Thanks for the nice addon,

 

[Mick West]

I cannot browse MetaMirror Test Tool. Page not found?

Are you clicking on this?


If so, then can you provide more info as to how your forum is set up, version, paths, etc?

 

[sweetpotato]

Thank you,
Fix it after reinstalling the addon.

 

[sweetpotato]

I can run test tool now but it seem to not work for XF 1.3. Who has this addon work with XF 1.3 please confirm for me?
Thank you,