MetaMirror 1.6.1

[Mick West]

Looks like curl is not enabled for php, the solution for this depends on your setup, so I'd suggest asking google with the error "Call to undefined function curl_init()" + your setup.

I'll have to add a check for this in later versions. It used curl to fetch the file size to avoid downloading overly large images. It could operate without it, but would need to download the file to calculate the size.

 

[russoroni]

Yeah I just migrated to a vps and am learning how to use webmin + virtualmin. I just looked through the software packages and I dont see anything pertaining to curl installed so I'm assuming thats what it is.

 

[Mick West]

From the command line it would probably be:

sudo apt-get install php5-curl

 

[Mick West]

Or possibly you might need more

sudo apt-get update
sudo apt-get install curl libcurl3 php5-curl

 

[russoroni]

I installed it but its still doing the same thing

 

[Mick West]

you will need to restart apache

/etc/init.d/apache2 restart

or maybe reboot if you can't do that

 

[Mick West]

Also, see:
http://www.turnkeylinux.org/forum/support/20090507/installing-curl-apache

You may need to add the following line to the /etc/php5/apache2/php.ini file:

extension=php_curl.so

(and restart)

 

[russoroni]

I'm running nginx

 

[russoroni]

ok all is well. Just installed curl and its working. Thanks for all your help.

 

[Mick West]

Great! Sorry for the lack of useful error messages. I'll fix that next update.

 

[russoroni]

wow man. This is a seriously cool plugin. Works perfectly.

 

[Stuart Wright]

Hi @Mick West, having read all 19 pages of this thread, I'm impressed with the effort and diligence you've put into maintaining this addon.
I asked in another Xenforo thread here about how to tackle cookie stuffers. These are spammers who post a genuine looking message, but then embed a 1x1 clear gif which generates (false) traffic on a website. They get paid for this activity.
In vBulletin we used to display the number of images in each post to the moderators so they could spot this kind of spamming.
@jakew009 suggested I use MetaMirror to solve this problem.
So I have some questions please.
The cookie stuffers sometimes use a redirect service like bit.do to point to the gif. How would MetaMirror deal with that?
Second, I assume that a 1x1 pixel gif would be saved as an attachment the same as any other?
If that gif is embedded using attach=full, the moderators would still not be able to see it and so while the cookie stuffing would be thwarted, the spammers would still go undetected? So this addon wouldn't really help us to kill the spammers?

Third question, and with reference to using MM for the purpose for which it was designed, bearing in mind we have 14 years worth of posts, if an image is broken, I would definitely want to replace that with a note saying that the image at [image url] can not be found. But MM does not do that at the moment, right?

And lastly has MM been used on a site with as many posts as we have? 20 million?
Thanks

 

[HWS]

@Stuart Wright : At a large forum you should really disable the direct linking of images due to security concerns. We simply show a link instead to our users.

If you use this add-on it may also solve your problems but you may run into a new problem because not all images are allowed to be rehosted due to copyrights.

 

[Stuart Wright]

@Stuart Wright : At a large forum you should really disable the direct linking of images due to security concerns. We simply show a link instead to our users.

If you use this add-on it may also solve your problems but you may run into a new problem because not all images are allowed to be rehosted due to copyrights.

Not showing images would kill our photography forum which is all about sharing photos on their Flickr accounts etc.
I didn't read one query about copyright in this whole thread.

 

[Mick West]

The cookie stuffers sometimes use a redirect service like bit.do to point to the gif. How would MetaMirror deal with that?

Metamirror handles redirects. Assuming the redirect service uses a 301 redirect (which bit.do does). I just tested a bit.do linked image, and it worked fine.

Second, I assume that a 1x1 pixel gif would be saved as an attachment the same as any other?

Correct, all images are handled the same.

If that gif is embedded using attach=full, the moderators would still not be able to see it and so while the cookie stuffing would be thwarted, the spammers would still go undetected? So this addon wouldn't really help us to kill the spammers?

Correct. Sounds like ideally you'd want something that's essentially a cookie stuffer detector. Something that flags very small images? Perhaps even auto-reports the post?

Third question, and with reference to using MM for the purpose for which it was designed, bearing in mind we have 14 years worth of posts, if an image is broken, I would definitely want to replace that with a note saying that the image at [image url] can not be found. But MM does not do that at the moment, right?

Correct. It simply retries a few times, and then skips over that image. This is something I might add as an option - it would be useful to be able to search for the broken links, and then try to fix them.

And lastly has MM been used on a site with as many posts as we have? 20 million?
Thanks

Not that I'm aware of, people have run it with a few hundred thousand posts with no problem. The most likely issues would be to do with the cron job which hosts old posts. it will take a few days to get through everything. The SQL query might be a bit slow with that many posts. It can be disabled after rehosting is complete.

And remember this is a one-way conversion. There is no easy undo. Definitely run it on a test version of the site first.

 

[yavuz]

MetaMirror works perfect most of the time but with posts that have a few images (and depending on the size) thread creation runs out of time a lot... Than I have every now and then users who would hit the send button a few times and the threads gets duplicated...

Maybe not directly releated to this addon but I had the idea in my head for some time to show the user, upon thread creation, information about the background process. Something like loading information (images are being rehosted, imagexzt is being rehosted....)

 

[Mick West]

MetaMirror works perfect most of the time but with posts that have a few images (and depending on the size) thread creation runs out of time a lot... Than I have every now and then users who would hit the send button a few times and the threads gets duplicated...

Maybe not directly releated to this addon but I had the idea in my head for some time to show the user, upon thread creation, information about the background process. Something like loading information (images are being rehosted, imagexzt is being rehosted....)

I think perhaps a better way (and possibly easier to program) would be to defer the processing somehow, so that it actually does happen in the background. Right now it's blocking while it fetches the images. The thread/post could then be immediately viewed with no lag (other than the local image fetch, which would generally be cached anyway). I'll have a look into it.

 

[yavuz]

I think perhaps a better way (and possibly easier to program) would be to defer the processing somehow, so that it actually does happen in the background. Right now it's blocking while it fetches the images. The thread/post could then be immediately viewed with no lag (other than the local image fetch, which would generally be cached anyway). I'll have a look into it.

That would be wonderful and maybe simpler compared with my fringe suggestion

 

[Mick West]

Mick West updated MetaMirror with a new update entry:

Flagging of broken image links

Added an option (off by default) to flag broken IMG links with a message, and convert the IMG tags to URL tags. Example result:

[Broken External Image]:http://www.natureasia.com/asia-materials/article_images/640.jpg

Use with caution. Some (rare, for me) images that might embed fine are not fetched correctly by curl, presumably because of some kind of hotlink protection (which is kind of backwards, as hotlinks work).
One such image being:...

Read the rest of this update entry...

 

[Mick West]

Third question, and with reference to using MM for the purpose for which it was designed, bearing in mind we have 14 years worth of posts, if an image is broken, I would definitely want to replace that with a note saying that the image at [image url] can not be found. But MM does not do that at the moment, right?

It does now (with caveats, above).