XF 2.3 Prevent bots from downloading resources

[Old Nick]

I allowed guests to download resources, but the download statistics are distorted by bots, how can I avoid this?
Maybe an extra step with a captcha? Is it easily implementable?
Thanks.

 

[Opus X]

when i looked at my site robots.txt i noticed tracy had Disallow: /resources/*/download in it.
i was looking how to block some of the bots that have been hitting the site.
i think he copied his over to my site since i do not have resources to download.
i remember he was complaining about this in our staff area on his site.

 

[Old Nick]

Interesting, i will try that. Thanks.

 

[Old Nick]

I have the impression that this does not work, the stats continue to increase abnormally compared to the activity and notoriety of my forum.

But maybe I have to wait longer for the robots.txt file to be taken into account? Or is it almost instantaneous?

 

[Miri]

Blocking via robots.txt is only useful for some bots generally the most known ones. CAPTCHA is much better in your case.

 

[Old Nick]

CAPTCHA is much better in your case.

Of course ! But how to implement the captcha step ?

 

[Old Nick]

I managed to get an extra step with this nice add-on i used on my previous forum:
https://xenforo.com/community/resources/d-c-style-advanced-download.8026/

I managed to implement CloudFlare Turnstile in the extra page:



But even if from time to time the Turnstile widget asks me for a manual action I can, without doing it out, still download the file.
How is this configured, so that the button is either not clickable or hidden until Turnstile gives the green light?

Thanks !

 

[Miri]

Sorry but I don't know, I just wanted to point out that between the two it was better to opt for CAPTCHA.

I'm just wondering why you allow guests to download attachments? If I were you I'd leave it to registered users only, but if you made that choice you have your reasons.

Your download link can also be shared by reliable sites, but they have a high amount of traffic but consume your resources without even visiting your site.

 

[Old Nick]

I'm just wondering why you allow guests to download attachments? If I were you I'd leave it to registered users only, but if you made that choice you have your reasons.

Quite simply because these are resources that are not exclusive to my forum, I find them myself on the net by searching everywhere. My little plus is that they are all cataloged in the same place, they are well presented with a cover photo and an extract (they are only books). And you know anyway, even if it's reserved for members, it only takes one malicious person and the resource is distributed elsewhere... Even the paid ones are widely distributed on null sites.

Your download link can also be shared by reliable sites, but they have a high amount of traffic but consume your resources without even visiting your site.

But in any case the link shared will be, in the vast majority, that of the resource and not that of the direct download I think. What do you think?

Frankly, if I managed to avoid downloading bots it would be really good, it would allow me to really know the popularity of this or that resource and to have statistics representative of reality.

 

[Old Nick]

But maybe this extra page will do the job on its own? How will the robots react? I should point out that the add-on allows you to delay the click on the download button with a countdown, set to 5 seconds by default. Maybe this will confuse the bots enough so that they don't download the resource. I'll follow this with the stats...

 

[Old Nick]

I removed the CF Turnstile widget and it seems to work well without it.
Since installing the add-on, there have been no more abnormal increases in statistics.

 

[Opus X]

I have the impression that this does not work, the stats continue to increase abnormally compared to the activity and notoriety of my forum.

But maybe I have to wait longer for the robots.txt file to be taken into account? Or is it almost instantaneous?

the bot has to honor it from what tracy said.
if it does not then the count will increase?
only way to get around that is a plug in probably.
or not allow downloads by guests.
from the discussion his big issue was google and the major search engines driving the count download counts up.

 

[Old Nick]

Well you know what? I'm finally going to let the bots "download" my resources, and this for an unexpected reason: as I have no or very little activity on my forum currently, the trending resources widget permanently displays the same resources... but with the random bot activity the widgets update regularly by changing the content displayed! Big up to them.