1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.2 Spam Prevention and Management

Discussion in 'Have You Seen...?' started by Mike, May 28, 2013.

  1. Mike

    Mike XenForo Developer Staff Member

    While 1.1.4 added some spam prevention tools, XenForo 1.2 takes these further. This is by no means the end of our fight against spam though.

    In 1.2, we've mostly focused on adding support for APIs and adaptable spam prevention techniques. Let's look at a few.

    Deeper Stop Forum Spam Integration
    We actually detailed this a long time ago. While 1.1.4 added support for checking a DNSBL that used the Stop Forum Spam database, 1.2 has a much more configurable integration:


    In addition to simply checking registrations against SFS, you can also get an API key and submit back to them automatically when spam cleaning a user.

    If you opt to use the SFS integration to check registrations, the Tornevall DNSBL will not be checked.

    Project Honey Pot HTTP Blacklist
    In addition to the existing DNSBLs, you can sign up for a Project Honey Pot API key to check their blacklist. While this often overlaps with SFS's database, there are some variations.


    Message Content Checking: Akismet and Spam Phrases
    Up until now, we've focused on preventing spam registrations. While that's the ideal situation, it's not always possible to detect spam until you see what they actually try to post.


    Here, we provide a simple "spam phrases" option, which uses simple pattern matching (or regular expressions) to detect spam. For example, here on XenForo.com, the vast majority of the spam we see fits a particular profile that can be matched with "watch * online". Content matching these phrases will be rejected or set to be manually approved. The checks are done on posts, profile posts and conversations. In situations where manual approval isn't possible (such as in conversations), the messages will be rejected.

    To limit the potential for false positives, the checks are limited to only the first few messages by a user.

    Then there's Akismet. It provides an API that uses various heuristics to tell you whether the message being submitted is spam. Note that depending on your site's purpose, Akismet may not be a free service.

    Spam Cleaner Can Delete Sent Conversations
    To help deal with conversation spam, the spam cleaner can now (optionally) delete all conversations started by the spammer. This will remove the conversations from the inbox of anyone that received them (and update their unread conversation counts).

    For Developers: General Spam Prevention API
    There is now a SpamPrevention model that is called in various scenarios. You can simply hook your spam prevention call into the correct position and return what action to take. This is called in the following situations:
    • When a new user registers
    • When a user is spam cleaned (to submit spam users and spam content)
    • To determine if the user posting a message needs to be spam checked
    • When a message is submitted
    • When a message that was moderated because of spam and is approved (submitting "ham")
    This system should make it easier to add-on developers to integrate their own spam prevention systems.
    Liam W, radu81, dieketzer and 85 others like this.
  2. Slavik

    Slavik XenForo Moderator Staff Member

    Excelent work Mike! (y)
  3. tenants

    tenants Well-Known Member

    ahh, StopForumSpam + Project Honey Pot are being added, good job Mike ;)

    - Adding StopForumSpam via this method makes every XF forum a reporting site. This is something XRumer users often try to avoid (so they avoid IP/Email address detection longer). If you then report lots of spammers, you become a top reporting site... and as such, there is a good chance many spammers will avoid you (y)

    Butta-bing Butta-bang, the more you report, the more you will be avoided :cool:

    This is a good thing, a really good thing. StopForumSpam integration will hurt them for a while!

    API's are a good idea for core software, reporting integration makes the API very strong

    Adding weak mechanism to the core is not a good idea, but adding APIs and integrating them is a very good idea for the core!
    Boothby likes this.
  4. DRE

    DRE Well-Known Member

  5. Chris D

    Chris D XenForo Developer Staff Member

    A HYS on Friday, Monday and Tuesday?

    What could it all mean? :cautious:
    DarkGizmo, Lisa, Veer and 2 others like this.
  6. Mike

    Mike XenForo Developer Staff Member

    Keep in mind that you still need an API key to do the reporting to SFS, so there is some friction there (unfortunately).
    Chris D likes this.
  7. Chris D

    Chris D XenForo Developer Staff Member

    Captcha is actually a prevalent point, but I guess it's also true that there's the potential to invest a lot of time into Captcha only to find it eventually gets cracked. At one point reCAPTCHA was THE Captcha to beat them all, now look at it. A piece of software can solve it.

    If anything, you could implement something really simple and effective, custom made for XenForo.

    CustomImgCaptcha from tenants is by far the most effective CAPTCHA I've seen and it's incredibly customisable and I can't see bots ever being able to crack it programatically.

    The premise is simple: An image is displayed and the user has to answer a question on that image. No fancy click and drag a jigsaw puzzle. No random letters that you have to reload 5 times before you can actually read it.

    Probably the best part about it is the fact that it's incredibly simple to customise and theme it for your board.

    EDIT: To add, one of the things that makes it so powerful in tenants' add-on, is the reporting you get back. It's feasible that eventually the answers to your CAPTCHA could end up in a database somewhere and be crackable automatically. But if you have reporting on:

    Attempts | Attempts Failed | Attempts Succeeded

    You can quickly identify those CAPTCHA that aren't as effective and take them out of circulation.
    DarkGizmo likes this.
  8. tenants

    tenants Well-Known Member

    I know that they changed their TOS a little while ago, but I think the only issues they are really concerned about is
    1) Gaining false positives
    2) There API being used like a firewall

    As long as the reporting method sends proof, and that the API look-ups for IP address/emails are cached for a certain amount of time (to avoid bots that attempt to register thousands of times in a short period) I can't see why they would ever have an issue with this. After all, core reporting is providing them with a tone of data making them statistically very strong.
  9. xf_phantom

    xf_phantom Well-Known Member

    Will there be a notice on the registration page, that the data (username,email ip) are being sent to a 3rd party for spam prevention, if the feature is enabled?
  10. jerremy

    jerremy Member

    I have to agree with this. I have been checking the tools that were and are being used to spam forums, and there is no tool or anything that can automatically solve questions in images.
  11. Mike

    Mike XenForo Developer Staff Member

    Oh I understand why, I just mean that people will need to take a proactive step to support the reporting, so it'll end up being a small subset of the total installs.

    We haven't changed anything here.
  12. Stuart Wright

    Stuart Wright Well-Known Member

    Will there be an option to submit a user to SFS when banning normally? Say if you know a spammer has registered, but haven't posted any content yet?

    I ask because we have a very successful method of preventing manual spam submissions on vB3.8 which I want to bring forward to our XF migration in August.

    When users register, we assign their account a country based on a lookup of their IP in the GeoIP Lite database. And then if anyone from a list of problematic countries (India, China, Russian Federation, Philippines, Bangladesh, Pakistan, Morocco, Ukraine, Sri Lanka, Anonymous Proxy, Indonesia, Nepal, Thailand, Vietnam and Nigeria) creates any message (via posting new or editing) or a signature which contains any web or email link, the submission is blocked with a spam warning, and admin are emailed with the message/signature which they tried to submit.
    We can tell whether it is spam or not and usually ban with SFS submission. The vast majority of posts from the above countries are spam, so the system works well and AVForums visitors rarely see any spam.

    The options you have detailed above will be great to prevent a lot of spam, but what about signatures, and specific targeting of all content submitted from certain countries which contains web links or email links?
  13. Mike

    Mike XenForo Developer Staff Member

    You can spam clean from a profile page, which would submit it to SFS.
  14. Jeffin

    Jeffin Well-Known Member

    I have seen a reduction in spam accounts recently. It must be the recent upgrades or the tougher Q&A I implemented. It's always better to stay ahead of the spammers so thank you for all the work on it.
  15. tenants

    tenants Well-Known Member

    There are tools, they are just not used in XRumer yet (just google OCR)... there are even open source artificial neural networks written in JavaScript, I can write a tool using ANNs, so the technology is there.

    What I can't do is identify custom objects within that image (at least not very easily). For example "a pink elephant stepping on a mouse", if the question was "what is the elephant stepping on" I would find it easy to extract the text from the images (most of the time), but quite hard to relate the text to the objects in the image... so I would be left with brute forcing for anything associated with "elephant" (mouse will come up in that list of associations). If I use a big arrow and point to the mouse, and ask, what is the arrow pointing at... my list of brute for attempt becomes very large, making the image CAPTCHA very hard to solve programmatically (yet still very easy for humans)
  16. CyclingTribe

    CyclingTribe Well-Known Member

    At long last - a solution to auto-suppress the "* kitchens" and "kitchens *" spammers ... :ROFLMAO:
  17. M@rc

    M@rc Well-Known Member

  18. Stuart Wright

    Stuart Wright Well-Known Member

    Using message checking will put submissions containing links from members with fewer than, say 5 posts into the moderation queue. However, if this happens for all countries, there will be too many messages in the queue and most of those being from UK posters, will be perfectly fine but frustrating for the new poster that their message is not published right away.
    Therefore I think that having the country as a filter is important.

    And by the way, our method of country targetting stops all the kitchen spammers (we had several attempts this morning) because they are all from India.
  19. ZeWildGuy

    ZeWildGuy Member

    Spam is one of the biggest problems with forums, but XenForo is the best with handling it! Great work guys! :D
  20. whynot

    whynot Well-Known Member

    ...but an email already arrived about the conversation into the receiver's email address.
    Alfa1 likes this.

Share This Page