Anti Spam & Website Security

[spamtrawler]

Hi,

Out of self defence we have developed a Website Firewall and Security Suite, which evolved into more than the initial side-project.

We started developing SpamTrawler, when even our development sites became unmanageable due to constant spam registrations and posts (Blog, Comments, Forums aso...)

Initially the software had been offered to a small group of people belonging to a community we are working for as 3rd party developers.

By now it is used successfully on around 90-100 websites, running all sorts of host systems (Wordpress, SocialEngine, phpFox, OpenCart, vBulletin and even custom developed websites)

As SpamTrawler can be used on literally any php/apache based website, we thought that it may be interesting to introduce it to the xenForo community

A few functions are:
- Block countries
- Block single IP addresses
- Block complete IP lists
- Block IP Ranges
- Check your visitors IPs against Project Honeypot or other DNSBLs
- Block unwanted Charsets (Chinese, Japanese, Cyrillic)
- Block User Agents
- Block Referrer
- Block By Satellite
(Advanced Edition. Satellites are a decentralized network of SpamTrawler Server instances, automatically fed by connected members)

Additionally to the firewall functions, you will find "File Integrity" functions, which can help you to check your site for malware or unauthorized changes.

As fighting spam is best accomplished in a community, we added a social network around it, so everybody interested in fighting spam, can stay connected and discuss ideas

If this is of interest for the xenForo community, then please feel free to ask any questions.

You can visit our website at:
http://www.spamtrawler.net

Thank you very much to the xenForo team to allow us to post in the "Off Topic" thread.

Best Regards
Oliver

 

[Shadab]

Dear Visitor,
Your IP address: 122.168.xxx.xxx has been blocked by our system.

Looks like I'm blocked out. Not a good first impression, to be honest.
But I understand why it's blocked.

 

[spamtrawler]

Hi Shadab,

Reason for you being blocked is the country.
Unfortunately we had to temporarily block certain countries due to bad experiences in the past.

This can of course be configured freely by the administrator

You can use the captcha to unblock yourself on our website.

This block has not much to do with the Spam filter itself, it is a pure country block in your case.

Best Regards
Oliver

 

[Netsultants]

What kind of resources does this take?

I did a deny ip in an .htaccess file and it slowed down loading pages by 3-4 seconds.

 

[AnthonyCea]

One thing every webmaster should understand about blocking entire countries, if you do, Google will in turn not show your website to searchers in those nations you block, that is a fact !!

So, if you block Russia, you will be removed from the Google.ru search index completely and this could also effect the SEO for your entire domain.

 

[Netsultants]

That's good info, but if you are blocking that country I think it would make sense not to show on search engines for that country.

 

[AnthonyCea]

Well, Google is much smarter than most people think, they are not going to send users to a site that blocks their IP ranges.

 

[spamtrawler]

Hi,

First of all thank you all for your replies

@netsultants:

What kind of resources does this take?

I did a deny ip in an .htaccess file and it slowed down loading pages by 3-4 seconds.

One issue with .htaccess blocking is that no caching is taking place and parsing a huge list of IPs everytime you open a page, will definitely lead to a slowdown.
On top of this, keeping .htaccess blocklists updated can be quite time consuming.

SpamTrawler uses SQLite3 as a database backend, which gives it an enormous speed boost compared to MySQL (For this kind of applications)
Additionally using SQLite, relieves your MySQL database Server and does not count towards your MySQL query limit.

It can be regarded as a "GateKeeper", which starts before your website and hence blocked visitors will not use up system resources usually needed by loading your whole application.
This can speed up your site for legitimate users.

Just did some fresh benchmarks with v2.1:
Firewall switched off (include statement only): 0.0102 seconds
First visit all checks except DNSBLs and Satellite: 0.053 seconds
First visit all checks (Satellite check switched on): 0.513 seconds (IP unknown to satellite before, hence running all checks including DNSBLs)
Second visit (Visitor recognized): 0.0215 seconds

IPs currently in our local Blacklist: 13020


@AnthonyCea:

One thing every webmaster should understand about blocking entire countries, if you do, Google will in turn not show your website to searchers in those nations you block, that is a fact !!

So, if you block Russia, you will be removed from the Google.ru search index completely and this could also effect the SEO for your entire domain.

Of course that's true.
If you block Russia, then you will not be listed in Russia with your website, except you forward Russian traffic to a specific page.
But I guess that is exactly what should happen, in order not to be targeted by Russian (Or whatever nation) visitors/spammers.
For example we do not do any business with China, Senegal and a few other countries, hence we are not interested in listings in their search engines.

This however did not influence our SEO in a bad way (yet. past 7 months)
If you do a Google search for: Anti Webspam (for example), you will usually find us listed on the 1st or 2nd rank, and we are using Country Blocking as part of our configuration.

We also found that our Bounce Rate decreased massively, hence helping Alexa Rankings (This has been experienced by a few people)

Well, Google is much smarter than most people think, they are not going to send users to a site that blocks their IP ranges.

I don't quite understand this.
The are crawling our site frequently and our listings on Google are pretty well from what I can say.
I am not exactly a SEO professional, and only speak from experience, so I would definitely be happy to get some enlightenment in the SEO area

 

[Cezz]

Honestly, I see this thread as nothing but self advertising, and so for a site that is meant to be anti-spam you are indeed spamming... As for adding a poll to your own post asking if you find it useful or not... well I will not say any more about that...

Good luck but no thanks.

 

[Mikey]

poll = pointless

 

[spamtrawler]

I appreciate your spam - enjoy

Honestly, I see this thread as nothing but self advertising, and so for a site that is meant to be anti-spam you are indeed spamming... As for adding a poll to your own post asking if you find it useful or not... well I will not say any more about that...

Good luck but no thanks.

I'm sorry but honestly speaking I think that these posts are a bit off.
Before putting any post about our software in these forums, I have been in contact with the xenForo team and asked for permission to do so.
If this was intended as being Spam, then I guess it would have happened differently.

We see that there are people who are interested in how the software works, and hence asked questions.
These questions have been answered as detailed as possible.

One thing I don't understand is the hostility of some people, when it comes to introducing new software.
If you are not interested, why can't you not just move on and leave the thread alone ? (Especially if there are people having a normal discussion going on...)

 

[spamtrawler]

Yes, maybe the poll is pointless, but does it harm anybody ?
But there are people on every forum, who don't participate actively in discussions, but would leave a vote on the poll.

So the poll may not be as pointless as it looks like...

 

[Floris]

I think it's great that you asked permission prior to posting.
But it has nothing to do with XenForo and everything to do with making money by advertising here. For yourself.
I don't quite understand why using this site to advertise third party services / products would be permitted.
I know a lot of people, and I would not mind asking them to use this site to advertise their services and products - of course with asking first .. but I would be wondering why yours would be permitted as perhaps others might be declined.
It's fine with me, especially if it was permitted by the team, I just am expressing I find it an odd decision, and that regardless of it I consider this spam.
In my opinion (when it is with permission from team) it would be more suitable for this site if it offered a plugin. Now it's just advertising that its great for vBulletin ..

 

[AnthonyCea]

Floris, this is a vital issue of importance to all forum owners, here are some alternatives:

First let me tell everyone that the best way to use any firewall or IP blocking script would be at the server level, not on the app level, because if you do it on the app level you will have to ban on each domain, I am looking at one tool to block for all the sites on the server before spambots can see anything on any of my domains, that way I don't have to block on each app.

www.botscout.com

www.stopforumspam.com

http://www.spambotsecurity.com/zbblock.php

and a lot more in this thread:

http://www.forumpostersunion.com/showthread.php?t=8600

 

[Edrondol]

I think this looks interesting and is definitely worth a second look. I realize that spam is a big problem and that XenForo has a pretty robust anti-spam function, but the XF spam control is reactionary and only uses CAPTCHA as a deterrent. This looks to be a bit more proactive in its scope, which is a good thing.

I'd need to see a few more things before I'd pay the money though. Like how much server load, extent of false positives, methodology to determine spam. Things like that.

The web site is well done, though. Only issue is vB 4.1 as your forum...

Edit: StopForum Spam is what I've used in the past and few people get through. They do but it's few and far in between.

 

[spamtrawler]

I think it's great that you asked permission prior to posting.
But it has nothing to do with XenForo and everything to do with making money

Let me quote your signature please, without getting offended:

The one and only XenForo enthusiasts community : http://xenfans.com/

Live Now : The XenForo Marketplace & Job Board, and 'Getting paid to Publish'.

Now this is advertising I would say.
There is already a xenForo community, which we are using here at the moment.
If I go to your website and have a look at the Marketplace, it says "Premium Membership", which means somebody has to pay, hence making you money (maybe because you have to keep the server running).

I wouldn't consider your signature as spam, because I don't mind it being there.
I have the freedom of just not reading it and hence would usually never reply to your signature.

In this respect I think my post, and your signature are both to be seen as equally advertising no ?

"I don't quite understand why using this site to advertise third party services / products would be permitted."

The decision to allow me to post here has been made by xenForo, and hence it would be better to ask there I guess.

"I know a lot of people, and I would not mind asking them to use this site to advertise their services and products - of course with asking first .. but I would be wondering why yours would be permitted as perhaps others might be declined."

Maybe something did not come accross the way it should have.
There are a multitude of webmasters struggling with Spam on their platforms, and the software mentioned helps a great deal of giving these people peace of mind.

In the first place, it is intended to help webmasters who are struggling with spam and can't help themselves otherwise.

It's fine with me, especially if it was permitted by the team, I just am expressing I find it an odd decision, and that regardless of it I consider this spam.

Spam would be pushing useless information on people who are not interested in it.
If this was the case, then of course you would be right.

But in the case of this thread, people have not been flooded with information they did not want to read.
Instead, the community was given the chance to get more info if interested:

If this is of interest for the xenForo community, then please feel free to ask any questions.

Now there were people who are interested in more detailed info, and hence this info has been delivered as detailed as possible and with actual examples.
What is wrong in answering somebodies questions as detailed as possible, without making any "Wonder" promises or give them a "Buy Now" link and so on ?

It was and is not the intent of this thread, to Spam anybody.
Those who are not interested, could easily just move on and forget about it instead of hijacking a legitimate thread.

In my opinion (when it is with permission from team) it would be more suitable for this site if it offered a plugin. Now it's just advertising that its great for vBulletin ..

SpamTrawler is designed to work with literally any PHP based application, so a plugin would not make sense, as it is rather limiting and would bring back the limitations of integrated modules which have to be maintained separately from each Admin Panel "example: CMS + Forum + Shop".

Now it's just advertising that its great for vBulletin ..

This is xenForo not vBulletin...

 

[Mike]

The meta-discussion that has taken over this thread isn't really going to go anywhere. If you have questions about the product, I'm sure you can ask the creator directly.