Elastic Search For Everyone - Even Free For Small Forums

[Anthony Parsons]

For those not in the know about Amazon Elastic Search, you can use that external to your server as an ES solution.

That means for those using shared hosting, you can use ES via Amazon.

The starter level is totally free, meaning those running a small forum, ES via Amazon for next to nothing. I believe they charge you disk storage only as they force the small accounts onto their EBS file system... so you may get a 50c or $1 charge per month for the tiny size of a small forums ES instance.

Being Amazon, it means you can additionally host it near to your server, regardless where it is in the world.

https://aws.amazon.com/elasticsearch-service/pricing/

Once you configure it you will be given an elastic URL which you enter into your XF ES field and the port will become 80. Simple as that once created at Amazon.

Start at: https://aws.amazon.com/elasticsearch-service/

I'm testing 1.1m posts on the smallest Amazon instance at present (partially free service), and its delivering without issue. Super fast... less crap with java bloat and RAM issues at my server.

 

[darenk]

Thanks for the heads up on this one, have just signed up and will give it a crack for a new site I'm putting together.

 

[oman]

Thanks for this info @Anthony Parsons

 

[Joe Blow]

OK, I have set this up but have no idea how to integrate it with my forum.

Any tips?

 

[Mouth]

The starter level is totally free

... for your first 12 months only, after which is will start at ~$13 per mth. Ref: https://aws.amazon.com/free/terms/

I believe they charge you disk storage only as they force the small accounts onto their EBS file system... so you may get a 50c or $1 charge per month for the tiny size of a small forums ES instance.

+ data transfer between your server and the ES instance. If you're not using other AWS services, then you'll be $0 since first 1Gb per month is free.

 

[Anthony Parsons]

OK, I have set this up but have no idea how to integrate it with my forum.

Have your access policy set to use your sites IP address, then with XF ES installed, go to the ES options in XF ACP and enter the URL it gave, without http or such, and use port as 80 and save.

That will connect your site IP to your Amazon ES instance. Rebuild the search cache is your last step.

 

[Anthony Parsons]

... for your first 12 months only

Yer... for a large forum it could get expensive, but for a small forum on shared hosting, viable.

 

[Anthony Parsons]

I found a simple solution for all of this now... having been pulling my hair out for weeks over issues with ES -- setup a single SSD linode for $10 a month (1core - 2gb ram - ssd), running ES5, locked down with fail2ban and firewall, then secured to your servers IP only via nginx. Any other IP attempts to connect and its rejected.

Took some playing around and learning, but got it nailed, isolated to its own instance so no more RAM affect due to web traffic peaks and troughs.

Found nginx isn't good for large rebuilds, so it must be disabled and direct remote connection to ES for the initial build, then activate it for its security purpose as a working instance.

 

[dvsDave]

I found a simple solution for all of this now... having been pulling my hair out for weeks over issues with ES -- setup a single SSD linode for $10 a month (1core - 2gb ram - ssd), running ES5, locked down with fail2ban and firewall, then secured to your servers IP only via nginx. Any other IP attempts to connect and its rejected.

Took some playing around and learning, but got it nailed, isolated to its own instance so no more RAM affect due to web traffic peaks and troughs.

Found nginx isn't good for large rebuilds, so it must be disabled and direct remote connection to ES for the initial build, then activate it for its security purpose as a working instance.

Why did you go this route instead of Amazon's ES setup? Just cost, or were there other headaches?

 

[Anthony Parsons]

Amazon was actually super simple and easy. Only one site qualifies for a new account, I have two, and the size of me second, whilst it ran fantastically on ES... it was not going to be cost effective to run that site from Amazon, even with the free trial period, as the bandwidth exceeded the free trial limits for a year. You can keep setting up amazon accounts, no question about it... but I don't want those hassles when a simple VPS for $120 a year can do the job without any further concern.

Like I said first up... if you have a small site and are on shared hosting or such, Amazon is likely your best and cheapest solution. For larger sites... not so much, unless of course you're running on Amazon EC2 or such, then you aren't paying for bandwidth either.

If you host on Amazon, the Amazon ES is just an all round winning solution. If you host external to Amazon... a cheap VPS will do the job for you likely cheaper.

 

[PJK]

Amazon was actually super simple and easy. Only one site qualifies for a new account, I have two, and the size of me second, whilst it ran fantastically on ES... it was not going to be cost effective to run that site from Amazon, even with the free trial period, as the bandwidth exceeded the free trial limits for a year. You can keep setting up amazon accounts, no question about it... but I don't want those hassles when a simple VPS for $120 a year can do the job without any further concern.

Like I said first up... if you have a small site and are on shared hosting or such, Amazon is likely your best and cheapest solution. For larger sites... not so much, unless of course you're running on Amazon EC2 or such, then you aren't paying for bandwidth either.

If you host on Amazon, the Amazon ES is just an all round winning solution. If you host external to Amazon... a cheap VPS will do the job for you likely cheaper.

When I setup AWS ES and created a new domain, it gives a VPC Endpoint. Is that what it used as the "Elasticsearch Server" in XF ES settings? I'm getting "cannot connect" when testing the connection in XF. If you aren't using the VPC Endpoint, what are you using? Thanks.

 

[Sumanta0018]

same question. @Anthony Parsons can you please guide, how to setup