Is 512MB RAM too less for installing XFES (e.g. elasticsearch)

[abdfahim]

I run a community with 20k posts, ~50-75 simultaneous users (total around 1500 members). So far I have been using a shared hosting without an issue.

Now I want to get XFES, hence will need elasticsearch. So, I got a VPS with 512MB RAM. Should I be worried about performance if I install XFES with elasticsearch 5.x in a CentOs 7.x system?

 

[Brent W]

Java will not install.

 

[abdfahim]

Did you mean I'll have performance issue if I install Java? Becuase I have actually installed OpenJDK already, but I am yet to migrate my site.

 

[Brent W]

Did you mean I'll have performance issue if I install Java? Becuase I have actually installed OpenJDK already, but I am yet to migrate my site.

How much swap does the server have? ES will more than likely not start because Java will use more memory than you have.

 

[Mouth]

You'll need at least 4GB if you want Web, DB, Java, and ElasticSearch all on the same server.

 

[abdfahim]

How much swap does the server have? ES will more than likely not start because Java will use more memory than you have.

Hmm, you are right, I could install ES but it is not starting.

You'll need at least 4GB if you want Web, DB, Java, and ElasticSearch all on the same server.

You mean 4GB RAM? Wow, way over my budget.

In that case, I think I have no reason to move to a VPS. The main reason I wanted to move from my shared server to a VPS was XFES. Though I also occasionally get max_user_connection reached error, that happens once in a few days.

 

[Brent W]

Hmm, you are right, I could install ES but it is not starting.


You mean 4GB RAM? Wow, way over my budget.

In that case, I think I have no reason to move to a VPS. The main reason I wanted to move from my shared server to a VPS was XFES. Though I also occasionally get max_user_connection reached error, that happens once in a few days.

Plenty of other reasons to move from Shared to a VPS than just ES.

 

[Mike Edge]

Plenty of other reasons to move from Shared to a VPS than just ES.

Not true. Our shared hosting services out perform a VPS and several of our clients use ES which is included with our hosting packages.

 

[Brent W]

Not true. Our shared hosting services out perform a VPS and several of our clients use ES which is included with our hosting packages.

There are still plenty of other reasons to move from shared to VPS. You are very much the exception and not the rule when it comes to shared hosting.

 

[rainmotorsports]

Elastisearch is the killer here. You can with a properly tuned server (disable MySQL Performance Schema for one) handle your load at 512 mb. No more than 250 concurrent users and going over 200 starts to see a small penalty. But ES needs ram and that doesn't make the cut. No ES, No Java, you would probably be fine.

 

[Paul B]

Hmm, you are right, I could install ES but it is not starting.

Do you mean the elasticsearch service on the server won't start, or the XFES add-on doesn't work?

 

[abdfahim]

Do you mean the elasticsearch service on the server won't start, or the XFES add-on doesn't work?

Elastic search. I haven't bought XFES yet, just purchased a VPS and was preparing the server for XFES. But it seems like it's a no go for XFES. I can't afford to buy a VPS 4GB+ RAM, unfortunately.

 

[abdfahim]

There are still plenty of other reasons to move from shared to VPS. You are very much the exception and not the rule when it comes to shared hosting.

You might be right, but with not much revenue, I am a bit skeptical of paying $100/year for a VPS where the site is now running on a $35/year shared hosting without any major issue.

Also, I am just thinking how much benefit I can expect from a VPS with 512MB RAM compared to my current shared 1GB RAM plan.

I don't have any issue with storage/BW so far, that's why I am emphasizing on the RAM only. Of course, I will be able to increase max_user_connection in a VPS, but I am thinking whether solving an issue that occurs seldom, once in a couple of days, worth spending extra $70-sh.

 

[Xon]

My testing environment fits into a 1gb Linode VM for $60 USD per year (or $5/m)

Restrict elastic search (5.x branch!) to 128m, MariaDB 10.2 to 128mb, nginx (1 worker), php-fpm to 128mb per worker with at most 4 workers and you need at least 512mb of swap.

Spoiler: Critical config bits

/etc/my.cnd.d/server.cnf

innodb_file_per_table=1
innodb_buffer_pool_size=128M
aria-pagecache-buffer-size=16M
default-storage-engine=innodb

innodb_flush_method=O_DIRECT
innodb_read_io_threads=64
innodb_write_io_threads=64
innodb_io_capacity=3000

/etc/elasticsearch/jvm.options

-Xms128m
-Xmx128m

/etc/php-fpm.d/www.conf

php_admin_value[memory_limit] = 128M

The absolutely critical part about this is "innodb_io_capacity" in MySQL config and having SSDs for MySQL & ElasticSearch

This works fine for when half a dozen or so of my staff hit it for testing.

There are some edge cases where you might need more maximum ram for php-fpm workers depending on add-ons and avatars being uploaded. Average php memory usage is about ~12-16mb per request, which is what makes this all work.

I write virtually all the add-ons my site uses, so I can ensure memory usage stays sane.

 

[abdfahim]

Thanks for sharing Xon. $60/yr for 1GB RAM is pretty good, I am now paying $100/yr for 512MB RAM.

So, can you please tell me whether I need to install my OS as well if I take VPS from Linode, or the OS is pre-installed?

 

[Xon]

So, can you please tell me whether I need to install my OS as well if I take VPS from Linode, or the OS is pre-installed?

You build the VM for a plan, and then apply an OS to the empty VM. They have a list of supported OS's, which includes CentOS 7

 

[abdfahim]

Thanks a lot. Out of curiosity, you didn't try 1GB RAM on you spacebattles or sufficientvelocity forum, did you?

 

[Brent W]

Thanks a lot. Out of curiosity, you didn't try 1GB RAM on you spacebattles or sufficientvelocity forum, did you?

Those sites wouldn't even come close to running on a 1GB plan.

 

[rainmotorsports]

Thanks a lot. Out of curiosity, you didn't try 1GB RAM on you spacebattles or sufficientvelocity forum, did you?

HAHAHA Yeah they run on dedicated servers or at least plans equivalent of such. I can't remember his specs but they are up there.

 

[Xon]

Thanks a lot. Out of curiosity, you didn't try 1GB RAM on you spacebattles or sufficientvelocity forum, did you?

Hell no. Spacebattles uses a $40/m Linode VM for the database purely based on the size of the database(+60gb). Sufficient Velocity gets away with a $20/m month Linode VM.

Both sites use multiple front-end nodes, (tiny) dedicated VM's for testing, and a pile of custom code for performance and a very careful eye for how add-ons are coded.

HAHAHA Yeah they run on dedicated servers or at least plans equivalent of such. I can't remember his specs but they are up there.

At some stage I'm thinking of renting dedicated servers and moving off Linode, but that isn't going to be required for performance reasons for a while yet. Cost-wise it is getting close, and mostly comes down to finding a good dedicated machine provider who is offering good pricing which requires me putting in time on it