New Search functionality

[Slavik]

I'm guessing that big boards => boards with 1 million+ posts. Or do we have a different definition of big (/large) boards?

I think $50 + $10 (recurring/year) is *nothing* for a big board; for the cost would be recovered within just a few hours if not minutes. I'm not sure how the new product improves the whole search experience, but I'm definitely going to give it a try when the right time comes.

I'm quite excited to see the announcement. Just that I've few more months to wait before I see 1.2 makes me a little nervous. Given my experience of using Beta; I think I'm going to upgrade as soon as we get the mobile style

K&M - release the mobile style with first beta, please. My users are dying for it.

Yes 1 Million is the general mark.

 

[tmb]

Price is a bit high probably in comparison to the entire product but with 6 million posts its a must purchase. Glad its finally here.

 

[Marc]

Yes 1 Million is the general mark.

On half a million with no problems on standard search and tbh the cost of the search doesnt look in any way expensive in comparison to some things I would pay for on my site. Therefore from the point of view of someone over the 1mill level I would see the cost as a proverbial drop in the ocean.

 

[wrx1]

So the next question is, what kind of resource requirements are we looking at to make the new search run, then run well??

For my community, we have a lot of information within the site. We are only around the 650k post mark and the current search works ok, but as time goes on, we will collect more posts. At the same time, our active community doesn't change much. So for the past 2-3 years, our online users and guests really don't change much (120-150 combined).

Russ

 

[Blackbeard]

Comparable solutions on vBulletin 3.x run about quite a bit more (about 1500), to be fare it does quite a bit more...but the only viable solution to big board owners.

This makes xenforo the base of all my forums functionality.

My only question is, can it incorporate into more, such as wordpress and such type of structures?

 

[high1976]

I won't expect any custom sphinx-plugin to be cheaper so it's fully ok for an official additional plugin.
Just out of curiosity, is there any chance for some more functionality to the new search system, namely search through conversations (as Ragteks plugin is pretty dead) and search within specific nodes via link (like vb's search.php?do=getnew&include=) in the future?

 

[Onimua]

I dont think that this is a abig deal for Xenforo.
How much (big) Boards will need - and buy - this search function?
So, if they really want to make fast money, there are other easier ways

Looking at the responses so far from a small sample, it looks like almost all of them will buy it.

I also think it's quite unfair to say that their motives here are to make fast money. As far as I have seen, their motives have been to make a fantastic product that make me want to give them money.

 

[Paul B]

So the next question is, what kind of resource requirements are we looking at to make the new search run, then run well??

You need Java virtual machine (JVM) on the server.

Some light reading if you're interested: http://www.elasticsearch.org/guide/

Edit: Just noticed Deebs already mentioned JVM: http://xenforo.com/community/threads/new-search-functionality.25735/#post-311392

 

[Deebs]

So the next question is, what kind of resource requirements are we looking at to make the new search run, then run well??

For my community, we have a lot of information within the site. We are only around the 650k post mark and the current search works ok, but as time goes on, we will collect more posts. At the same time, our active community doesn't change much. So for the past 2-3 years, our online users and guests really don't change much (120-150 combined).

Russ

Some stats from my forum....

MySQL index:

8.7K 2011-10-20 20:51 xf_search_index.frm
1.4G 2012-01-10 12:56 xf_search_index.MYD
3.1G 2012-01-10 12:56 xf_search_index.MYI

The same content indexed by ElasticSearch:

size: 2.9gb
docs: 4053718

I have allocated 4gb to my JVM running ElasticSearch to ensure that it does not run out of memory to service my index. The 1.4gb of RAM used by the MyISAM index file has been reclaimed to increase the memory allocated to InnoDB on my SQL server.

 

[Slavik]

So the next question is, what kind of resource requirements are we looking at to make the new search run, then run well??

For my community, we have a lot of information within the site. We are only around the 650k post mark and the current search works ok, but as time goes on, we will collect more posts. At the same time, our active community doesn't change much. So for the past 2-3 years, our online users and guests really don't change much (120-150 combined).

Russ

Your available RAM is the biggest contributor, Elasticsearch holds open a lot of files at once, so having the capacity for that without writing to SWAP is key.

Comparable solutions on vBulletin 3.x run about quite a bit more (about 1500), to be fare it does quite a bit more...but the only viable solution to big board owners.

This makes xenforo the base of all my forums functionality.

My only question is, can it incorporate into more, such as wordpress and such type of structures?

Elasticsearch uses a prefix to seperate different applications, so yes, as long as they have an Elasticsearch pluggin it can handle multiple data sources.

 

[Slavik]

Some stats from my forum....

MySQL index:

The same content indexed by ElasticSearch:

I have allocated 4gb to my JVM running ElasticSearch to ensure that it does not run out of memory to service my index. The 1.4gb of RAM used by the MyISAM index file has been reclaimed to increase the memory allocated to InnoDB on my SQL server.

We have allocated 1.5gb for around 2.1 M docs and find its running quite happily

 

[Deebs]

We have allocated 1.5gb for around 2.1 M docs and find its running quite happily

What size is your index on disk under ES? Have you done any tweaking of ES? (I spent a little time tweaking and playing with a 3 node cluster)

 

[Slavik]

What size is your index on disk under ES? Have you done any tweaking of ES? (I spent a little time tweaking and playing with a 3 node cluster)

Im having trouble with SSH tonight (ISP problems) so I cant get into the server to tell you.

Configuration wise we moved the ES directory to /var/elasticsearch and that was it. Oh and also increased the open file limit of the ES user.

 

[Deebs]

Im having trouble with SSH tonight (ISP problems) so I cant get into the server to tell you.

Configuration wise we moved the ES directory to /var/elasticsearch and that was it. Oh and also increased the open file limit of the ES user.

Ah ok, when you do gain access would be interested in knowing your disk usage.

 

[Slavik]

Ah ok, when you do gain access would be interested in knowing your disk usage.

Sure, how are you pulling your exact numbers from ES though? the service wrapper doesn't come with much documentation

 

[ManagerJosh]

IF you're running a shared server or even a VPS, there's no need for you to have Elastic Search. You're probably too small. However, those who have dedicated servers just for their forums do need this.

Price too expensive? 50.00 one time license + 10.00 renewal doesn't seem that bad. Consider total cost of ownership. You likely will need less hardware to do the search processing.

 

[ManagerJosh]

BTW, a virtual cookie to Karelke for catching it might be Elastic Search.

http://xenforo.com/community/thread...lk-about-version-1-2.23613/page-2#post-291656

It's the earliest reference I saw to Elastic Search back in 2011.

 

[Deebs]

Sure, how are you pulling your exact numbers from ES though? the service wrapper doesn't come with much documentation

Go install your new best friend: http://mobz.github.com/elasticsearch-head/

 

[Slavik]

Go install your new best friend: http://mobz.github.com/elasticsearch-head/

I love you.

 

[Deebs]

Here is a screen grab of my cluster... (the second node doesn't always stay active, only when I am playing..)