XF 1.4 Struggling to import styles and install add-ons?

[Mindzipper]

There has to be something wrong here. Whenever I do anything that requires the server to do an import it takes a very long time.

for instance, I did a test, and installed 3 styles. it took over 15 minutes just to import the XML on all of them.

I installed Sportsbook as well, it too an hour to import that xml file.

this can't be normal? the server is very lean with only one website running on it. there are plenty of available resources, and when i monitor the DB server, there are very long delays between times transactions happen. almost 90% of that 15 minutes is spent with the db user account sleeping

 

[Paul B]

That sounds like a server configuration issue.

I have done several installations which took almost an hour to do the template rebuilds.

What are the server specs?

I presume it is VPS and not shared.

 

[Mindzipper]

it's not a VPS, it's an instance on Digital Ocean. So it's 'kind of' a VPS it has 2gb RAM, 2CPUs and a 40gb drive. This single website is all that's on it, and it's just in a test bed, there hasn't been more than 2 users on it at the same time. This is an instance we use just to stage websites, then move them off to their final home. that's why it surprised me, while it's certainly no powerhouse, it's got PLENTY of resources for a single site.

The db server is a dedicated machine that has almost disutingly overdone specs, and it's only running dbs for two sites right now (we're ramping up).

We have servers co-located throughout several datacenters, as well as many instances running on EC2 at Amazon. I've been doing servers and networking for 28 years. that's what is confusing me. the server load doesn't go over 1, and like I said I monitor the db server which is a real physical machine, and i monitor the db transactions and they're extremely low.

That's why it's so confusing, if it were eating up cycles i could track it down, nice the threads etc to compensate, but i don't find anything being smashed so i don't know what to look at right now.

I think i'll have to do a deep dive here and dig this out. I'm very familiar with the core developers at XF and I'm familiar with their 'previous' work ") i had been using it since it's incepetion. so i don't for a second think this is a software issue, i'm just kind of lost at what the problem is

 

[Sheratan]

The db server is a dedicated machine that has almost disutingly overdone specs, and it's only running dbs for two sites right now (we're ramping up).

So your xf database is remote database?

I use $5 DO and installing addon and style is only take 15-20 seconds.

 

[Mindzipper]

Yes it's a remote Db server, the DO instance is a 20 dollar version, but we use backend networking to manage it, so it's not coming out public IP space. it's actually using a monster server at ThePlanet that we co-locate (16cpus, gobs of ram etc) and that's what I expected was a few seconds to to import an XML, I mean I know the server chews cycles doing the calcs but then it's just doing quieries to add data to the db and changing templates etc. right?

i have also monitored the traffic and it's not stalling in transition either

 

[Luke F]

Network latency between web and db server?

 

[Mindzipper]

nah, like I said, i watched that too, it wasn't having network trouble, and nothing else network was having issues, so that's not it, although that thought did occur which is why i checked it. good thought

 

[Mike]

You make reference to importing the XML, but I'm not sure if you're referring to the first step (which is importing the XML itself) or the entire rebuild process (which is what would come up to tell you what each part is doing).

Regardless, both of these run significant numbers of queries, so I'm still going to suggest that network latency is a significant part of this. There may not be an explicit problem, but having millisecond-level delays will cause significant overhead when the processes need to run a large amount of queries.

 

[Mindzipper]

Well, while I understand what you're saying, we do networking for a living and monitoring is something we do constantly, i honestly don't think it's latency. we've got a very fat pipe and a great route between the two boxes, and i just don't think that's it. Now, I'm not guaranteeing you this, but i'll do some testing since you do bring it back up. You know this software better than do without question obviously since you're one of the developers, so I'll follow your advice. My thought is more than anything I'd expect hard server resources before latency, but I'll let you know the results of my testing.

There's always a chance i've got things cranked down too tight on the MySQL server itself I suppose. we do highly customize them. Well actually let me correct that, we're using mariadb since this is running on CentOS 7 (due to crazy server recommendations from VB). could that possibly be an issue? I mean the syntax shouldn't be a problem, but my guess is you're going to recomend MySQL

 

[Mike]

People use MariaDB without issue.

Really, the only other thing would be general performance optimization stuff. When it comes to MySQL, if you haven't optimized InnoDB, the defaults are going to be absolutely terrible: http://www.percona.com/blog/2013/09/20/innodb-performance-optimization-basics-updated/ This is doubly true when it comes to write-heavy processes (changing the write mode, adjusting how the flushes are called can have significant improvements)

You'd also want to ensure that PHP is optimized with an opcode cache, but that probably wouldn't cause such a significant issue here.

 

[Mindzipper]

opcode cache and a heavily customized MySQL are both done deals. In fact, this entire server was optimized for running forums. Though admittedly it was focused on VB becuase a month ago that's what I thought we were going to be using, and the db server will be managing two large forums. but I'll review the settings again and start there.

one thing to respond to, you said i didn't really mention where the slow down is. it's start to finish, from the first refresh of the page after uploading the XML file. i did this both picking the file locally and doing it from the server (although that shouldn't matter, the byte count is tiny anyway) but it seems to really slow down rebuilding phrases. that seems to be the one that takes the bulk of the time. And for refernce, it happens with all themes, as well as when I installed the resource manager and enhanced search.

to give you an idea, there's a mod called 'merge threads' (found here: https://xenforo.com/community/resources/wmtech-double-post-merge.833/" that's a very light mod . if you post twice in the same thread it merges them to one post to keep from having double posts.. in fact I believe all it really does is change some template code (would need to double check) it took 23 minutes to import the xml file and complete the install.

EDIT: after looking closer at the XML it's 61 lines in total, with 30 phrase statements in it)

 

[Paul B]

In fact, this entire server was optimized for running forums. Though admittedly it was focused on VB becuase a month ago that's what I thought we were going to be using

The optimising you did for VB definitely won't be the same as for XF so it would be worth revisiting that.

it seems to really slow down rebuilding phrases

How many styles, languages and add-ons do you have installed?

 

[Mindzipper]

ok i'll start over and tune the server. I guess I thought that my tweaks would translate. different realize the difference. my bad.

one language (default)

I have the following add-ons
rep system
sportsbook
merge threads (merge double posts)

i have 3 styles other than the default.

there won't be any additional addons or themes, i don't like bloating things. in fact, i don't like adding in a big betting addon, but the community demands it. This forum has been running since 2008 and the users have not only not slowed on the booking, they've increased activity (this is an MMA related forum)

 

[Mindzipper]

Wow, you sure were right Mike. this was all configured for MyISAM and not for configured for InnoDB

life's good, followed that link, but switched the config to do transactional databases and we're good to go. no problems now.

 

[Mindzipper]

Well, I was incorrect. We followed your advice but the problem still exists, I do not think this is a db issue. monitoring the db transactiosn shows it spends the lionshare of the time sleeping. monitoring the front end shows the load goes up, but only a little. it never went above 1.5 i believe.

importing the 3rd party addon yesterday took 45 minutes. i'm at a loss on what to look for. this is ONLY an issue doing imports. otherwise the front end and forum side work fast and flawlessly

we optimized the server based on your link too. but i will be honest, we are not InnoDB pros so this is a new direction

 

[Mindzipper]

here is an example. this cannot be normal is it?

this is the Private Message import. the forums are starting now, but it's been 23 minutes and it hasn't started counting.

Imported 22,967 items. (2 hours 17 minutes 42.67 seconds)

 

[Mindzipper]

I have a question about the import process (bringing over VB)

I seem to be able to bring over the private messages, and forums, but whenever I try to import the threads and posts, i get the error that MySQL has gone away. it's far under the timeout limit thought and I can see the client connected to the MariaDB server. So i'm not sure how to proceed.

if I go back to imports, I can continute it, but it does the same thing. when it does, I'm able to import other items, like right now it's importing the profile messages, but if I start importing the threads and posts again it will fail with that error again.

i have been running this import now since 11pm and it's not 2:53. i haven't even gotten started on the threads and posts, but my gut tells me I'm going to havce to start this whole thing over. is there any advice before doing so?

 

[Mindzipper]

I was right.

42 seconds after trying to start importing threads this is the result

Server Error[/quote]

Mysqli prepare error: MySQL server has gone away

1. Zend_Db_Statement_Mysqli->_prepare() in Zend/Db/Statement.php at line 115
2. Zend_Db_Statement->__construct() in Zend/Db/Adapter/Mysqli.php at line 381
3. Zend_Db_Adapter_Mysqli->prepare() in Zend/Db/Adapter/Abstract.php at line 478
4. Zend_Db_Adapter_Abstract->query() in Zend/Db/Adapter/Abstract.php at line 734
5. Zend_Db_Adapter_Abstract->fetchAll() in XenForo/Importer/vBulletin.php at line 2522
6. XenForo_Importer_vBulletin->stepThreads() in XenForo/Importer/Abstract.php at line 97
7. XenForo_Importer_Abstract->runStep() in XenForo/ControllerAdmin/Import.php at line 187
8. XenForo_ControllerAdmin_Import->_runStep() in XenForo/ControllerAdmin/Import.php at line 239
9. XenForo_ControllerAdmin_Import->_startStep() in XenForo/ControllerAdmin/Import.php at line 182
10. XenForo_ControllerAdmin_Import->actionStartStep() in XenForo/FrontController.php at line 347
11. XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
12. XenForo_FrontController->run() in /www-data/sand01/httpdocs/xen/admin.php at line 13

and see the attachment for where I am at. i cannot continute.

 

[Mindzipper]

actually, it might have something to do with this. there are 57 forums shows in that report. but looking at the nodee tree, i see: Showing 106 of 106 items

as well, when reviewing the PMs, it brought them over, but none of them are grouped, if there are 5 messages in a PM from back and forth? it posts each one separately, they are not showing relation to each other

i have a feeling the 'mysql has gone away' is caused by the server running out of memory but it's purely a guess.

 

[Paul B]

It certainly sounds like something is not configured correctly on the server.
It may be worth posting your config here, so some of our resident server experts can chime in: https://xenforo.com/community/forums/server-configuration-and-hosting.48/

when reviewing the PMs, it brought them over, but none of them are grouped, if there are 5 messages in a PM from back and forth

That's correct for imported PMs.