XF 1.4 Attachments in database or filesystem?

[Grover]

Hi,

If there ever comes a proper Blog solution for Xenforo, we can finally migrate from vBulletin 3.8. to XF.
In relation to this migration I have a question:

In my vB installation the attachments are stored in the database. I understood Xenforo keeps the attachments in a separate filesystem?

Question: for the migration process, is it advisable that I first put the vB attachments in the vB filesystem as well, and after that migrate to Xenforo?

---

Another question, which is more or less unrelated to XF specifically, but dare I ask it? :

My hostingprovider wants us to move from a Shared server to a much more expensive VPS. They say our vB platform uses too much resources:

- Overhead on database (3GB in size)
- Bad queries (they run for more than 10 secs)
- Indexes on tabels

Strange they complain now, since the last 5 years the amount of visitors/traffic to our forums has been reduced by almost 40% (!).

So, before migrating to XF (which is still impossible to do at the moment without a Blog solution), I would like to know if putting these attachments in the filesystem does any good to the performance? In other words, what are the pros and cons of putting attachments in the database or filesystem?

Thanks,

 

[Mike]

In terms of the import process, it really doesn't matter where the attachments are. They can be read in both instances. Having them in the database does make managing the import a little simpler (don't need to worry about paths).

 

[Digital Doctor]

without a Blog solution

What kinda Blog systems are you looking for ?
Multi-user Blog system ?
A Blog for your site (not for users) ?

My hostingprovider wants us to move from a Shared server to a much more expensive VPS. They say our vB platform uses too much resources:

Aint that what they always say ? It's called upselling

Get MattW to check your site before you go the VPS route.
Better yet, switch to Xenforo.
Why not use Wordpress as a blog solution and XenWord as the bridge ?

 

[Grover]

In terms of the import process, it really doesn't matter where the attachments are. They can be read in both instances. Having them in the database does make managing the import a little simpler (don't need to worry about paths).

Thank you.

In terms of performance benefits I was wondering:

So, before migrating to XF (which is still impossible to do at the moment without a Blog solution), I would like to know if putting these attachments in the filesystem does any good to the performance? In other words, what are the pros and cons of putting attachments in the database or filesystem?

Maybe someone has some insights in this general question. If there are (performance) benefits to putting it all into the filesystem, then it can maybe save me from the need to upgrade to a much more expensive VPS.

 

[Paul B]

Maybe someone has some insights in this general question. If there are (performance) benefits to putting it all into the filesystem, then it can maybe save me from the need to upgrade to a much more expensive VPS.

Surely you would be better off asking that on the vb support forum?

 

[Grover]

What kinda Blog systems are you looking for ?
Multi-user Blog system ?
A Blog for your site (not for users) ?

Bit off-topic, but I mentioned the blog issue myself to begin with. I am looking for exactly what vBulletin 3.8 offers. Nothing more and nothing less.

Aint that what they always say ? It's called upselling

Well... they can upsell to somebody else. I'm not buying into it.

Get MattW to check your site before you go the VPS route.
Better yet, switch to Xenforo.

Trust me, I wanted to switch to XF since the start, but really there was no point since it did not even offer UserAlbums for 4 years. And our User Albums are KEY to the income we generate from our forums. The Blogs are of a lesser value (at the moment) compared to the User Albums, but still they are highly valuable to us and our members. We wanted to offer Blogging for commercial members to generate even more income on our vB powered platform, but we are holding off since XF does not... yes repeat... blogs at all. So we are kinda stuck. For several vB sites out there, vB's Blog addition is a fantastic way to generate income for yourself.

Why not use Wordpress as a blog solution and XenWord as the bridge ?

I don't know XenWord. Does it change the Wordpress interface to a Xenforo one? What I mean is... Xenforo (the forum package to be precise, not so much the XFMG) has an outstanding interface: the most intuitive out there for forumusers. So why on earth should I as the boardowner offer my members a total different interface (Wordpress) for their blogging needs?! Never understood this vision. If my members are already used to the great XF interface, additional products should be offered exactly the same interface, obviously. So Wordpress is a clear no-go.

But I will not find a solution here, that became clear today. Thanks for your suggestions Digital D.

 

[Paul B]

So if I have this correct, you make a lot of money (by your own admission) from user albums and blogs but in the five years you have been posting here have not considered having those add-ons custom developed for your needs?

Instead you just want to spend something like $50 on an official XF add-on?

 

[Alpha1]

I always had the impression that it is best to have attachments in the file system if you have a large forum. But I read that @digitalpoint uses the database system so that seems to disprove my impression.

For big boards(IIRC @Grover has one too), would you advise database or file system?

 

[digitalpoint]

I store attachments and avatars in the database because I have multiple web servers that share nothing (no file system sharing) so there isn't a single point of failure.

If I had a single web sever, I'd store them in the file system. But it gets super complex when you are trying to sync a zillion attachment/avatar files across a bunch of servers in realtime. The load on the database server is negligible and was just a simple way to make them available on multiple web servers in realtime.

 

[Alpha1]

Why didn't you go for a virtualization of 1 web server and run that on multiple machines? That's the approach I am going for. I'm currently just running the virtualization on 1 high end machine, but will likely need another one in the near future.

 

[digitalpoint]

Why didn't you go for a virtualization of 1 web server and run that on multiple machines? That's the approach I am going for. I'm currently just running the virtualization on 1 high end machine, but will likely need another one in the near future.

I need to use *real* servers for some of the stuff I do. My setup is basically so that everything scales linearly simply by adding machines if you need it. Any server can go offline and any other server can pick up the slack. Trying to use virtualization for that stuff runs into some problems with database services, and since I don't use separate machines for separate tasks (remember... any server can replace any other server in my setup), it wouldn't work really well.

Virtualization doesn't work really great when you bind processes to specific physical CPUs (something else I do for performance).

Basically, I run Nginx, MySQL Cluster, Elastic Search, Memcached and PHP-FPM on all my servers. All servers are fully write capable for database queries, etc.

Also, the less complexity, the better as far as a I'm concerned. Like I got rid of my hardware load balancers and then also got rid of HAProxy. It turns out Nginx makes for a fantastic load balancer when you configure it right (even for things like MySQL connections).

Anyway... getting off topic. But yeah... in my setup, virtualization really wouldn't work great. And since I have a database cluster that's already capable of 30M queries/second, fully redundant, has 43.2TB of drive space available, 2TB of memory available, automatically sharding across physical servers, etc. Why not use it for attachments/avatars?

On top of it, I use Cloudflare to cache avatars in whatever data center is closest to the user... so at the end of the day, me storing avatars and attachments in the database probably yields somewhere around 1 query needed every 5 minutes. And when you database cluster can handle around 9 billion queries (30M/sec *300 seconds) in that same 5 minutes, adding 1 really doesn't matter much.

 

[Alpha1]

Interesting. Thanks for sharing. I have a feeling that there will be a point in time where I need to take another look at my approach and consider your approach. But that's still a long way off I think.