Interview with Shawn Hogan (Digital Point)

Slavik said:
How do you handle your post edit table by the way? Just curious more than anything else. Eg ive seen 2 monthods used in current addons, the one I use makes a new column in the xf_post table... the other adds a new table alltogether.
Click to expand...
I take back what I previously said about editing the post table. I started looking at it a bit closer this morning and I'm going do it with separate tables. The necessary JOIN overhead is really low since the query that the JOIN is added to only returns 20 results (20 posts per page in a thread). That overhead is so low that the overhead of the other option (adding 4 bytes [an INT column] to 20M+ records even if they weren't edited) is greater. No point in adding ~80MB to the post table, which is already huge. If it only "cost" 4 bytes per EDITED record, then sure... but since it's 4 bytes regardless...
 
digitalpoint said:
I take back what I previously said about editing the post table. I started looking at it a bit closer this morning and I'm going do it with separate tables. The necessary JOIN overhead is really low since the query that the JOIN is added to only returns 20 results (20 posts per page in a thread). That overhead is so low that the overhead of the other option (adding 4 bytes [an INT column] to 20M+ records even if they weren't edited) is greater. No point in adding ~80MB to the post table, which is already huge. If it only "cost" 4 bytes per EDITED record, then sure... but since it's 4 bytes regardless...
Click to expand...

I had something (quite important) to ask you earlier but its completely gone now.


How irritating.
 
Lol... must have been *super* important. :p

If you were going to ask if I made my setup so users can edit their thread titles, yeah... I just made that addon real quick.
 
I would assume there's addons that do it already... no?

It's actually a *super* simple addon to do it... only one controller method is extended and one template edit (a little annoying you have to do a template edit, but no hooks are available, so...)
 
Slavik said:
The only one with it comes bundled with a load of other functionality we already have being provided by other addons, so as a standalone, no, its not available.
Click to expand...
Well, I'm not going to bundle it up because I'm too lazy, but it's super easy (I know you are smart enough to know what this is and how to make it work.. hah)

PHP: 
<?php

class DigitalPointEditThreadTitle_ControllerPublic_Post extends XFCP_DigitalPointEditThreadTitle_ControllerPublic_Post
{
	public function actionSave()
	{
		$response = parent::actionSave();
		
		$postId = $this->_input->filterSingle('post_id', XenForo_Input::UINT);
		
		$ftpHelper = $this->getHelper('ForumThreadPost');
		list($post, $thread, $forum) = $ftpHelper->assertPostValidAndViewable($postId);
		
		$this->_assertCanEditPost($post, $thread, $forum);
		
		$title = $this->_input->filterSingle('title', XenForo_Input::STRING);
		
		// Double check that title was even submitted in form (I dunno, maybe some other obscure template uses this save method, so just to be safe...)
		if ($thread['first_post_id'] == $post['post_id'] && isset($_POST['title']) && $title != $thread['title'])
		{
			$dw = XenForo_DataWriter::create('XenForo_DataWriter_Discussion_Thread');
			$dw->setExistingData($thread['thread_id']);
			$dw->set('title', $title);
			$dw->save();
		}

		return $response;
	}
}

Then just need to add the "title" field to the post_edit template if $post.post_id == $thread.first_post_id
 
hm, there's one check missing:p

if $thread['title'] != $_POST['title']

if it's the same, you don't need to run the dw code
i thought you're a perfectionist?:P
*scnr*
 
digitalpoint said:
Lol... well if there's ever a proper hook in post_edit, I'll probably wrap it up and release it... but don't really want to mess with trying to support people who want to know why it stopped worked every upgrade when they overwrite their template. :)
Click to expand...
never understood, why you're not using tms:P
 
ragtek said:
hm, there's one check missing:p

if $thread['title'] != $_POST['title']

if it's the same, you don't need to run the dw code:p
*scnr*
Click to expand...
True... good point. Edited.
ragtek said:
never understood, why you're not using tms:p
Click to expand...
I dunno... mostly because I get scared installing stuff from 3rd party developers. :) So far the only addon I have installed that wasn't made by me was the XenForo Enhanced Search one.

I'll probably install it at some point... just always nervous about injecting other people's code into my setup. :)
 
I've been back at the import system again... cleaned some stuff up, fixed some formatting issues with it and added a bunch more things that it imports... reported posts/PMs/visitor messages into the XF report system, and a bunch of stuff that is specific to our setup (post edit history, watched forums, marketplace items, transactions, auctions/bids, etc.)

All in all, it generates 66 txt files (1 for each table) that are ready to be loaded into XF just as if you were restoring a backup.

It takes ~33 minutes to generate it all (18M+ posts, 1.5M threads, 650k+ users, etc.), which amounts to 74.5M records to be imported into XF (it takes about 25 minutes to import it all into XF)... then you are good to go (other than rebuilding search index, which takes about 18 minutes).

Code: 
#################################################
### Digital Point A.R.F.I. vB4 -> XF Importer ###
#################################################
 
1 - Posts
2 - Users
3 - Threads
4 - Private Messages
5 - Avatars
6 - Attachments
7 - Forums
8 - Miscellaneous
9 - Everything
 
Make Selection [1-9]: 9
 
exporting posts...
          [forking process for posts]
          post IPs (18,007,231 records in 537.21s)...
          likes (405,330 records in 4.53s)...
          post edit history (269,767 records in 40.05s)...
          reported posts (184,715 records in 84.92s)...
          reported post comments (211,405 records in 61.45s)...
          reported profile posts (713 records in 0.17s)...
          reported profile post comments (788 records in 0.36s)...
          reported private messages (5,962 records in 1.52s)...
          reported private message comments (6,825 records in 2.29s)...
Total time for posts: 732.62s
 
exporting users...
          [forking process for users & user authenticate]
          user external auth (17,694 records in 2.26s)...
          user follow (210,683 records in 0.57s)...
          user ignored (10,736 records in 0.20s)...
          user notes (30,649 records in 3.17s)...
          user options (654,462 records in 5.19s)...
          user profile (654,462 records in 65.50s)...
          user privacy (654,462 records in 5.97s)...
          user warnings (410,608 records in 60.84s)...
          user bans (23,963 records in 7.55s)...
          warning action triggers (23,963 records in 3.39s)...
          user field (187,730 records in 108.30s)...
          user registration IPs (654,462 records in 8.03s)...
          user account confirmation IPs (213,661 records in 5.93s)...
          user change log (464,391 records in 9.22s)...
          user change log IPs (337,774 records in 3.78s)...
          profile posts (54,412 records in 1.32s)...
          profile post IPs (49,840 records in 1.54s)...
          user upgrades active (483 records in 0.06s)...
          user upgrades expired (1,966 records in 0.04s)...
          user upgrades log (7,207 records in 0.59s)...
Total time for users: 293.44s
*** Don't forget to rebuild user cache ***
 
 
exporting threads...
          [forking process for threads]
          watched threads (10,838,001 records in 30.97s)...
          thread user post (12,752,077 records in 105.70s)...
          polls (240,405 records in 13.68s)...
          thread read (261,856 records in 1.67s)...
Total time for threads: 152.02s
*** Don't forget to rebuild poll cache ***
 
 
exporting private messages...
          [forking process for conversation master]
          conversation message (449,871 records in 264.23s)...
          conversation recipient (679,335 records in 18.66s)...
          conversation user (679,335 records in 13.00s)...
          conversation IPs (439,731 records in 7.65s)...
Total time for private messages: 303.66s
 
performing avatar diff...
        1 users with new avatars
733016
Total time for avatars: 14.07s
 
performing attachment diff...
        1 new attachments
98836
        updating attachment view count
Total time for attachments: 1.52s
 
exporting forums...
          nodes (147 records in 0.23s)...
          forums (140 records in 0.15s)...
          link forums (10 records in 0.15s)...
          forum read (10,713 records in 0.23s)...
          watched forums (3,164 records in 0.15s)...
Total time for forums: 0.90s
 
exporting micellaneous stuff...
          deletion log (996,651 records in 11.44s)...
          moderator log (894,108 records in 63.27s)...
          daily stats (23,263 records in 99.99s)...
          directory categories (11 records in 0.15s)...
          directory entries (119 records in 0.13s)...
          marketplace (224,965 records in 4.46s)...
          marketplace bids (84,538 records in 1.47s)...
          marketplace bid IPs (71,961 records in 6.79s)...
          marketplace transactions (6,044 records in 0.81s)...
          advertising transactions (4,577 records in 0.53s)...
          ebook transactions (478 records in 1.33s)...
          ebooks (804 records in 0.56s)...
          watched items (478 records in 0.14s)...
          marketplace licenses (447 records in 0.15s)...
          marketplace ratings (68 records in 0.13s)...
          marketplace files (804 records in 0.15s)...
          marketplace item attachments (804 records in 18.09s)...
          marketplace item attachments data (804 records in 0.76s)...
          marketplace item file data (804 records in 67.80s)...
Total time for micellaneous stuff: 278.16s
 
          [waiting on process for users & user authenticate] DONE.
          [waiting on process for conversation master] DONE.
          [waiting on process for threads] DONE.
          [waiting on process for posts] DONE.
 
    Grand Total For Everything: 33 minutes, 3 seconds (74,587,823 records)
 
digitalpoint said:
I've been back at the import system again... cleaned some stuff up, fixed some formatting issues with it and added a bunch more things that it imports... reported posts/PMs/visitor messages into the XF report system, and a bunch of stuff that is specific to our setup (post edit history, watched forums, marketplace items, transactions, auctions/bids, etc.)

All in all, it generates 66 txt files (1 for each table) that are ready to be loaded into XF just as if you were restoring a backup.

It takes ~33 minutes to generate it all (18M+ posts, 1.5M threads, 650k+ users, etc.), which amounts to 74.5M records to be imported into XF (it takes about 25 minutes to import it all into XF)... then you are good to go (other than rebuilding search index, which takes about 18 minutes).
Click to expand...


Very impressive.
 
digitalpoint said:
Code: 
#################################################
### Digital Point A.R.F.I. vB4 -> XF Importer ###
#################################################
 
1 - Posts
2 - Users
3 - Threads
4 - Private Messages
5 - Avatars
6 - Attachments
7 - Forums
8 - Miscellaneous
9 - Everything
 
[/quote]
Click to expand...

very nice man, now just need to release this importer for others to use.. :) ;) :P i would have loved to have had this importer before haha.
 
vVv said:
very nice man, now just need to release this importer for others to use.. :) ;) :p i would have loved to have had this importer before haha.
Click to expand...
Yep... after I'm done with it, hopefully I can find someone willing to take it over and turn it into something a little more generic so others could use it.
 
Top Bottom