Implemented Allow searching of soft deleted threads/posts/profile posts

[digitalpoint]

We use search extensively to find spammers, people slinging the same crap from the same websites, etc. It was quite a shock when I realized that soft deleted threads, posts and profile posts aren't searchable by mods/admins. Digging a bit, I realized the DataWriters actually delete the soft deleted items from the search index completely.

I ended up changing that and now posts, threads and profile posts that are soft deleted are searchable by mods/admins (the system already filters them out of search results for users who can't see them normally).

But it got me thinking... Why does XF by default go out of it's way to make soft deleted items not searchable for mods/admins?

 

[Sadik B]

Probably because they thought that once deleted by an admin / mod, under normal circumstances, admins won't want to see them again.

It could be an ACP search option. Search Results to show soft deleted content for mods/admins (yes / no).

 

[digitalpoint]

I'd think if that was the reasoning and a mod/admin never wanted to see it again, they would hard delete it.

 

[MagnusB]

What I have found is that the only way to find soft deleted threads are by going through the moderator log or finding them by accident. It is annoying, to say the least.

 

[CTXMedia]

I hit this problem recently when a moderator accidentally used the Spam Cleaner on a long-term member.

After manually restoring all threads they'd started I had to use a database SET query to make the posts 'visible' again because they wouldn't come up in a search - I then had to rebuilt my ES search index too.

I agree with others - please keep soft-deleted items indexed/searchable, but only for mods/admins.

Cheers,
Shaun

 

[Wildcat Media]

I agree with others - please keep soft-deleted items indexed/searchable, but only for mods/admins.

This was one thing my staff asked me about--if they could search for deleted items. I would think it's a simple thing to add--searching on the "visible" attribute.

 

[digitalpoint]

It's not quite that easy because I don't think the search index stores the message_state (visible, deleted, moderated, etc.)

The thing I did (just block it from deleting things from the index on a soft delete) was fairly simple and just makes it work like vBulletin does... if you have access to view the post, you will see it in the results, if you don't, you won't.

 

[Wildcat Media]

It's not quite that easy because I don't think the search index stores the message_state (visible, deleted, moderated, etc.)

Good point--I'd considered adding that data to the search index, but blocking the removal of soft-deleted items from the index would work just as well IMHO. Seeing the presence of soft-deleted threads and posts is permissions-based already, so it's not a stretch to have them show up among all search results for mods and admins.

 

[Luxus]

Soft-deleted posts should land in a trash can within the yet to be developed mod-cp, as simple as that.

 

[Mendo]

I think it's a good idea to have some way of displaying the soft-deleted threads.

 

[Mouth]

The thing I did (just block it from deleting things from the index on a soft delete) was fairly simple[...]

Would you share that change with us, please?

 

[digitalpoint]

It's a little complicated to just zip up since I have a "Digital Point Search" addon that does a whole lot more than just allow searching of soft deleted posts... but here are some relevant parts...

For posts:

<?php

class DigitalPointSearch_DataWriter_DiscussionMessage extends XFCP_DigitalPointSearch_DataWriter_DiscussionMessage
{
	/**
	 * Updates the search index for this message.
	 * 
	 * Index regardless of soft deletion
	 * 
	 */
	protected function _indexForSearch()
	{
		$this->_insertOrUpdateSearchIndex();
	}
}

For threads:

<?php

class DigitalPointSearch_DataWriter_Discussion extends XFCP_DigitalPointSearch_DataWriter_Discussion
{
	/**
	 * Updates the search index for this message.
	 * 
	 * Index regardless of soft deletion
	 * 
	 * @param array $messages List of messages in this discussion. Does not include text!
	 */
	protected function _indexForSearch(array $messages)
	{
		if ($this->get('discussion_state') == 'visible')
		{
			if ($this->getExisting('discussion_state') != 'visible')
			{
				$this->_insertIntoSearchIndex($messages);
			}
			else if ($this->_needsSearchIndexUpdate())
			{
				$this->_updateSearchIndexTitle($messages);
			}
		}
	}
}

 

[Wildcat Media]

Even if there were a way to temporarily modify things so soft-deleted posts/threads would show up in searches would help us out--not having this ability (which we had in vB) is seriously hampering the staff's ability to research past issues with members who have lengthy histories with us. (We've been online 10 years now, so that can amount to a lot of history.) In fact, that is one of the top three complaints the staff has about XF.

Being on ES, I could easily regenerate the search indexes if a change were made.

 

[Wildcat Media]

It's a little complicated to just zip up since I have a "Digital Point Search" addon that does a whole lot more than just allow searching of soft deleted posts... but here are some relevant parts...

With all the buzz about XF 1.2, the staff is again asking that I make the deleted posts searchable. But given the code above, and the fact that we're using XFES, I have no idea where I'd even find similar code to look at in the system. I would think it's a simple matter to keep the deletions from getting removed from the index, but where to do this is what I'm stuck at.

Has anyone else tried this yet?

 

[Wildcat Media]

I did some digging to try to get to the bottom of this. These files are from XenForo 1.2.0.

First of all, in /library/XenForo/DataWriter/Discussion.php, at line 611 is the protected function _indexForSearch(). Here's the block of code:

    /**
    * Updates the search index for this discussion.
    */
    protected function _indexForSearch()
    {
        if ($this->get('discussion_state') == 'visible')
        {
            if ($this->getExisting('discussion_state') != 'visible')
            {
                $this->_insertIntoSearchIndex();
            }
            else if ($this->_needsSearchIndexUpdate())
            {
                $this->_updateSearchIndexTitle();
            }
        }
        else if ($this->isUpdate() && $this->get('discussion_state') != 'visible' && $this->getExisting('discussion_state') == 'visible')
        {
            $this->_deleteFromSearchIndex();
        }
    }

And in library/XenForo/DataWriter/DiscussionMessage.php, line 730, the same function:

    /**
    * Updates the search index for this message.
    */
    protected function _indexForSearch()
    {
        if ($this->get('message_state') == 'visible')
        {
            if ($this->getExisting('message_state') != 'visible' || $this->isChanged('message'))
            {
                $this->_insertOrUpdateSearchIndex();
            }
        }
        else if ($this->isUpdate() && $this->get('message_state') != 'visible' && $this->getExisting('message_state') == 'visible')
        {
            $this->_deleteFromSearchIndex();
        }
    }

I'm mentally exhausted as I type this (programming logic eludes me at the moment ), but would it be as simple as getting rid of both "else if" statements? Or, could we cleanly do away with anything having to do with message_state != 'visible' or == 'visible'?

In order to make all of the soft-deleted messages appear in search, then, would I need to rebuild the search indexes? It makes me wonder if I would have to modify something else in order to get the forum to rebuild the indexes but with the soft-deleted threads and posts intact.

My only concern after these edits: if we decide to keep these soft-deleted posts/threads in the index, will forum permissions prevent these from appearing for non-staff members? I am thinking they'll be OK, but wanted to check...

 

[DRE]

Make an addon for this plse lol

 

[Wildcat Media]

It's out of my league...I'm done coding for a living.

 

[digitalpoint]

I've made it into an addon. The addon also does many other interesting things with search like add some searchable content types (users, conversations & reports).

At some point after I get my site migrated to 1.2, I'll be wrapping that up into something installable.

 

[DRE]

I need this option too after encountering a problem with a deleted spammer brendabenson. You can't import if a user Conversations and Profile Posts if a conversation or soft-deleted profile post is still around from a spammer. http://xenforo.com/community/threads/xenforo-1-2-to-xenforo-1-2-importer-error.56494/

 

[digitalpoint]

Ended up rolling this functionality into the Search addon I released today...

https://marketplace.digitalpoint.com/digital-point-search.1772/item

Ability To Search Soft-Deleted Content
Mods/admins are able to search non-public content.

They also see a quick visual representation in search results for items that are soft-deleted.