Lack of interest Option to 410 Deleted Content

[XenBurger]

Server status of 410 on a deleted thread makes the most sense for forum owners, and for optimal SEO.

I have learned from experience that if you leave a 404, that page can sit in googles index for months, if not years. Google defines 404 pages as "missing", which means they will keep it in the index and keep checking back. They give quite a long grace period.

Google defines 410 as "Gone", which tells them the page will not be returning, and its absence is intentional. This keeps forum SEO listings very clean, and speeds up the process of URL removal.

The option to define 404 or 410 (dropdown menu option in the Admin maybe?) for deleted threads, deleted profile pages/ deleted users, deleted tag pages, and anything else that can be intentionally deleted (on a page level) would be outstanding.

As it stands, there are no Addons for this, and I may need to pay my guy to create it.

By the way - very thankful the XenForo folks knew to at least 404 that content! vBulletin leaves it as a 301, which is really bad! It results in "Soft 404's" which Google actively penalizes for (unlike standard 404s).

Thanks!

-----------------------------------
The ability to 301 redirect a deleted thread would also be incredible. Sending it to an existing thread that has better content, or even if the forum owner wants to redirect it to the forum home page.

 

[Chris D]

I think this has been suggested before, but I will have to check.

404 is likely the best response code to return for deleted content - by which I mean content which is soft deleted. Of course with soft deleted content it is not necessarily the case that it is not returning, because you can undelete it. So, if there is a big distinction between how Google handles each one, a 410 isn't ideal - especially if this is a mistake and wasn't actually supposed to be deleted.

Of course there is then the scenario of a piece of content which previously existed and you have since permanently deleted it. Unfortunately there's no way to distinguish between content which has existed and now doesn't, vs content which has never existed.

On balance, a 404 is the only sensible code to return in all of these scenarios.

Overall, is there a big benefit to SEO either way? No. I don't believe so. I don't believe I've experienced Google keeping 404'd pages in their index for that length of time.

Here's a really good video on the subject and how Google actually does handle them. They really downplay any perceived benefits of it in terms of SEO:

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

 

[XenBurger]

a piece of content which previously existed and you have permanently deleted it.

This is the scenario 100% of the time for us. We don't undelete things. I honestly can't even think of a scenario where a forum owner would delete a thread and then restore it enough tims that a 404 would be considered the better option, overall. Permenant deletions are the vast majority of the time. Probably approaching 99% on most forums. This indicates a 410 is the better solution.

There are two options when deleting:
Soft delete (would generate a 404).
Permenantly delete (would generate a 410).

From the video you linked above: Matt Cutts: “If a page is gone and you think it’s temporary, go ahead and use a 404. If the page is gone ... and you know that that page is going to never come back, then go ahead and serve a 410.”

On balance, a 404 is the only sensible code to return in all of these scenarios.

Only on a forum where the vast majority of the time, threads are deleted and restored. Which I don't think ever happens. 404 is for content that is missing, and possibly to return. 410 is to tell Google "Its gone". Therefore 404 would be the least applicable option for most deleted threads.

Looks like Im going to have to have this created as a custom add-on. Can you give me any tips as to how we would do this in the code?

SEO is a huge part of running a forum. Like a massive, massive part. And there are so many extremely important elements that are missing from most forum software in this regard. Or misunderstandings (like I saw some of the developers telling people a robots.txt is just as good as a noindex tag in another thread). Proper page handling, handling of deleted content, Proper Title tags, proper robots disallows, etc etc all play a huge role in how well the site will rank overall. I hope input will be considered from users who have struggled with SEO on their forums like we have, for nearly 16 years.

Edited Post: For clarity.

 

[Chris D]

This is pretty much the scenario 100% of the time for us. We don't undelete things.

I honestly can't even think of a scenario where we would delete something and later decide to put it back live. So everything is permanently deleted on our forum.

I have a feeling this is the case for 95% of deletes on most forums too. Leaving the 404 option to be the extreme minority of cases.

Although this may be true, it an't be true 100% of the time, therefore 410 is not the correct code to give. I appreciate it is a rare case, though.

There are two options when deleting. Soft delete (would generate a 404). Permenantly delete (would generate a 410).

There's no way to distinguish between content which has been permanently deleted and content which has never existed. We don't track any content that once existed but has now been permanently deleted. So if I try to access thread 999 and it isn't in the database, we have no accurate way of discovering if it ever existed. Arguably we could try and detect if any threads exist with an ID greater than 999, and if they do it would suggest that 999 once existed but no longer does. This isn't infallible though.

Which I don't think ever happens. 404 is for content that is missing, and possibly to return. 410 is to tell Google "Its gone". Therefore 404 would be the least applicable option for threads which are deleted.

Well, arguably 200 or 301 would be the least applicable option for threads which are deleted. 410 may be more correct (in the case of permanently deleted content), but you cannot suggest that 404 is wrong. Remember the end result of a 404 is that the entry is removed from Google's index. A deleted thread as a 404 is only considered to "possibly return" until Google decides it's waited long enough, decided that it's unlikely and removes it from the index.

In the grand scheme of things, this apparently matters very little to Google, and makes very little impact on the end result of what happens to a deleted thread in Google's index so it likely isn't worth the added complexity in terms of prompting a moderator to decide which is which; especially bearing in mind most moderators aren't going to understand the difference between a 404 or 410.

Looks like Im going to have to have this created as a custom add-on. Can you give me any tips as to how we would do this in the code?

Looks like an add-on already exists. Hope that helps.
https://xenforo.com/community/resources/410-gone.4976/

 

[Alpha1]

It would make sense to add something like this in the core, so that requests for non-existing content result in 410 status.

 

[XenBurger]

I actually just migrated from vBulletin to XenForo on a forum that has about 700,000 posts.

We now have over 200,000 error 404's in our Google Search Console, that I am having to "Mark as Fixed" at 1,000 per day (the max that Google will show you per day).

So this is going to take me 167 days, minimum, just to clean up the mess that ensued after the migration. While 404's dont hurt a sites ranking, they do create a mess, and leave a *ton* of deleted content in Googles index for a long time. This means users will, for months to come, see pages ranked from your forum, click on them, and hit a dead page. This is considered "bad" by all metrics.

You can see examples of people reporting over 7 months of deleted content still sitting in Googles index when its just 404'd.

If these vBulletin pages were reporting as 410 I know they'd be gone in a jiffy, so its just a preference I have based on all basic logic that I can muster 404 missing / 410 gone.

Thanks for the help!

 

[Alpha1]

Are you using the vbulletin redirects? Why are there 200k 404s?

 

[XenBurger]

Well vBulletin was horrible with SEO. (this forum - thank you guys) is much better. I hired some Google gurus and they basically said: Tens of thousands of pages of "member profiles" and "tags" and "archives" were useless and even harmful for our rankings.

Basically all this stuff "dilutes" your truly useful content (threads), and makes it harder to find. It also tells Google: "Hey we have 167,000 pages and 90% of it is useless crap - so go ahead and rank us poorly". When we Googled our forums after 7 years, all we saw coming up was pages and pages of user profiles, and tags.

This brings me to my next request (sorry for being so annoying) - There really needs to be a way to set a noindex tag on these elements. A proper noindex tag. I know that you can disallow via Robots. We did this prior to releasing the new XenForo last week. But I guess vBulletin had told google that the /tags/ directory on our domain was spiderable already, so it ignored our Robots for Xenforo:

http://www.hairlosstalk.com/robots.txt (see /tags/ listed)
Google has already indexed tags on the new XenForo install: https://www.google.com/?gws_rd=ssl#q=site:https://www.hairlosstalk.com/interact/tags/*

(ugh)

So now we need to "noindex" our tag pages. Probably user profile pages to be safe. I did find another thread discussing how a "noindex" tag could be added to template files. My developer said templates are "shared" and if he "noindexed" the tag pages, it would also end up on other pages I don't want noindexed. Not sure if that's accurate.

In any case, vBulletin also had no option for "noindexing" things. So when we deletedt it, it was impossible to write proper 301 redirects for all of them, and we painfully accepted 167,000 404 errors.

Thanks so much for putting up with my comments guys - I apologize for the 'tude. Just stressed about all this SEO stuff. I have run this site for 16 years and we have been heavily penalized by Penguin, Panda, and other various Google animals. The people I hired indicated these things play an important role in our lost rankings over time. I am convinced as we clean this stuff up, and our rankings have begun to recover, that the websites which rank well, do so because they follow the minutia Google has set forth, to the "T".