Deleted threads should return a 410 server message instead of 404

[JackieChun]

By definition, a 404 message means that the page currently cannot be found but may come back in the future. It should be used when you don't know if the condition is temporary or permanent.

The 410 error is much more specific. The HyperText Transfer Protocol (HTTP) 410 Gone client error response code indicates that access to the target resource is no longer available at the origin server and that this condition is likely to be permanent.

By design, XenForo threads, posts, members etc that are permanently deleted will not come back. Therefore, the 410 message is the appropriate server response.

Why does this matter?

The best practice for web admins is to parse through 404 URLs that your site returns on a regular basis (via Google Search Console or another method). Some 404'ed pages warrant measures, such as redirecting to another resource or URL reinstatement.

On the other hand, if the site returns a 410 on deleted content, there is nothing for the admin to fix, because the URL is, indeed, gone forever. It's a cemetery of URLs of sorts.

It would be a relatively small investment and effort to update the XenForo engine to return the correct message on deleted content in exchange for a significant result: helping keep the web organized and reducing work for web admins.

 

[Mendalla]

Hm. Not sure I have ever actually seen a 410 when browsing but based on the definition you give, it would make sense.

 

[Mr Lucky]

Hm. Not sure I have ever actually seen a 410 when browsing

That means it works as expected, ie they get dropped from indexing. In an ideal world browsing will not return 410s

 

[MySiteGuy]

Ultimately it really doesn't matter which you use, according to Google's John Mueller:

From our point of view, in the mid term/long term, a 404 is the same as a 410 for us. So in both of these cases, we drop those URLs from our index.

We generally reduce crawling a little bit of those URLs so that we don’t spend too much time crawling things that we know don’t exist.

The subtle difference here is that a 410 will sometimes fall out a little bit faster than a 404. But usually, we’re talking on the order of a couple days or so.

So if you’re just removing content naturally, then that’s perfectly fine to use either one. If you’ve already removed this content long ago, then it’s already not indexed so it doesn’t matter for us if you use a 404 or 410.

Before him, Matt Cutts said pretty much the same.

 

[K a M a L]

This would require keeping track of deleted records which means wasting resources for no real value

 

[cwe]

There is no way for XF to know if a page request was to a thread that was deleted permanently or to a made up / never existed page (maybe a typo in the URL).

 

[JackieChun]

This would require keeping track of deleted records which means wasting resources for no real value

There is no way for XF to know if a page request was to a thread that was deleted permanently or to a made up / never existed page (maybe a typo in the URL).

No need to keep track. XenForo knows which threads were deleted permanently because all threads are numbered. If a requested thread number is lower than the current highest number and it's not soft-deleted, then it's deleted permanently.

A URL that appears made up should return a 404.

 

[MySiteGuy]

No need to keep track. XenForo knows which threads were deleted permanently because all threads are numbered. If a requested thread number is lower than the current highest number and it's not soft-deleted, then it's deleted permanently.

A URL that appears made up should return a 404.

Not necessarily the case with imported forums.

 

[skicomau]

Ultimately it really doesn't matter which you use, according to Google's John Mueller:



Before him, Matt Cutts said pretty much the same.

They say that, but I've tested it using my CMS (not XF) and 410 definately disappear from index faster than 404's

However, it's high-hanging fruit, and the SEO/UX ROI is very very low. Too many webmasters either did not know or did not care about 410 management and so Google et al were left with having to treat 404's as proxy 410's.

In the end, consistent 404's get removed from index and so that's that. Submitting the XF sitemap XML into Google Search Console is the best way to ensure consistant 404's get removed in timely manner.

 

[MySiteGuy]

They say that, but I've tested it using my CMS (not XF) and 410 definately disappear from index faster than 404's

What part of Google saying "in the mid term/long term" wasn't clear?

 

[Kirby]

No need to keep track. XenForo knows which threads were deleted permanently because all threads are numbered.

Things often look easy on the surface while in reality they are not

Think about this request:
GET /threads/did-this-url-ever-exist.12313/page-5

Database table xf_thread does not have a record for for ID 123123.

This could have multiple reasons

1. A record existed in the past but the thread was deleted
2. A record existed in the past but the thread was merged with another thread
3. A record with such an ID never existed (because it was manually skipped or unused due to (past) MySQL configuration)

In case 1) a 410 might be semantically correct - if thread page 5 ever existed as well and if the title was correct.
If either of those 2 additional conditions were not true it should be a 404 as you said:

A URL that appears made up should return a 404.

How should XenForo detect if the URL is "made up" or not if it has no information if the requested thread page ever existed and if the title is correct?

In case 2) the most correct reply might be a 301 if the target still exists, otherwise a 410

Case 3) should give a 404.

Distinguishing all those cases (and maybe even more) would require extensive tracking - that’s really not worth it.

 

[MySiteGuy]

Plus a forum could have multiple import attempts, without properly resetting the ids before the last import, so the thread_ids are not mapped 1 to 1 to the source forum.