Traffic Got A Hit Last Week; Down ~40% . Am I Doing Anything Wrong?

TheBigK

TheBigK

Well-known member
I noticed that around the beginning of the last week; our traffic started declining and it's currently at about 40% of the initial. Here are the things I did that might have led to the decline in traffic -

Site Setup: Wordpress (on main domain) & Xenforo on /community/

PS: Traffic is declining on the forums, not to the blog.
  1. I removed a large number of useless tags [ about 10k of them, I guess ].
  2. I updated my Robots.txt from allow everything to: http://www.crazyengineers.com/robots.txt
  3. I had added widget framework sidebar to all the pages of the site in the last week of August 2012.
  4. I had two large rectangles placed below the title for a long time (while the traffic was improving), while having site welcome text on the top. I clearly marked the ads as 'Ads By Google'. So the ads were in the 'middle' of the text; while still being above the fold.
  5. I added sitemap add-on to Xenforo.
What do you think could have caused the traffic to go down? What should I do to recover from it?
 
Hmm, yeah I guess. I suppose the danger is content gets indexed twice otherwise...

Do you use Google Webmaster Tools?

This may be worth doing. It can flag up when Google has problems indexing your site so it might be worth taking a look.
 
Chris Deeming said:
Hmm, yeah I guess. I suppose the danger is content gets indexed twice otherwise...

Do you use Google Webmaster Tools?

This may be worth doing. It can flag up when Google has problems indexing your site so it might be worth taking a look.
Click to expand...
So should I keep it or not? :confused:

I do use Google Webmaster tools. It threw following messages at me-

1. Large increase in the number of 404 (most of them are the removed tags, of course).
2. Sudden decrease traffic for top URL. Surprisingly, google still shows the URL on #1 position for the keywords. I've received the same message for the same URL in the past, but the overall forum traffic never went down.

I must also mention that our site has been very reputable and has been linked back by Big Folks! We've not done anything illegal (that Google strictly says no to) and have grown only by focusing on content. I wonder what's going wrong with the site.

Is it possible that Google's bit scared because it now finds 'new pages' on the old URLs (because of the addition of sidebar)?
 
Another possibility is Google algorithm updates, there's been a few recently: http://www.seomoz.org/google-algorithm-change

Then again, they're happening all the time and often they don't cause a massive impact.

The removal of tags might be a problem... Are you sure those tags weren't getting you a lot of traffic?

There was a point in time where we were getting 200 hits a day on the search term "woman dies having sex with a horse" ... don't ask ... but what I'm saying is, although you may have thought those tags were redundant actually they may have been generating traffic somehow.
 
No, the tags were removed from our wordpress blog which is tracked separately. Actually the traffic to blog is steady but the traffic to the forum has gone down. I checked SEOMOZ already, but it does not list any update during the time our traffic went down. Not a single algorithm update affected us so far because we've been very strict about the content. I'm quite sure that it's not the regular algorithm update.

The possibilities I think have been listed above. Is there any way of discovering what exactly went wrong?
 
/posts/ is nothing to worry about, it reduces double entries in Google, which is a good thing. Just check this out: http://xenforo.com/community/posts/414124/

That redirects to: http://xenforo.com/community/thread...nd-its-impact-on-seo.36283/page-2#post-414124

If you disallowed /threads/ you should worry ;)

I also disallow /register/ and /login and /search, cause all of those pages are not really that interesting too Google.

Did the removal of those tags cause allot of dead URL's? I am not sure if Google penalize you for that, but they are not happy about it. I see now they even warn you when the increase. However, since they have no good way of bulk removing URL's. Using the sidebar to promote your content is perfectly acceptable, it shouldn't impact anything, unless you go crazy with internal links.

If your SERP is unchanged, do you see a reduction of impressions? Usually a decline in traffic indicates your SERP is changed, but if it isn't my next guess is that there is less people searching for it.
 
You haven't had a big/influential site or blog remove their link to you have you? Have you checked your sources and is the drop directly attributable to Google specific search traffic?
 
Also have you checked the sitemap add-on is working correctly, that Google is collecting the file data, and that the content covers as many of your pages as you would expect? (ISTR reading that the add-on only accounted for max. 50,000 threads/post - although I could be wrong on that - I can't check it from my phone!).

Cheers,
Shaun :D
 
MagnusB said:
Did the removal of those tags cause allot of dead URL's? I am not sure if Google penalize you for that, but they are not happy about it. I see now they even warn you when the increase. However, since they have no good way of bulk removing URL's. Using the sidebar to promote your content is perfectly acceptable, it shouldn't impact anything, unless you go crazy with internal links.

If your SERP is unchanged, do you see a reduction of impressions? Usually a decline in traffic indicates your SERP is changed, but if it isn't my next guess is that there is less people searching for it.
Click to expand...

1. Removal of URLs caused lot of dead URLs. It was a mistake that I shouldn't have done, I guess. The sidebar has links to the content on the site, but I'm not sure if it's 'too' much.
2. Didn't get your point about SERPs. What am I supposed to check in GWT? Bit confused, sorry.

Clickfinity said:
You haven't had a big/influential site or blog remove their link to you have you? Have you checked your sources and is the drop directly attributable to Google specific search traffic?
Click to expand...
Nope, they're still in place. In fact, the traffic was 'steady', even increased 40% in the week before. It started declining and went down below the 'regular traffic'. Our news articles go into Google News and cause traffic spikes to the blog. My biggest concern is about the traffic to the forum.
 
Clickfinity said:
Also have you checked the sitemap add-on is working correctly, that Google is collecting the file data, and that the content covers as many of your pages as you would expect? (ISTR reading that the add-on only accounted for max. 50,000 threads/post - although I could be wrong on that - I can't check it from my phone!).

Cheers,
Shaun :D
Click to expand...
Google Sitemap worked fine and it showed 99.5% links as 'indexed'. I've removed now because I think it just doesn't add any value. Google will list the site anyway.
 
TheBigK said:
1. Removal of URLs caused lot of dead URLs. It was a mistake that I shouldn't have done, I guess. The sidebar has links to the content on the site, but I'm not sure if it's 'too' much.
2. Didn't get your point about SERPs. What am I supposed to check in GWT? Bit confused, sorry.


Nope, they're still in place. In fact, the traffic was 'steady', even increased 40% in the week before. It started declining and went down below the 'regular traffic'. Our news articles go into Google News and cause traffic spikes to the blog. My biggest concern is about the traffic to the forum.
Click to expand...
Is your CTR changed? My point is that if your SERP is unchanged, a loss of traffic from Google might indicate the keyword is less popular. From what I saw, your sidebar should not be harmful in anyway, and the scale you are suggesting are far beyond what a single text block should cause, and besides, xF use well defined sections, so Google should be more than able to separate your content from your sidebars.

As for your sitemap, you said you removed it, did you delete it from GWT as well? You might want to add a line to your robots.txt disallowing them to access it as well, otherwise Google might use a stale sitemap for your site.
 
Sorry to hear that. I would suggest you the following
1. first of all you should 301 redirect all 404 pages in the forum to the forum homepage, this would resolve the 404 errors issue.
2. you should disallow /posts/ since /posts/ and /threads/ are duplicate content, I would suggest just using xenforo.com/robots.txt file, it's a good one.
3. My forum which uses vBulletin was affected from the last Panda update, and the daily traffic dropped from 27k-30k to 20k-24k, but it recovered lately (2 weeks ago).. this also can be the cause...
4. Do you use Google Analytics? if Yes, just check if the drop was caused by single keywords drop, or all keywords dropped..
 
MagnusB said:
Is your CTR changed? My point is that if your SERP is unchanged, a loss of traffic from Google might indicate the keyword is less popular. From what I saw, your sidebar should not be harmful in anyway, and the scale you are suggesting are far beyond what a single text block should cause, and besides, xF use well defined sections, so Google should be more than able to separate your content from your sidebars.

As for your sitemap, you said you removed it, did you delete it from GWT as well? You might want to add a line to your robots.txt disallowing them to access it as well, otherwise Google might use a stale sitemap for your site.
Click to expand...
I must mention, I really appreciate your responses and the help all XF community members are offering. Thanks a lot folks! You are rockstars!

1. For some of the keywords, it's improved a lot while for others, it's gone down a bit. I can't arrive at any conclusion from them webmaster tools.
2. I removed the sitemap from GWT as well. Might add up a line to robots.
 
Khaleal said:
Sorry to hear that. I would suggest you the following
1. first of all you should 301 redirect all 404 pages in the forum to the forum homepage, this would resolve the 404 errors issue.
2. you should disallow /posts/ since /posts/ and /threads/ are duplicate content, I would suggest just using xenforo.com/robots.txt file, it's a good one.
3. My forum which uses vBulletin was affected from the last Panda update, and the daily traffic dropped from 27k-30k to 20k-24k, but it recovered lately (2 weeks ago).. this also can be the cause...
4. Do you use Google Analytics? if Yes, just check if the drop was caused by single keywords drop, or all keywords dropped..
Click to expand...
Thank you for your response.

1. The site has already indexed all the new URLs and old ones have been removed. The 301 redirects are in place. It's been several months before we migrated, so that's not an issue.
2. Thanks, I'll update the robots.txt.
3. I'm sure it's not Panda for us. How long did it take you to recover and what steps did you take?
4. Yes, I use GA. Good thing to discover. Will start investigating.
 
TheBigK said:
Thank you for your response.

1. The site has already indexed all the new URLs and old ones have been removed. The 301 redirects are in place. It's been several months before we migrated, so that's not an issue.
2. Thanks, I'll update the robots.txt.
3. I'm sure it's not Panda for us. How long did it take you to recover and what steps did you take?
4. Yes, I use GA. Good thing to discover. Will start investigating.
Click to expand...
You are welcome bro :)
1. I do not mean to redirect old URLs to xF's URLs, I mean, the tags URLs that were deleted now generates 404 pages.. and by using a htacces rule you can just 301 redirect these broken tags URLs to the forum homepage instead of showing 404 error messages.. this would resolve the deleted tags issue completely..
2. I see that MagnusB suggestions are also good, consider disallowing these pages..
3. It took about one week.. In this week I've edited the h1 tags to be bigger (xx-large size), and also added a plugin which generates search keywords in the thread, those keywords are the actual keywords that were searched to find that thread..
4. Also check in the webmasters tools, check your keyword impressions vs actual clicks if they dropped.
Good luck :)
 
1.
Khaleal said:
You are welcome bro :)
1. I do not mean to redirect old URLs to xF's URLs, I mean, the tags URLs that were deleted now generates 404 pages.. and by using a htacces rule you can just 301 redirect these broken tags URLs to the forum homepage instead of showing 404 error messages.. this would resolve the deleted tags issue completely..
Click to expand...
So these URLs were http://www.crazyengineers.com/tag/<tag> , and I had over 10k of them that I removed. I decided only to keep the best ones. How do I redirect only the ones that were removed? I'm not experienced in setting up redirects. Plus the current set of tags list well in Google, so I'd keep them intact. Can we automatically make the 404s to redirect to the homepage?

2. I see that MagnusB suggestions are also good, consider disallowing these pages..

K: Not sure how can I disallow the 404s. I'll have to look for a wordpress plugin to do that.

3. It took about one week.. In this week I've edited the h1 tags to be bigger (xx-large size), and also added a plugin which generates search keywords in the thread, those keywords are the actual keywords that were searched to find that thread..

K: Interesting that you got it back in just one week. I think we've the font size properly set (we're using XF default). I'm considering addition of XenTags plugin to link our content. But the task of adding keywords to our existing contents is going to be HUGE.

4. Also check in the webmasters tools, check your keyword impressions vs actual clicks if they dropped.
Good luck :)

K: This is how it is -

Screen Shot 2012-09-17 at 3.56.01 PM.webp



The traffic decline (as indicated by Analytics) started on 5th of September, after attaining a peak on 4th. It was unusual traffic decline.
 
Back
Top Bottom