XF 1.5 Not Getting Any Crawlers On Site

[Winterstride]

Hi, I barely know what I'm doing.

Our website stopped getting crawlers at some point and no longer seems to be indexed all that well on the Googles. We realized that we didn't have something called an SSL Certificate, which my admins tell me was very important to have for this sorta thing, so we got it and... still nothing!

From my research it seems we may just be in the "wait it out" phase and Google may get on top of it sometime down the line but I thought hey, in the meantime, why not ask around and see what other things may be contributing to our lack of Google indexing? Couldn't hurt, right?

Our site here if you need to take a look for some reason: www.sakugacity.com

Thank you very much for your time.

 

[Winterstride]

there should be a file named .htacccess in public_html folder. your ftp client might not show it by default as some apps hide files starting with . at the start of them. try to find this particular file and paste its content here.

You are correct. I found it, here's the contents:

Spoiler

RewriteCond %{HTTP_USER_AGENT} (spider|crawl|bot|slurp) [NC]
RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$
RewriteRule . - [F,L]
ErrorDocument 401 default
ErrorDocument 403 default
ErrorDocument 404 default
ErrorDocument 405 default
ErrorDocument 406 default
ErrorDocument 500 default
ErrorDocument 501 default
ErrorDocument 503 default

#    Mod_security can interfere with uploading of content such as attachments. If you
#    cannot attach files, remove the "#" from the lines below.
<IfModule mod_security.c>
    SecFilterEngine Off
    SecFilterScanPOST Off
</IfModule>

<IfModule mod_rewrite.c>
    RewriteEngine On

    #    If you are having problems with the rewrite rules, remove the "#" from the
    #    line that begins "RewriteBase" below. You will also have to change the path
    #    of the rewrite to reflect the path to your XenForo installation.

    #RewriteBase /xenforo

    #    This line may be needed to enable WebDAV editing with PHP as a CGI.

    #RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

    RewriteCond %{SERVER_PORT} 80
    RewriteRule ^(.*)$ https://%{HTTP_HOST}/$1 [R,L]

    RewriteCond %{HTTP_HOST} ^[^.]+\.[^.]+$
    RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
    RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$
    RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [L,R=301]
    
    RewriteCond %{REQUEST_FILENAME} -f [OR]
    RewriteCond %{REQUEST_FILENAME} -l [OR]
    RewriteCond %{REQUEST_FILENAME} -d
    RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
    RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$
    RewriteRule ^.*$ - [NC,L]
    RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
    RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$
    RewriteRule ^(data/|js/|styles/|install/|favicon\.ico|crossdomain\.xml|robots\.txt) - [NC,L]
    RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
    RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$
    RewriteRule ^.*$ index.php [NC,L]
</IfModule>

<IfModule mod_deflate.c>
  # Compress HTML, CSS, JavaScript, Text, XML and fonts
  AddOutputFilterByType DEFLATE application/javascript
  AddOutputFilterByType DEFLATE application/rss+xml
  AddOutputFilterByType DEFLATE application/vnd.ms-fontobject
  AddOutputFilterByType DEFLATE application/x-font
  AddOutputFilterByType DEFLATE application/x-font-opentype
  AddOutputFilterByType DEFLATE application/x-font-otf
  AddOutputFilterByType DEFLATE application/x-font-truetype
  AddOutputFilterByType DEFLATE application/x-font-ttf
  AddOutputFilterByType DEFLATE application/x-javascript
  AddOutputFilterByType DEFLATE application/xhtml+xml
  AddOutputFilterByType DEFLATE application/xml
  AddOutputFilterByType DEFLATE font/opentype
  AddOutputFilterByType DEFLATE font/otf
  AddOutputFilterByType DEFLATE font/ttf
  AddOutputFilterByType DEFLATE image/svg+xml
  AddOutputFilterByType DEFLATE image/x-icon
  AddOutputFilterByType DEFLATE text/css
  AddOutputFilterByType DEFLATE text/html
  AddOutputFilterByType DEFLATE text/javascript
  AddOutputFilterByType DEFLATE text/plain
  AddOutputFilterByType DEFLATE text/xml

  # Remove browser bugs (only needed for really old browsers)
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  BrowserMatch ^Mozilla/4\.0[678] no-gzip
  BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
  Header append Vary User-Agent
</IfModule>

<ifModule mod_expires.c>
     ExpiresActive On

     ############################################
     ## Add default Expires header
     ## http://developer.yahoo.com/performance/rules.html#expires

     <FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
     ExpiresDefault "access plus 1 year"
     </FilesMatch>
</ifModule>

 

[Chromaniac]

ok. this is kind of out of my area of expertise. there are a lot of statements in that file that i do not understand.
i see some statements that seems to be required by cpanel. some for https certificate from comodo. and a lot of optimizations that could be causing the problems you are facing.

backup the .htaccess file and create a new one with these options... i think these might be enough to debug and support xenforo requirements.

<IfModule mod_rewrite.c>
RewriteEngine On

RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$

RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^.*$ - [NC,L]
RewriteRule ^(data/|js/|styles/|install/|favicon\.ico|crossdomain\.xml|robots\.txt) - [NC,L]
RewriteRule ^.*$ index.php [NC,L]
</IfModule>

 

[Winterstride]

@OperaManiac

Just wanted to report back that while we didn't end up using the .htacces code you generously supplied here we did use your help to diagnose the issue. Google is now reporting no issues with crawling our site and I'm waiting for a full-fledged proper index as we speak.

Thank you (and everyone else) very much for your repeated attempts to assist me. I really appreciate your kind efforts.