When having issues with Google indexing your site, be sure and check your .htaccess file for excess bloat that may be restricting search engines from doing their jobs. It turns out Google Page Fetch was not working properly against my site. Recently I realized my site was not showing up in google search rankings. I had no idea why and admittedly, I was not doing the best job keeping up with the latest search engine trends needed for SEO. Since I am John Zeren and my site is on johnzeren.com, it seemed a given that I would always remain in the first page of rankings. However, Google is continuously changing how it gauges SEO and how it determines if sites are valid.
To figure this one out, I had to start at the beginning. I checked via several SEO tools. It seems the content was relevant. I made sure I had a sitemap and it seemed to be working. Eventually, I checked the Google Search Console. Once I had my site registered, I went to try to validate a page and it was not working. I was getting 404s back from Google. When I tried submitting my sitemap (which was rendering just fine in the right format), I was also getting errors. Specifically I was receiving a message that Google Page Fetch was not working properly.
The Root Cause
Google Page fetch – “Failed: Not found (404)"I checked as many things around access as I could think of. The pages and sitemap all seemed to be rendering fine. My redirects seemed ok and I even added a very open robots.txt (which I didn’t have) based on best practices from Yoast. Eventually, as with most problems, I searched the exact error message and got a hit. This article from the WordPress forum pointed me in the right direction: https://wordpress.org/support/topic/google-page-fetch-failed-not-found-404/. The problem was with my .htaccess file.
The Solution
After checking my .htaccess file, all seemed fine (after all, it was working only a few months ago). However, I noticed autogenerated lines in the file. These lines were not necessarily needed (although they may have been recommended at one time). Once I removed the excess bloat, Google could immediately index my site once again. Since the site is running on WordPress site, most everything that is needed can be handled through configs or hosting settings. The only items that are necessary are the ones that are needed by WordPress:
#BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
#End WordPress
This is a good reminder to do cleanup and maintenance on your site from time to time. Also, it makes good sense to keep up with the latest trends from Google if you want to keep up with SEO. I’m not exactly making millions on my blog, but I would expect to be at the top of my own search rankings!