I recently did some search engine optimization work for a medium-size company. They didn’t realize it, but their website was quickly dropping off the search engine result pages. Not dropping down in the results, dropping out completely. 98% of the pages left in the results, were stale and no longer existed on their site.

The first place I looked was the robots.txt file. This problem was easily located. There were only two lines in the file and it looked like this:

User-agent: *
Disallow: /

Their robots.txt file was telling search engines “move along, nothing to see here”. Searching google for their company name listed only the link to their main page, then links to other sites that mentioned their company. Not good for a site, REALLY not good for a marketing site.

Although I didn’t find out definitively what created the robots.txt, I think it may have been a misconfiguration with their CMS. I’m going to keep an eye on it to be sure it doesn’t revert.

After editing the robots.txt file and creating a sitemap, I think their site traffic will be picking up within a week or so. Just another gotcha to watch out for on your websites.