Have you ever visited a web page only to find out that the information that Google showed required a username and password or registration to view? How did Google get the information to index for its search engine?

The secret is that the website recognizes Google’s robot (or spider) and delivers different content than it does to normal website visitors. What it looks for to determine this is what’s called the user-agent identity given by google’s robot. When you visit a web page using the latest version of Firefox, your browser identifies itself to the website as {Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12} and the website knows how to deliver the content to you.

Using an add-on for Firefox, you can impersonate the googlebot and fool the website into thinking you are Google looking for its latest content. What’s more, you can create your own user-agent strings to personalize your visits to sites if you want to. (their statistics software would see it, so the website owner might get a kick out of it)

The add-on required to do this is aptly called user-agent switcher and it adds an option to your Tools menu in Firefox that lets you choose the browser robot you want to impersonate. You used to have to manually add the option for Googlebot, but the latest version has it as an option from the start. Here’s the menu options you have after installation.

Default delivers your default user agent string, and the othes are self-explanatory. After you set your user agent to one of the options, visit Webgeek and see what you user agent string is delivering to websites.