GoogleBot - Spiders Everywhere!

By
Real Estate Agent with Douglas Elliman Real Estate in Babylon NY

GoogleBot - Since I've been using Sitemeter to see how people find my blog and real estate website, I've been seeing the location: Mountainview, California a lot. It was making me a little concerned because it's many times per day, everyday. They would view many pages and the duration times were always over 3 minutes.

So, I clicked on the link to bring me to the details about the "person" and the domain name is Googlebot.com. Since my inquiring mind wanted to know, I googled googlebot and found this:

Googlebot is the search bot software used by Google, which collects documents from the web to build a searchable index for the Google search engine.

If a webmaster wishes to restrict the information on their site available to a Googlebot, or another well-behaved spider, they can do so with the appropriate directives in a robots.txt file, or by adding the meta tag <meta name="Googlebot" content="nofollow" /> to the webpage. Googlebot requests to Web servers are identifiable by a user-agent string containing "Googlebot" and a host address containing "googlebot.com".

Currently Googlebot only follows HREF links and SRC links. Googlebot discovers pages by harvesting all of the links on every page it finds. It then follows these links to other web pages. New web pages must be linked to other known pages on the web in order to be crawled and indexed.

A problem which webmasters have often noted with the Googlebot is that it takes up an enormous amount of bandwidth. This can cause websites to exceed their bandwidth limit and be taken down temporarily. This is especially troublesome for mirror sites which host many gigabytes of data. Google provides "Webmaster Tools" that allow website owners to throttle the crawl rate.

Now I know it's not a who, it's a spider! This is the only spider I like!

 

Comments (9)

Nu Nu
777S INC (Videos by Vegas Bob) - Las Vegas, NV

This is interesting but what does it mean? I am not much of a techie and I don't like spiders.

Jan 07, 2010 01:21 PM
Paul Guenther
Empire Title Agency - Chandler, AZ
The Extra Mile Is Part Of My Regular Route!

Jackie,

 That is cool and lucky for you to have found it too..  I am going to check out site master

 

Jan 07, 2010 01:24 PM
Jackie Connelly-Fornuff
Douglas Elliman Real Estate in Babylon NY - Babylon, NY
"Moving at The Speed of YOU!"

Hi Nu Nu, It's the system where Google searches for website content to index into google so people can find your website, blogs, etc. I hope I made sense!

Hi Paul, It's sitemeter.com I'm glad I looked into it a little further. It was bugging me,, lol.

Jan 07, 2010 01:28 PM
Ruth Vogt
Fairway Independent Mortgage, LLS. Equal Housing Opportunity. Regulated by the Division of Real Estate. - Colorado Springs, CO
719-592-0855 www.ReverseLoansInColorado.com

Jackie, good paly on words... spider was bugging you! LOL! I did add sitemeter after reading one of your blogs, but must admit I've figured it out yet. It's on my to-do list!

Jan 07, 2010 03:19 PM
Jackie Connelly-Fornuff
Douglas Elliman Real Estate in Babylon NY - Babylon, NY
"Moving at The Speed of YOU!"

Hi Ruth! To do this! Get off the pot Ruth :)

Jan 07, 2010 11:45 PM
Michelle Francis
Tim Francis Realty LLC - Atlanta, GA
Realtor, Buckhead Atlanta Homes for Sale & Lease

Jackie,  

Great info about the robots and spiders on your web site.  I was just doing some analyzing last night and realize I need to more!

Thanks for the tips.  All the best, Michelle

Jan 08, 2010 05:20 AM
Jackie Connelly-Fornuff
Douglas Elliman Real Estate in Babylon NY - Babylon, NY
"Moving at The Speed of YOU!"

Hi Michelle! Thank you. I was glad I found the info I did. Now I have an even better understanding about search engines than I did before :)

Jan 08, 2010 05:45 AM
Ruth Vogt
Fairway Independent Mortgage, LLS. Equal Housing Opportunity. Regulated by the Division of Real Estate. - Colorado Springs, CO
719-592-0855 www.ReverseLoansInColorado.com

You know, I've been thinking about this... shortly after you made this post, I saw several "spam" emails on one of your other posts... wonder if there is a coincidence here??? Something tied to something else in the confusing world of spiders, etc? (BTW, at first I thought your earlier response to me had something to do with irregularities!) ha-ha-ha!

Jan 09, 2010 06:05 AM
Jackie Connelly-Fornuff
Douglas Elliman Real Estate in Babylon NY - Babylon, NY
"Moving at The Speed of YOU!"

Hi Ruth, I deleted all those spam advertising. There were at least 20 of them posted. People will do that. I've been seeing Googlebot since I installed Sitemeter onto my blog.

Jan 09, 2010 10:39 AM