Google announced last week that Site Speed is now one of the ranking factors. According to Google this will only effect less than one percent of all websites. In the Google Webmasters tools there is a Site Speed tool that will tell you how fast your site is in comparison to all of the sites that Google crawls. So unless your site is slower than 99% of all Internet websites you have nothing to worry about.
Google also now had an add-on for Firefox that will tell you the loading speed for the site that you are visiting. It will also give you helpful information on how to make the site faster.