Last year, Google's Matt Cutts dropped the bomb (to put it in the exaggerated tone that many took the news in), that Google was considering taking site speed into consideration as one of many potential ranking factors for search results.
This of course freaked a lot of people out, but as Matt and Google as a whole has maintained, this would not trump relevance. It would be taken more into consideration when there are two sites of relatively equal relevance, but one site loads faster and delivers a better user experience.
Maile Ohye, Senior Developer Programs Engineer for Google at SMX, offers some advice about website performance (speed), how that pertains to search rankings and the user experience, and some tips for making sure your site is up to speed, so to speak.
As far as site speed as a ranking factor, Ohye pretty much makes the same point as Cutts, and it's probably not going to be something where all of a sudden all of the faster sites are ranking better and the slower ones are doing worse. But it does enhance the user experience, and she refers to a study that found that an optimized site actually increased conversions by 16%. So if you're not optimizing your site's performance for Google, maybe that's a good enough reason on its own.