IN SHORT: no, you don’t.
IMO webpage load times and server performance doesn’t help SEO anywhere near what people think it does. There’s some kind of notion going around that super fast sites rank higher on Google and that’s not true. Tons of top-ranked sites are slow as heck. Google penalizes you for being very slow, they don’t reward you for being fast. They don’t penalize you for being “a little slow”.
HOWEVER….your site speed will affect your user experience and that certainly affects Google rankings somewhat. If your site is so slow that users leave often or don’t click through as many pages, that will reflect on your SEO a little bit.
Ultimately…site speed affects user experience and sales, more so than SEO.