I know some people are very meticulous about this, but I'm a very laid-back person who refuses to get unnecessarily anal about brewing tea or anything else.
I frequently let my tea leaves sit in the pot on the counter overnight and use them again the next day. I have never noticed any souring, mold growth, or off-taste in doing this, but I also keep my home very cool. I think if the temperature is too warm, you might get some fuzzy mold issues. It hasn't killed me or ruined my tea experience yet.