You only need to cache phrases that are not referenced directly in templates.
So, this phrase:
my_template
<p>{xen:phrase my_phrase}</p>
... does not need to be cached, while the one referenced here:
another_template
<p>{xen:phrase $phraseName}</p>
... needs to be cached or it won't work.
It's not really a matter of caching for performance, it's a matter of caching so that the phrase is available at all. If the phrase content can be known at the point at which you save the template, it likely does not need to be cached.