It looks like the header tries to load its logo and buttons as the page refresh itself, which is bizarre (or I'm doing something wrong).
I don't know if this bug affects this: http://xenforo.com/community/threads/litespeed-sends-text-html-with-css-304-responses.54185/
Since I don't have LiteSpeed, it seems that it is irrelevant to my issue (I have the regular configuration of Apache +PHP + XCache which I tried to disable and no hope for me.)
Here is a short demo that shows the difference between Chrome, Firefox and Safari. It seems that Safari is the worst of the three, and Firefox is the best in terms of caching the header. At the end, I compare it to the default design which loads instantly and caching the header most of the time.
Better to watch it in 720/1080p:
Thanks for the help.