I have noticed this issue repeatedly on my Mac using Chrome on both the XenForo site and my forums. I end up having to exit Chrome completely and then when I restart everything is running fine - so in MY case it's a browser issue that I just haven't taken the time to look into. I can (when it is happening for me) open Safari and my speed is just fine.
As
@Amaury indicated, try clearing your browser cache and see if that helps. Also, when it is happening run a speed test to verify that your ISP is not having a cruddy connection with you.