The Smilie Count add-on will limit the number of smilies allowed in a post. By default the add-on will limit five smilies per post.
To change the maximum number of smilies allowed in a post:
Admin CP -> Options -> Messages -> Maximum Smilies Per Post
This add-on only works on XenForo v1.2 and above.