This is really strange. :| I just removed all notices and then added a new notice with only admin rights, it worked fine. Going back in to the admin panel, I added permission for user is guest and user is logged in to view the notice and it has disappeared again.
I might try and install a fresh instance of xenforo and try this again.