The FAQ for creating the SNS subscription does not mention whether or not to enable:
Enable raw message delivery
I did not enabled this before but I think this was the problem. It instantly confirms if that is enabled. It will fail if its not enabled.