There is no limit on PHP's mail system. There may be a limitation set upstream of that (like when it hands off the mail to Sendmail or Postfix), but in all likelihood, the limit is going to be simply how many emails the transport system can take in (limits of the server resources). If you try to send a billion emails at once for example, you are going to run into disk i/o problems of writing that many emails to be delivered to the file system, you are also going to run into network bottlenecks trying to deliver that many concurrently, and you are also going to be CPU bound because making decisions on where to send a billion emails at the same time is going to be taxing on the server's CPU.
So even if there's no hard limit, you are going to run into practical limits. Similar to your server is not going to be able to serve up a website like Facebook, Google or YouTube on it's own even though there's nothing preventing it from trying.