I think I've narrowed this down to mobile and suspect that the attachment URLs are being handed over to the system or PDF viewer for download, and then of course they don't have permission to get it so the download fails. I assume even of desktop though, if there is some kind of download manager in use the same problem will occur.
Testing on my desktop with Firefox and Adobe Reader it works fine but testing on Android with Chrome it fails. Desktop Firefox will download the file first, then open it in Reader.
On my forum I have some attachments in restricted forums, but even those in the general forum don't download. The only way I can find to allow attachments to always work is to allow the Unregistered/Unconfirmed usergroup to 'view attachments to posts'.
Is there any other solution to this? Would changing over to XSendfile to hand-off the downloads to Apache rather than PHP help, or am I going to hit the same roadblock?