Server Issue: Connection refused from computer on same network

I've got this weird issue with my server, CentOS 7, running nginx, that I can't seem to figure out. Basically trying to access the server from the same network on two devices doesn't work. If I access the site on my desktop, it works fine, open it up on mobile device on same network, it'll open. Then if I refresh the desktop it'll time out and give me a "connection refused" error of some kind. If I don't do anything on the mobile device, just wait about 2 minutes, it'll start working again. It's like something is not letting two devices from the same network view the site at the same time.

I looked in nginx.conf, firewalld, csf, and any other place I can think of to see if somehow there was a firewall setting or iptables or something that would lead to this and can't figure it out.

Any ideas?