The only check you can make in the backend is, if the page was requested via AJAX. There's no way to differentiate between an overlay, menu, tab or any other AJAX call. If you need specifically overlays, you'll need to extend the overlay trigger script to pass a custom parameter to the server.