There isn't really anything that can do it by default, so it would require custom code to handle it.
Vaguely speaking if the form is being submitted using the AutoValidator then you can listen to certain events, prevent the default actions where applicable, and then just instantiate an overlay.
There's a few code examples in XFMG where an overlay is launched entirely using JS, though none specifically as a consequence of submitting a form. In XFMG 1.1:
Code:
options = {};
options.speed = XenForo.speed.fast;
this.trigger = { href : $(this).attr('href') };
this.OverlayLoader = new XenForo.OverlayLoader($(this.trigger), true, options);
this.OverlayLoader.load();
Where "$(this).attr('href')" is the URL of the overlay that needs to be opened.