Are you comfortable with javascript? What you're trying to do is going to require quite a bit of javascript.
If you open up js/xenforo/full/xenforo.js and look for where XenForo.AutoValidator.prototype is defined (line 6971) you'll see it fires some events during the form submit process. If you're trying to hook into the reply feature, this is what you're going to want to use - as the reply forms get run through the AutoValidator. This is the line you're going to be interested in:
PHP:
/**
* Event listeners for this event can:
* e.preventSubmit = true; to prevent any submission
* e.preventDefault(); to disable ajax sending
*/
eDataSend = $.Event('AutoValidationBeforeSubmit')
This is the name of the event that gets fired before the form gets officially submitted.
For example, if I was going to tap into the quick reply form I would do something like this:
PHP:
$('#QuickReply').on('AutoValidationBeforeSubmit', function(event) {
// this is where your javascript handling is going to go
// this is where you will open your modal, most likely with a new form in it containing your captcha
// your logic will go something like this:
triggerOverlay(); // note this is not the function you will use, this part will be the hardest part
if (validCaptcha) {
// this is where we will re-submit the original data -- and allow it to go all the way through
} else {
// they entered an invalid captcha, we're going to prevent the form from submitting
event.preventSubmit = true; // this will prevent the form from being submitted, nothing will happen
}
});
Note that this is just a very rough sketch of what the logic will look like, this is by no means something you can just paste in and make work. What you are trying to do is not going to be overly simple by any means, so I can't write it all for you here without me writing the actual add-on for you.
What might be easier for you is to include the captcha ON the page with the form, instead of having a popup once they click the submit button. If you have the captcha on the form with the button, you can send their captcha value along with the post data -- validate the captcha in your PHP controller, and if it's valid then continue submitting the data, else if it's not valid you could send back an error saying it's not valid.
Does this at least point you in the right direction?