Are you comfortable with javascript? What you're trying to do is going to require quite a bit of javascript.
If you open up js/xenforo/full/xenforo.js and look for where XenForo.AutoValidator.prototype is defined (line 6971) you'll see it fires some events during the form submit process. If you're trying to hook into the reply feature, this is what you're going to want to use - as the reply forms get run through the AutoValidator. This is the line you're going to be interested in:
PHP:
eDataSend = $.Event('AutoValidationBeforeSubmit')
This is the name of the event that gets fired before the form gets officially submitted.
For example, if I was going to tap into the quick reply form I would do something like this:
PHP:
$('#QuickReply').on('AutoValidationBeforeSubmit', function(event) {
triggerOverlay();
if (validCaptcha) {
} else {
event.preventSubmit = true;
}
});
Note that this is just a very rough sketch of what the logic will look like, this is by no means something you can just paste in and make work. What you are trying to do is not going to be overly simple by any means, so I can't write it all for you here without me writing the actual add-on for you.
What might be easier for you is to include the captcha ON the page with the form, instead of having a popup once they click the submit button. If you have the captcha on the form with the button, you can send their captcha value along with the post data -- validate the captcha in your PHP controller, and if it's valid then continue submitting the data, else if it's not valid you could send back an error saying it's not valid.
Does this at least point you in the right direction?
