I didn't want to do the whole ajax template coming up with the input field like is done with attachments so messed around a bit and am doing it a different way which I think would be better for XenForo as a whole.
In XenForo you have your 3 buttons. Submit, Upload and a 3rd one that changes. To create a button that works similar to the flash upload button simply create the file input below the submit and 3rd button but in front of the upload a file button. Then give it opacity: 0;. Few margins here and there and some JS so the :hover gets added to the button under the input[file]. But basicaly the end result is a button that you click that works the same as the flash button minus all the features you get with flash. Use your ajax from there and add your file.
I can explain better if you like but you should get the idea.