That would require significant modifications. We use TinyMCE, but the output is converted to BB code and we don't have a table BB code.
Modifications and customizations are beyond something that we can provide official support for. I would recommend you post in our development forums. However, this is likely a large job - conversions to and from HTML need to be implemented, as well as the additional button and the necessary customized overlay for modifying the table.