I am getting this issue as well, after searching through this site on attachment uploading issues I've pinned down the problem to the Flash Uploader. If I disable it, I can use the "Upload a File" button. If it is enabled, I cannot use the button, but I can drag & drop the file into the editor window and that works.
Just to note that mod_security can't always be adjusted from .htaccess (probably true more often than not). Your host should be able identify the rule that is being hit and whitelist it for your account.
I figured this one out. The issue was not related to mod security at all. The issue was occurring on an installation that is currently password protected as it is a dry-run conversion that is under construction. It turns out that password protecting the XF installation directory will interfere with the flash uploader.
I figured this out because another XF license I'm running with the same host did not exhibit the issue. I removed the PW protection and the flash uploader worked, then put the protection back on without refreshing the page and the flash uploader failed.
So just FYI for anyone seeing the same issue on a test installation, that's the reason. Disable the uploader in ACP and then enable it when you go live, problem solved.