I have a form which sends data to a custom php file in my root directory. However, when I try processing it, XenForo says a security error has occured.
You might need to include examples of the code but if this is a large file then it will be down to settings within PHP, including upload_max_filesize and/or post_max_size.
If it's not file size related and it's as a result of some custom HTML or script then it could be the lack of CSRF token included in the form which XF uses to authorise the request.