XF will start bugging you about missing files when you delete the directory iirc. Anyhow, it doesn't matter if it's in place or not. There's no harm that can be done through it. It's password-restricted the same way your ACP is, and only available to super administrators. The worst someone could do through it would be upgrading your site for you, which would require upgrade files to be present in the first place.