Doing so would cause problems, including reporting missing files and will prevent one-click upgrades (triggered from your control panel) from working.
The install system is password protected and, even if it wasn't, the installer is "locked" so no destructive operations can be triggered. If we wanted people to delete it we would tell you to do that very clearly during the install or upgrade process.