Yes, the library directory can be emptied.
We keep it in the download package though (I can't recall why) but it's empty and unused so delete it but don't be alarmed to see it reappear.
When you can, it's worth running the clean up command as there are other unused files and directories.
If you vaguely know your way around code then you may be able to figure out the other files / dirs to remove from that cleanup command script.