I was able to solve the problem after reading this article:
https://www.vpsbasics.com/security/how-to-block-bad-bots-using-modsecurity-with-directadmin
I don't have any control panel, I run my VPS with command line, so I made my own modification on the article method to suit my needs...