Apache UserAgentで制限

# Apache2.4系の場合
SetEnvIf User-Agent "Bytespider" bothersome_bot
BrowserMatchNoCase "Bytespider" bothersome_bot
<RequireAll>
Require All Granted
Require not env bothersome_bot
</RequireAll>

https://www.advantech.jp/archives/3619

bothersome_bot

<IfModule mod_setenvif.c>
    SetEnvIf User-Agent "Bytespider" block-bot
</IfModule>

<IfModule mod_authz_core.c>
    <RequireAll>
        Require all granted
        Require not env block-bot
    </RequireAll>
</IfModule>

block-bot

https://zbnr-hp.com/denybot/
denybot