解決屏蔽流氓蜘蛛抓取，如MJ12bot 、DotBot 、BLEXBot 、PetalBot 、DataForSeoBot

---

1、robots文件屏蔽

User-agent: MJ12bot
Disallow: /
User-agent:DotBot
Disallow: /
User-agent:BLEXBot
Disallow: /
User-agent:PetalBot
Disallow: /
User-agent:DataForSeoBot
Disallow: /

2、Nginx等服務器規則屏蔽

if ($http_user_agent ~*  （MJ12bot|DotBot|BLEXBot|PetalBot|DataForSeoBot） )
{return 403;
}

注意：在前期如果流量少，可以設置下防火墻，屏蔽掉轟炸式蜘蛛，蜘蛛來訪也可以增加收錄率，所以根據自己需求來屏蔽

if ($http_user_agent ~* (YandexBot|spbot|DnyzBot|Researchscan|semrushbot|yahoo|AhrefsBot|DotBot|Uptimebot|MJ12bot|MegaIndex.ru|ZoominfoBot|Mail.Ru|SeznamBot|BLEXBot|ExtLinksBot|aiHitBot|Barkrowler)){return 403;
}