Nginx限制爬虫

修改 nginx.conf，禁止网络爬虫的 ua，返回 403，具体配置如下：

server{
	listen 80;
	server_name 127.0.0.1; 
	# 添加如下内容即可防止爬虫
	if ($http_user_agent ~* "qihoobot|Baiduspider|Googlebot|Googlebot-Mobile|Googlebot-Image|Mediapartners-Google|Adsbot-Google|Feedfetcher-Google|Yahoo! Slurp|Yahoo! Slurp China|YoudaoBot|Sosospider|Sogou spider|Sogou web spider|MSNBot|ia_archiver|Tomato Bot") 
	{
		return 403;
	}
}

Nginx限制浏览器访问

if ($http_user_agent ~* "Firefox|MSIE")
{
     return 403;
}