蜘蛛UAspider-ua
收录百度、谷歌、必应、神马等常见搜索引擎与工具的爬虫 User-Agent 字符串，便于识别日志与配置抓取规则并复制。
收录常见搜索引擎与工具的爬虫 User-Agent，当前显示 20 条，可用于识别日志、配置 robots 规则。UA 版本号会随时间更新，仅供参考。
| 爬虫名称 | 归属 | User-Agent | 复制 |
|---|---|---|---|
| Baiduspider | 百度 | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | |
| Baiduspider-render | 百度渲染 | Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html) | |
| Sogou web spider | 搜狗 | Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) | |
| 360Spider | 360 搜索 | Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36 360Spider | |
| YisouSpider | 神马搜索 | Mozilla/5.0 (compatible; YisouSpider/5.0; +http://www.yisou.com/help/help_faq.html) | |
| Bytespider | 字节跳动 | Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) | |
| Googlebot | 谷歌 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | |
| Googlebot-Image | 谷歌图片 | Googlebot-Image/1.0 | |
| Bingbot | 必应 | Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) | |
| YandexBot | Yandex | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | |
| DuckDuckBot | DuckDuckGo | DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) | |
| Applebot | 苹果 | Mozilla/5.0 (compatible; Applebot/0.1; +http://www.apple.com/go/applebot) | |
| PetalBot | 华为花瓣 | Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) | |
| Yahoo! Slurp | 雅虎 | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | |
| Sosospider | 搜搜 | Sosospider+(+http://help.soso.com/webspider.htm) | |
| AhrefsBot | Ahrefs | Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) | |
| SemrushBot | Semrush | Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html) | |
| MJ12bot | Majestic | Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) | |
| facebookexternalhit | facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) | ||
| Twitterbot | Twitter/X | Twitterbot/1.0 |