Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses

百度站长平台是百度网页搜索为网站管理员搭建的站长工具和交流平台，提供sitemap、死链提交、站点索引量查询、站点抓取异常、seo优化建议、站点压力反馈、robots、等站长工具，同时提供百度官方资料区站长论坛讨论区，引导优质内容运营，提升网站用户体验，帮助互联网行业健康发展