); ">
You can identify DigitalShadowsBot by the user agent string in a request:
Mozilla/5.0 (compatible; digitalshadowsbot/1.0; +http://www.digitalshadows.com/bot/)
What is DigitalShadowsBot?
DigitalShadowsBot is the generic name for Digital Shadows’ main web crawler.
“Crawler” is a generic term for any program (such as a robot or spider) that is used to automatically discover content on the Internet by following links from one webpage or public resource to another. Digital Shadows main crawler is called DigitalShadowsBot.
How DigitalShadowsBot accesses your site
DigitalShadowsBot obeys robots.txt on both HTTP and FTP services. DigitalShadowsBot will crawl your site or service with a maximum of three concurrent threads at any one time so as not to impact bandwidth or performance.
DigitalShadowsBot will revisit your site or service no more frequently than every 7 days on average. However, due to delays it is possible that the frequency will be slightly greater on occasion.
How to control DigitalShadowsBot
If you want to limit or stop DigitalShadowsBot’s activity you have two options:
– To prevent DigitalShadowsBot from indexing your site or service or control what areas it indexes a robots.txt file can be configured. The user agent token to reference in your robots.txt file is digitalshadowsbot.
– To prevent future attempts by DigitalShadowsBot to connect to your site or service please email us. We actively maintain an exclusion list for DigitalShadowsBot.