Apple Inc. (NASDAQ:AAPL) seems to be running an automated web crawler that’s crawling HTML websites for an unknown purpose. The crawler is apparently written in the Go programming language and turns off sometimes.
Developer discovers Apple crawler
Apple Insider points to a tweet from developer Jan Moesen, who seems to be the first to have run across the mysterious crawler and asks whether anyone knows why Apple is running it.
Does anyone know why Apple is running a web crawler (written in @golang, no less)? http://t.co/5JDTZuplQp
— Jan Moesen (@janmoesen) November 6, 2014
Apparently it comes from an IP address that starts with the number 17 and that all IP addresses come from Apple’s servers. The developer found evidence of the web crawler dating back to Oct. 15, and he’s not the only one to notice it. Others have said that there seems to be quite a bit of traffic from Apple’s mysterious web crawler.
What is the crawler for?
The automated bot only requests information from HTML pages and doesn’t actually seem to get into the CSS part of the website or the JavaScript or images that are on the pages. According to Moesen, the bot gives an identification of Mozilla/ 5.0 (compatible; Fetcher/0.1).
According to Apple Insider, it’s unclear whether the web crawler is officially from Apple or whether it’s some sort of unofficial side project being run by an employee at the company. The website suggests that the bot could have something to do with Apple’s improvements in its Spotlight feature, which is in both iOS 8 and OS X Yosemite.
Both of them received updates recently that enable users to directly access online content without using third party search tools like Google Inc (NASDAQ:GOOGL) (NASDAQ:GOOG)’s Search or Microsoft Corporation (NASDAQ:MSFT)’s Bing search engine. Apple has been trying to get away from using competitors to bring up information for searches or queries done through its Siri digital assistant. Currently Apple uses Bing as its official search partner for Siri.
