Apple Inc. Said To Be Running Suspicious Web Crawler [REPORT]

Apple Inc. (NASDAQ:AAPL) seems to be running an automated web crawler that’s crawling HTML websites for an unknown purpose. The crawler is apparently written in the Go programming language and turns off sometimes.

Developer discovers Apple crawler

Apple Insider points to a tweet from developer Jan Moesen, who seems to be the first to have run across the mysterious crawler and asks whether anyone knows why Apple is running it.

Michael Mauboussin: Here’s what active managers can do

michael mauboussin, Credit Suisse, valuation and portfolio positioning, capital markets theory, competitive strategy analysis, decision making, skill versus luck, value investing, Legg Mason, The Success Equation, Think Twice: Harnessing the Power of Counterintuition, analysts, behavioral finance, More Than You Know: Finding Financial Wisdom in Unconventional Places, academics , valuewalkThe debate over active versus passive management continues as trends show the ongoing shift from active into passive funds. Q2 2020 hedge fund letters, conferences and more At the Morningstar Investment Conference, Michael Mauboussin of Counterpoint Global argued that the rise of index funds has made it more difficult to be an active manager. Drawing Read More

Apparently it comes from an IP address that starts with the number 17 and that all IP addresses come from Apple’s servers. The developer found evidence of the web crawler dating back to Oct. 15, and he’s not the only one to notice it. Others have said that there seems to be quite a bit of traffic from Apple’s mysterious web crawler.

What is the crawler for?

The automated bot only requests information from HTML pages and doesn’t actually seem to get into the CSS part of the website or the JavaScript or images that are on the pages. According to Moesen, the bot gives an identification of Mozilla/ 5.0 (compatible; Fetcher/0.1).

According to Apple Insider, it’s unclear whether the web crawler is officially from Apple or whether it’s some sort of unofficial side project being run by an employee at the company. The website suggests that the bot could have something to do with Apple’s improvements in its Spotlight feature, which is in both iOS 8 and OS X Yosemite.

Both of them received updates recently that enable users to directly access online content without using third party search tools like Google Inc (NASDAQ:GOOGL) (NASDAQ:GOOG)’s Search or Microsoft Corporation (NASDAQ:MSFT)’s Bing search engine. Apple has been trying to get away from using competitors to bring up information for searches or queries done through its Siri digital assistant. Currently Apple uses Bing as its official search partner for Siri.