Apple has its own Web crawler, called Applebot, scouring websites for its own search results just like Google, but it isn't likely a move to take over the search engine market. Instead, Apple says its Web crawler is collecting information for Siri and Spotlight.
Applebot is scouring websites for Siri and Spotlight search results
Apple talks about Applebot in a Knowledge Base article saying,
Applebot is the web crawler for Apple, used by products including Siri and Spotlight Suggestions. It respects customary robots.txt rules and robots meta tags. It originates in the 17.0.0.0 net block.
Web crawlers are the tools Google, Microsoft, and other companies running Internet search engines use to build their massive databases so we can find dancing kitten videos online. Since Apple is using its own Web crawler to gather up search results, it could be planning on competing with Google and Bing, but for now that doesn't seem likely.
Apple already relies on Wolfram Alpha and Microsoft's Bing for Siri search results, and for the most part users aren't aware Bing plays a role in figuring out just how long the Kessel Run is. Apple is effectively branding Bing's search results as its own.
Instead of starting over with its own search engine, much like it did with Maps, Apple can instead supplement those results with its own data to give Siri potentially better overall results than it can with Google or Bing alone. Those results can help retain current Apple device users and draw new buyers into the fold, too.
For Apple, it's about selling hardware, and all of the services and apps the company offers are on some level designed with that goal in mind. Applebot likely serves the same purpose by enhancing Siri, Spotlight, and possibly Maps search results.
That doesn't mean Apple won't ever go head to head with Google or Bing it just means for now that's not Apple's likely goal.
[Thanks to 9to5 Mac for the heads up]