If you have ever tried to add your iCloud calendar in Google Calendar, you probably saw this error: "Could not fetch the url because robots.txt prevents us from crawling the url." That's because the https://www.icloud.com/robots.txt file tells "web crawlers" not to index anything under /calendar/ which is a good idea if we're talking about something like Googlebot but should not apply when done in response to an interactive action initiated by a human. However, for reasons unknown, Google decided that Google Calendar should follow Apple's robots.txt instruction.[1]
I have looked for
