Crawling Google Trivia How many bots does Google have? More than 25 and less than 1,000 Crawling Hosting and Servers Should I use the Vary HTTP header on URLs that redirect based on user-agent? Yes, it's still used by Google though maybe not other services Crawling Website Coding How does Google handle "not found" pages that don't return a 404? Google tends to treat as regular 404s Crawling Website Content Should I block Googlebot from crawling a dynamically generated calendar? Yes Craw
Additional text has been truncated due to copyright reasons. Things without URLs and private things don't get truncated.