What is the primary purpose of a robots.txt file?
Improve page load speed
Provide crawling instructions to search engine bots
Prevent search engines from indexing an entire site
Display a human-readable privacy policy for users