DigitalOcean Interview Question

How does the robots.txt file do?