Lyft Interview Question

How to design a web crawler?