OpenAI Interview Question

Write a web crawler. Emphasis on multithreading/multiprocessing