Some websites can block access to prevent web scraping, that can be easily detected if your Python script is sending multiple requests in a short period of time.
To not get banned you can try to add random delays between queries.
For this you can use the Python’s
sleep() function that suspends (waits) execution of the current thread for a given number of seconds with the
randint() function that returns a random integer.
Cool Tip: How to set the ‘User-Agent’ HTTP request header in Python! Read More →
Random Sleep Timer in Python
Use the following Python’s code snippet to sleep a random number of seconds:
from random import randint from time import sleep # Sleep a random number of seconds (between 1 and 5) sleep(randint(1,5))
Most of the time you need to create some random delays between iterations in a loop.
In the example below the Python is sleeping a random number of seconds between printing a message:
from random import randint from time import sleep while True: delay = randint(1,5) print("Sleep " + str(delay) + "s") sleep(delay)
Sleep 5s Sleep 1s Sleep 2s Sleep 2s Sleep 4s