Add some efficiencies to prevent unnecessary requests#42

Merged

freekmurze merged 3 commits intospatie:mainfrom tsjason:main

Apr 19, 2024

+39-10

Commits on Apr 17, 2024

Support providing a RobotsTxt object directly.
tsjason
committed
Also allow withTxt to be passed a RobotsTxt object
tsjason
committed
Get the url's content on the Robots class in order to reduce the number of requests to the remote server.
tsjason
committed