feat: add timeout param to scrape and crawl

[ezhil56x]

/claim #59

[rafaelsideguide]

@ezhil56x awesome job! Could you add some e2e tests?

[ezhil56x]

@rafaelsideguide
Should we have a default timeout or is it not required?

[ezhil56x]

@rafaelsideguide
Can I get a review once?

[rafaelsideguide]

@ezhil56x
for /scrape the default should be 20 seconds
for /crawl the default should be this.limit x 20 seconds

[ezhil56x]

@rafaelsideguide
Done!

[rafaelsideguide]

@ezhil56x crawl is not working properly. I added a test on e2e_withAuth with how it's supposed to work.

[ezhil56x]

@rafaelsideguide
Updated my PR

[rafaelsideguide]

The v0/crawl test does not pass. The timeout function runs, but the job is still running, so it crawls the pages and returns a success response. This should not happen.

[nickscamara]

@rafaelsideguide we should also support partial docs on the webhook, just got a customer request for that!

[nickscamara]

/tip $10

[algora-pbc]

🎉🎈 @ezhil56x has been awarded $10! 🎈🎊

• Share on socials
• Give feedback

[nickscamara]

@ezhil56x thanks for the initial contribution, unfortunately we are closing this for now as we will need some architectural changes to be able to do timeout in the crawl reliably.