fix (backend): Patching the SSRF vulnerability in Github/Web Search/Request related blocks

[jackfromeast]

Background

Refer to the report on huntr.com: https://huntr.com/bounties/1d91e1e1-7d45-4bda-bc27-bfe9052fd975

Changes 🏗️

• Add the domain validation (i.e., github.com) to all the blocks under /blocks/github
• Add the URL validation to the Web Search block and Web Request block: Block all the private IP requests by default unless listed in the config file's trust_endpoints_for_requests field.

Testing 🔍

Tested locally.

Configuration Changes 📝

• Add a trust_endpoints_for_requests field for the developer to create whitelist domains to request.

[CLAassistant]

All committers have signed the CLA.

[github-actions]

This PR targets the master branch but does not come from dev or a hotfix/* branch.

Automatically setting the base branch to dev.

[qodo-merge-pro]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 Security concerns SSRF protection: The PR introduces URL validation to prevent Server-Side Request Forgery (SSRF) attacks. This is a significant security improvement, but the implementation should be carefully reviewed to ensure it covers all possible scenarios and edge cases. Pay special attention to the validate_url functions in http.py and search.py, and make sure they are consistent and comprehensive in their checks.

⚡ Recommended focus areas for review Security Concern The validate_url function is a critical security measure. Ensure it correctly prevents SSRF attacks and properly handles all edge cases. Redundant Code The is_github_url function is defined multiple times across different files. Consider moving it to a shared utility module to improve maintainability. Code Duplication The validate_url method in the GetRequest class is nearly identical to the one in the http.py file. Consider refactoring to use a shared implementation.

[netlify]

✅ Deploy Preview for auto-gpt-docs canceled.

Name	Link
🔨 Latest commit	86c3f50
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/672d593faea3d80008003aa2

Addressed

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.