feat(backend): RedisEventQueue into Pub/Sub

[majdyz]

Background

There are two issues with the current Redis event queue implementation:

1. It's a queue, which means there can only be 1 event subscriber.
2. It's implemented in a suboptimal way on ws_api:
   • The subscription loop lives in a fast API task (which is supposed to be async).
   • The task is implemented with a continuous sync loop with sleep.

Changes 🏗️

• Converted RedisEventQueue --> RedisEventBus, and provided two versions, the async & sync version.
• The WebSocket service / fast API will use the async version for subscribing to the event update.
• The ExecutionManager will use the sync version for publishing the events.

Testing 🔍

Note

Only for the new autogpt platform, currently in autogpt_platform/

• Create from scratch and execute an agent with at least 3 blocks
• Import an agent from file upload, and confirm it executes correctly
• Upload agent to marketplace
• Import an agent from marketplace and confirm it executes correctly
• Edit an agent from monitor, and confirm it executes correctly

[qodo-merge-pro]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 Security concerns Sensitive information exposure: The Redis connection details (host, port, password) are being loaded from environment variables. Ensure these are properly secured and not exposed in logs or error messages.

⚡ Recommended focus areas for review Error Handling The error handling in the _deserialize_message method could be improved. It currently catches all exceptions, which might mask specific issues. Connection Management The connection management for both synchronous and asynchronous Redis connections could be refactored to reduce code duplication. Exception Handling The event_broadcaster function catches all exceptions. Consider handling specific exceptions separately for better error reporting and recovery.

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.