Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(backend): Move credentials storage to prisma user #8283

Merged
merged 103 commits into from
Oct 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
27c62a1
feat(frontend,backend): testing
ntindle Sep 30, 2024
4293a70
feat: testing
ntindle Sep 30, 2024
b4d2935
feat(backend): it works for reading email
ntindle Oct 1, 2024
9e15497
feat(backend): more docs on google
ntindle Oct 1, 2024
f079983
fix(frontend,backend): formatting
ntindle Oct 1, 2024
59d8ccd
feat(backend): more logigin (i know this should be debug)
ntindle Oct 1, 2024
ac8e99b
feat(backend): make real the default scopes
ntindle Oct 1, 2024
95d6c28
feat(backend): tests and linting
ntindle Oct 1, 2024
fc6084c
fix: code review prep
ntindle Oct 1, 2024
59f0094
feat: sheets block
ntindle Oct 1, 2024
5be1ce0
feat: liniting
ntindle Oct 1, 2024
81d1052
Update route.ts
ntindle Oct 1, 2024
b01da5a
Update autogpt_platform/backend/backend/integrations/oauth/google.py
ntindle Oct 1, 2024
25c5206
Update autogpt_platform/backend/backend/server/routers/integrations.py
ntindle Oct 1, 2024
392cf7a
fix: revert opener change
ntindle Oct 1, 2024
fac9a53
feat(frontend): add back opener
ntindle Oct 1, 2024
b5dded0
feat(frontend): drop typing list import from gmail
ntindle Oct 1, 2024
56cabd9
fix: code review comments
ntindle Oct 1, 2024
b6f17dc
feat: code review changes
ntindle Oct 1, 2024
5ec1206
feat: code review changes
ntindle Oct 1, 2024
7491b6a
fix(backend): move from asserts to checks so they don't get optimized…
ntindle Oct 1, 2024
58324a7
fix(backend): code review changes
ntindle Oct 1, 2024
3ec0aa5
fix(backend): remove google specific check
ntindle Oct 1, 2024
9484a4e
fix: add typing
ntindle Oct 1, 2024
af1ef1a
fix: only enable google blocks when oauth is configured for google
ntindle Oct 1, 2024
65de432
fix: errors are real and valid outputs always when output
ntindle Oct 1, 2024
2004d80
fix(backend): add provider detail for debuging scope declines
ntindle Oct 1, 2024
3ebc7dd
Update autogpt_platform/frontend/src/components/integrations/credenti…
ntindle Oct 1, 2024
0da12a8
fix(frontend): enhance with comment, typeof error isn't known so this…
ntindle Oct 1, 2024
d43c944
feat: code review change requests
ntindle Oct 1, 2024
432ea92
fix: linting
ntindle Oct 1, 2024
572e1b0
fix: reduce error catching
ntindle Oct 1, 2024
c629cc6
fix: doc messages in code
ntindle Oct 1, 2024
954c12f
fix: check the correct scopes object :smile:
ntindle Oct 1, 2024
b1b8354
fix: remove double (and not needed) try catch
ntindle Oct 1, 2024
3cd1b58
fix: lint
ntindle Oct 1, 2024
02458f1
Merge branch 'master' into add-gmail
ntindle Oct 1, 2024
42deeee
fix: scopes
ntindle Oct 2, 2024
23e395b
feat: handle the default scopes better
ntindle Oct 2, 2024
b521c63
feat: better email objectification
ntindle Oct 2, 2024
101e9f4
feat: process attachements
ntindle Oct 2, 2024
5d66689
fix: lint
ntindle Oct 2, 2024
bd78850
Update google.py
ntindle Oct 2, 2024
8d3f453
Update autogpt_platform/backend/backend/data/block.py
ntindle Oct 2, 2024
7eaa7b8
fix: quit trying and except failure
ntindle Oct 2, 2024
59ba400
Merge branch 'add-gmail' of https://github.com/Significant-Gravitas/A…
ntindle Oct 2, 2024
e4c4bb8
Update autogpt_platform/backend/backend/server/routers/integrations.py
ntindle Oct 2, 2024
39a99e3
Merge branch 'add-gmail' of https://github.com/Significant-Gravitas/A…
ntindle Oct 2, 2024
9ca4e1e
feat: don't allow expired states
ntindle Oct 2, 2024
2629daf
fix: clarify function name and purpose
ntindle Oct 3, 2024
b4f26d7
feat: code links updates
ntindle Oct 1, 2024
b332d3f
feat: additional docs on adding a block
ntindle Oct 2, 2024
60e295e
fix: type hint missing which means the block won't work
ntindle Oct 2, 2024
ec63197
fix: linting
ntindle Oct 2, 2024
cf0804d
fix: docs formatting
ntindle Oct 2, 2024
154b033
Merge branch 'master' into oauth-docs-updates
ntindle Oct 4, 2024
9cb1b0f
Update issues.py
ntindle Oct 4, 2024
1cdf524
fix: improve the naming
ntindle Oct 4, 2024
0ce1d88
fix: formatting
ntindle Oct 4, 2024
8d0148f
Update new_blocks.md
ntindle Oct 4, 2024
fbb64d1
Update new_blocks.md
ntindle Oct 4, 2024
4a30f72
feat: better docs on what the args mean
ntindle Oct 4, 2024
5517708
feat: more details on yield
ntindle Oct 5, 2024
868aea8
Update new_blocks.md
ntindle Oct 5, 2024
4bbb368
Merge branch 'master' into oauth-docs-updates
majdyz Oct 7, 2024
44cb3dc
Merge branch 'master' into oauth-docs-updates
ntindle Oct 7, 2024
9eb55a8
fix: remove ignore from docs build
ntindle Oct 7, 2024
1991556
feat: initial migration
ntindle Oct 7, 2024
0c65618
feat: migration tested with supabase-> prisma data location
ntindle Oct 7, 2024
66d8299
Merge branch 'master' into move-oauth-to-prisma-user
ntindle Oct 8, 2024
4ff1487
add custom migrations and script
aarushik93 Oct 10, 2024
649b570
update migration command
aarushik93 Oct 10, 2024
9979418
formatting and linting
aarushik93 Oct 10, 2024
8571a89
updated migration script
aarushik93 Oct 10, 2024
63f3394
add direct db url
aarushik93 Oct 10, 2024
31b3349
add find files
aarushik93 Oct 10, 2024
6b39f21
rename
aarushik93 Oct 10, 2024
2bebc14
use binary instead of source
aarushik93 Oct 10, 2024
58a62c4
Merge branch 'master' into move-oauth-to-prisma-user
ntindle Oct 10, 2024
29352e8
temp adding supabase
aarushik93 Oct 10, 2024
818fde0
remove unused functions
aarushik93 Oct 16, 2024
55a085c
Merge branch 'dev' into move-oauth-to-prisma-user
ntindle Oct 17, 2024
62134f4
adding missed merge
ntindle Oct 17, 2024
8ee37d4
fix: commit hash for lock
ntindle Oct 17, 2024
f04f328
ci: fix lint
ntindle Oct 17, 2024
8c0dd41
fix: minor bugs that prevented connecting and migrating to dbs and auth
ntindle Oct 17, 2024
ce3df9f
fix: linting
ntindle Oct 17, 2024
b008a53
fix: missed await
ntindle Oct 17, 2024
83fda8c
Merge branch 'dev' into move-oauth-to-prisma-user
majdyz Oct 18, 2024
9c07633
fix(backend): phase one pr updates
ntindle Oct 18, 2024
b624510
fix: handle error with returning user object from database_manager
ntindle Oct 18, 2024
b854e16
fix: linting
ntindle Oct 21, 2024
b776fb1
Merge branch 'dev' into move-oauth-to-prisma-user
ntindle Oct 22, 2024
fce9e01
Address comments
majdyz Oct 22, 2024
70cbe29
Make the migration safe
majdyz Oct 22, 2024
65f6371
Update migration doc
majdyz Oct 22, 2024
802f597
Move misplaced model functions
majdyz Oct 22, 2024
94dcfde
Grammar
majdyz Oct 22, 2024
5a8c097
Revert lock
majdyz Oct 22, 2024
2c596e1
Remove irrelevant changes
majdyz Oct 22, 2024
0779b3e
Remove irrelevant changes
majdyz Oct 22, 2024
d6346cf
Merge branch 'dev' into move-oauth-to-prisma-user
aarushik93 Oct 22, 2024
fbaa723
Avoid adding trigger on public schema
majdyz Oct 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import secrets
from datetime import datetime, timedelta, timezone
from typing import TYPE_CHECKING, cast
from typing import TYPE_CHECKING

if TYPE_CHECKING:
from redis import Redis
from supabase import Client
from backend.executor.database import DatabaseManager

from autogpt_libs.utils.synchronize import RedisKeyedMutex

Expand All @@ -18,8 +18,8 @@


class SupabaseIntegrationCredentialsStore:
def __init__(self, supabase: "Client", redis: "Redis"):
self.supabase = supabase
def __init__(self, redis: "Redis", db: "DatabaseManager"):
self.db_manager: DatabaseManager = db
self.locks = RedisKeyedMutex(redis)

def add_creds(self, user_id: str, credentials: Credentials) -> None:
Expand All @@ -35,7 +35,9 @@ def add_creds(self, user_id: str, credentials: Credentials) -> None:

def get_all_creds(self, user_id: str) -> list[Credentials]:
user_metadata = self._get_user_metadata(user_id)
return UserMetadata.model_validate(user_metadata).integration_credentials
return UserMetadata.model_validate(
user_metadata.model_dump()
).integration_credentials

def get_creds_by_id(self, user_id: str, credentials_id: str) -> Credentials | None:
all_credentials = self.get_all_creds(user_id)
Expand Down Expand Up @@ -90,9 +92,7 @@ def delete_creds_by_id(self, user_id: str, credentials_id: str) -> None:
]
self._set_user_integration_creds(user_id, filtered_credentials)

async def store_state_token(
self, user_id: str, provider: str, scopes: list[str]
) -> str:
def store_state_token(self, user_id: str, provider: str, scopes: list[str]) -> str:
token = secrets.token_urlsafe(32)
expires_at = datetime.now(timezone.utc) + timedelta(minutes=10)

Expand All @@ -105,17 +105,17 @@ async def store_state_token(

with self.locked_user_metadata(user_id):
user_metadata = self._get_user_metadata(user_id)
oauth_states = user_metadata.get("integration_oauth_states", [])
oauth_states = user_metadata.integration_oauth_states
oauth_states.append(state.model_dump())
user_metadata["integration_oauth_states"] = oauth_states
user_metadata.integration_oauth_states = oauth_states

self.supabase.auth.admin.update_user_by_id(
user_id, {"user_metadata": user_metadata}
self.db_manager.update_user_metadata(
user_id=user_id, metadata=user_metadata
)

return token

async def get_any_valid_scopes_from_state_token(
def get_any_valid_scopes_from_state_token(
self, user_id: str, token: str, provider: str
) -> list[str]:
"""
Expand All @@ -126,7 +126,7 @@ async def get_any_valid_scopes_from_state_token(
THE CODE FOR TOKENS.
"""
user_metadata = self._get_user_metadata(user_id)
oauth_states = user_metadata.get("integration_oauth_states", [])
oauth_states = user_metadata.integration_oauth_states

now = datetime.now(timezone.utc)
valid_state = next(
Expand All @@ -145,10 +145,10 @@ async def get_any_valid_scopes_from_state_token(

return []

async def verify_state_token(self, user_id: str, token: str, provider: str) -> bool:
def verify_state_token(self, user_id: str, token: str, provider: str) -> bool:
with self.locked_user_metadata(user_id):
user_metadata = self._get_user_metadata(user_id)
oauth_states = user_metadata.get("integration_oauth_states", [])
oauth_states = user_metadata.integration_oauth_states

now = datetime.now(timezone.utc)
valid_state = next(
Expand All @@ -165,10 +165,8 @@ async def verify_state_token(self, user_id: str, token: str, provider: str) -> b
if valid_state:
# Remove the used state
oauth_states.remove(valid_state)
user_metadata["integration_oauth_states"] = oauth_states
self.supabase.auth.admin.update_user_by_id(
user_id, {"user_metadata": user_metadata}
)
user_metadata.integration_oauth_states = oauth_states
self.db_manager.update_user_metadata(user_id, user_metadata)
return True

return False
Expand All @@ -177,19 +175,13 @@ def _set_user_integration_creds(
self, user_id: str, credentials: list[Credentials]
) -> None:
raw_metadata = self._get_user_metadata(user_id)
raw_metadata.update(
{"integration_credentials": [c.model_dump() for c in credentials]}
)
self.supabase.auth.admin.update_user_by_id(
user_id, {"user_metadata": raw_metadata}
)
raw_metadata.integration_credentials = [c.model_dump() for c in credentials]
self.db_manager.update_user_metadata(user_id, raw_metadata)

def _get_user_metadata(self, user_id: str) -> UserMetadataRaw:
response = self.supabase.auth.admin.get_user_by_id(user_id)
if not response.user:
raise ValueError(f"User with ID {user_id} not found")
return cast(UserMetadataRaw, response.user.user_metadata)
metadata: UserMetadataRaw = self.db_manager.get_user_metadata(user_id=user_id)
return metadata

def locked_user_metadata(self, user_id: str):
key = (self.supabase.supabase_url, f"user:{user_id}", "metadata")
key = (self.db_manager, f"user:{user_id}", "metadata")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return self.locks.locked(key)
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ class OAuthState(BaseModel):
token: str
provider: str
expires_at: int
scopes: list[str]
"""Unix timestamp (seconds) indicating when this OAuth state expires"""


Expand All @@ -64,6 +65,6 @@ class UserMetadata(BaseModel):
integration_oauth_states: list[OAuthState] = Field(default_factory=list)


class UserMetadataRaw(TypedDict, total=False):
integration_credentials: list[dict]
integration_oauth_states: list[dict]
class UserMetadataRaw(BaseModel):
integration_credentials: list[dict] = Field(default_factory=list)
integration_oauth_states: list[dict] = Field(default_factory=list)
2 changes: 1 addition & 1 deletion autogpt_platform/backend/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ WORKDIR /app

# Install build dependencies
RUN apt-get update \
&& apt-get install -y build-essential curl ffmpeg wget libcurl4-gnutls-dev libexpat1-dev gettext libz-dev libssl-dev postgresql-client git \
&& apt-get install -y build-essential curl ffmpeg wget libcurl4-gnutls-dev libexpat1-dev libpq5 gettext libz-dev libssl-dev postgresql-client git \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

Expand Down
4 changes: 2 additions & 2 deletions autogpt_platform/backend/README.advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ We use the Poetry to manage the dependencies. To set up the project, follow thes
5. Generate the Prisma client

```sh
poetry run prisma generate --schema postgres/schema.prisma
poetry run prisma generate
```


Expand All @@ -61,7 +61,7 @@ We use the Poetry to manage the dependencies. To set up the project, follow thes

```sh
cd ../backend
prisma migrate dev --schema postgres/schema.prisma
prisma migrate deploy
```

## Running The Server
Expand Down
2 changes: 1 addition & 1 deletion autogpt_platform/backend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ We use the Poetry to manage the dependencies. To set up the project, follow thes

```sh
docker compose up db redis -d
poetry run prisma migrate dev
poetry run prisma migrate deploy
```

## Running The Server
Expand Down
20 changes: 20 additions & 0 deletions autogpt_platform/backend/backend/data/user.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
from typing import Optional

from autogpt_libs.supabase_integration_credentials_store.types import UserMetadataRaw
from fastapi import HTTPException
from prisma import Json
from prisma.models import User

from backend.data.db import prisma
Expand Down Expand Up @@ -48,3 +50,21 @@ async def create_default_user(enable_auth: str) -> Optional[User]:
)
return User.model_validate(user)
return None


async def get_user_metadata(user_id: str) -> UserMetadataRaw:
user = await User.prisma().find_unique_or_raise(
where={"id": user_id},
)
return (
UserMetadataRaw.model_validate(user.metadata)
if user.metadata
else UserMetadataRaw()
)


async def update_user_metadata(user_id: str, metadata: UserMetadataRaw):
await User.prisma().update(
where={"id": user_id},
data={"metadata": Json(metadata.model_dump())},
)
5 changes: 5 additions & 0 deletions autogpt_platform/backend/backend/executor/database.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
)
from backend.data.graph import get_graph, get_node
from backend.data.queue import RedisEventQueue
from backend.data.user import get_user_metadata, update_user_metadata
from backend.util.service import AppService, expose
from backend.util.settings import Config

Expand Down Expand Up @@ -73,3 +74,7 @@ def wrapper(self, *args: P.args, **kwargs: P.kwargs) -> R:
Callable[[Any, str, int, str, dict[str, str], float, float], int],
exposed_run_and_wait(user_credit_model.spend_credits),
)

# User + User Metadata
get_user_metadata = exposed_run_and_wait(get_user_metadata)
update_user_metadata = exposed_run_and_wait(update_user_metadata)
9 changes: 5 additions & 4 deletions autogpt_platform/backend/backend/executor/manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
from backend.data.model import CREDENTIALS_FIELD_NAME, CredentialsMetaInput
from backend.integrations.creds_manager import IntegrationCredentialsManager
from backend.util import json
from backend.util.cache import thread_cached_property
from backend.util.cache import thread_cached
from backend.util.decorator import error_logged, time_measured
from backend.util.logging import configure_logging
from backend.util.process import set_service_name
Expand Down Expand Up @@ -417,7 +417,7 @@ def on_node_executor_start(cls):
redis.connect()
cls.pid = os.getpid()
cls.db_client = get_db_client()
cls.creds_manager = IntegrationCredentialsManager()
cls.creds_manager = IntegrationCredentialsManager(db_manager=cls.db_client)

# Set up shutdown handlers
cls.shutdown_lock = threading.Lock()
Expand Down Expand Up @@ -670,7 +670,7 @@ def run_service(self):
)

self.credentials_store = SupabaseIntegrationCredentialsStore(
self.supabase, redis.get_redis()
redis=redis.get_redis(), db=self.db_client
)
self.executor = ProcessPoolExecutor(
max_workers=self.pool_size,
Expand Down Expand Up @@ -701,7 +701,7 @@ def cleanup(self):

super().cleanup()

@thread_cached_property
@property
def db_client(self) -> "DatabaseManager":
return get_db_client()

Expand Down Expand Up @@ -857,6 +857,7 @@ def _validate_node_input_credentials(self, graph: Graph, user_id: str):
# ------- UTILITIES ------- #


@thread_cached
def get_db_client() -> "DatabaseManager":
from backend.executor import DatabaseManager

Expand Down
11 changes: 6 additions & 5 deletions autogpt_platform/backend/backend/integrations/creds_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,10 @@
from redis.lock import Lock as RedisLock

from backend.data import redis
from backend.executor.database import DatabaseManager
from backend.integrations.oauth import HANDLERS_BY_NAME, BaseOAuthHandler
from backend.util.settings import Settings

from ..server.integrations.utils import get_supabase

logger = logging.getLogger(__name__)
settings = Settings()

Expand Down Expand Up @@ -51,10 +50,12 @@ class IntegrationCredentialsManager:
cause so much latency that it's worth implementing.
"""

def __init__(self):
def __init__(self, db_manager: DatabaseManager):
redis_conn = redis.get_redis()
self._locks = RedisKeyedMutex(redis_conn)
self.store = SupabaseIntegrationCredentialsStore(get_supabase(), redis_conn)
self.store = SupabaseIntegrationCredentialsStore(
redis=redis_conn, db=db_manager
)

def create(self, user_id: str, credentials: Credentials) -> None:
return self.store.add_creds(user_id, credentials)
Expand Down Expand Up @@ -131,7 +132,7 @@ def delete(self, user_id: str, credentials_id: str) -> None:

def _acquire_lock(self, user_id: str, credentials_id: str, *args: str) -> RedisLock:
key = (
self.store.supabase.supabase_url,
self.store.db_manager,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless self.store.db_manager magically stringifies to a string representing the DB connection, this doesn't work. Can be fixed by either of:

  • Replace self.store.db_manager with a variable uniquely identifying the DB that the credentials are stored in
  • Omit self.store.db_manager if we can absolutely assume that IntegrationCredentialsStore will never be used with multiple different DBs within the same system

f"user:{user_id}",
f"credentials:{credentials_id}",
*args,
Expand Down
10 changes: 6 additions & 4 deletions autogpt_platform/backend/backend/server/integrations/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from fastapi import APIRouter, Body, Depends, HTTPException, Path, Query, Request
from pydantic import BaseModel, Field, SecretStr

from backend.executor.manager import get_db_client
from backend.integrations.creds_manager import IntegrationCredentialsManager
from backend.integrations.oauth import HANDLERS_BY_NAME, BaseOAuthHandler
from backend.util.settings import Settings
Expand All @@ -19,7 +20,8 @@
logger = logging.getLogger(__name__)
settings = Settings()
router = APIRouter()
creds_manager = IntegrationCredentialsManager()

creds_manager = IntegrationCredentialsManager(db_manager=get_db_client())


class LoginResponse(BaseModel):
Expand All @@ -41,7 +43,7 @@ async def login(
requested_scopes = scopes.split(",") if scopes else []

# Generate and store a secure random state token along with the scopes
state_token = await creds_manager.store.store_state_token(
state_token = creds_manager.store.store_state_token(
user_id, provider, requested_scopes
)

Expand Down Expand Up @@ -70,12 +72,12 @@ async def callback(
handler = _get_provider_oauth_handler(request, provider)

# Verify the state token
if not await creds_manager.store.verify_state_token(user_id, state_token, provider):
if not creds_manager.store.verify_state_token(user_id, state_token, provider):
logger.warning(f"Invalid or expired state token for user {user_id}")
raise HTTPException(status_code=400, detail="Invalid or expired state token")

try:
scopes = await creds_manager.store.get_any_valid_scopes_from_state_token(
scopes = creds_manager.store.get_any_valid_scopes_from_state_token(
user_id, state_token, provider
)
logger.debug(f"Retrieved scopes from state token: {scopes}")
Expand Down
3 changes: 2 additions & 1 deletion autogpt_platform/backend/backend/server/rest_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from backend.data.credit import get_block_costs, get_user_credit_model
from backend.data.user import get_or_create_user
from backend.executor import ExecutionManager, ExecutionScheduler
from backend.executor.manager import get_db_client
from backend.integrations.creds_manager import IntegrationCredentialsManager
from backend.server.model import CreateGraph, SetGraphActiveVersion
from backend.util.cache import thread_cached_property
Expand Down Expand Up @@ -97,7 +98,7 @@ def run_service(self):
tags=["integrations"],
dependencies=[Depends(auth_middleware)],
)
self.integration_creds_manager = IntegrationCredentialsManager()
self.integration_creds_manager = IntegrationCredentialsManager(get_db_client())

api_router.include_router(
backend.server.routers.analytics.router,
Expand Down
Loading
Loading