TLDR - Exploring SAM with Python
We need to grab reel information from the SlateAPP API, but the response time from the API end points is not performant and since the data from that API doesn't change much, we need a caching mechanism in between - this is where this mini project comes in.
Let's go serverless!
What we need is a simple API to connect to and get the reels information fast! We also want to be able to tell this API to cache the data either thru user action or on schedule (daily).
This project provide 2 entry points:
- /reel/{id} - which will get you the cache reel information or will get the reel information in real-time (pass thru) if the data is not available in cache
- /cache - which will cache all the reel data
The solution will use the following technologies:
- SAM
- Lambda
- API Gateway
- SNS
- DynomoDB
- SSM
- Cognito
- SES - (see TODO)
- Cloud Watch (see TODO)
- CodeBuild
Yes - there is probably better ways to do this, and potentially even get API Gateway to cache, but this is an education related project, where I am trying to familiarise myself with all AWS.
- Figure out how to properly structure project
- Find out how to use request library, which Lambda doesn't have
- Find out how to add API Gateway
- Update Dynomodb
- Workout how to get SSM working to remove API Key from variable/repo
- Finish DynomoDB integration via SAM template
- Work out how to monitor SNS Topic, and consumption of messages
- getReel, workout how to handle cache for items not in DynomoDB (invoke lambda? Or is there better?)
- Add Cognito to API Gateway, so only auth users can invoke it
- If anything goes wrong, we need to email the user (cache didn't work, etc)
- Cloud Watch event setup, so that cache is invoked at regular time
- Break template.yaml into sub template for better maintenance
- Work out how to use pytest and create a unit test
- Add tests, lots of tests
- Add various Env parameter (Prod, Stage, Dev)
- Proper use of ENV parameter (Prod, Stage, Dev)
- Add Output to template
- Make LayerVersion work, need to read up on how to use it properly
- CI/CD thru Github Actions
- CI/CD thru CodeBuild
- Add versioning
- Work out how to use Layer functions in tests
- Work out how to trigger tests in CI/CD
- Cognito - find out how to setup HostedUI + App client full setup via CloudFormation/SAM
- get Reel function, handle better when no reel found
To run local tests: python3 -m venv .venv source .venv/bin/activate pip install --upgrade pip pip install -r tests/requirements.txt export PYTHONPATH=$PWD/src/library:$PWD/src:$PYTHONPATH python3 -m pytest tests
Things I need to remember:
- export AWS_PROFILE='profile' before running sam deploy
- SSM Parameter Store "REEL_API_KEY" should match the "Default" value
- "is in ROLLBACK_COMPLETE state and can not be updated." - need to manual remove the stack, seems the stack is empty, but its in a state, that cannot be deployeed, there is a way to use create-stack --on-failure, but that doesn't work with deploy.
- You can view all logs related to Lambda via lambda/home -> Monitoring
- To check your items in Dynomodb in the aws console UI ensure to refresh the page :)
- Always check what Policies is required for the functions
- "An error occurred (ExpiredToken) when calling the PutObject operation: The provided token has expired." -> export AWS_PROFILE=
https://aws.amazon.com/blogs/compute/introducing-aws-sam-pipelines-automatically-generate-deployment-pipelines-for-serverless-applications/ https://aws.plainenglish.io/a-practical-guide-surviving-aws-sam-part-3-lambda-layers-8a55eb5d2cbe https://hands-on.cloud/how-to-test-python-lambda-functions/
https://github.com/aws-samples/aws-serverless-workshops https://stackoverflow.com/questions/65270647/running-pytest-in-aws-sam-doesnt-use-env-vars-in-template-yaml https://tenmilesquare.com/resources/software-development/aws-sam-api-with-cognito/