Add ability to use local embeddings model

[Tymec]

Background

This change adds the ability to use a local embeddings model (sBERT) using the SentenceTransformers library. This change gives the users an alternative to OpenAI's ada embeder which costs money.

Changes

• Renamed the get_ada_embedding function to get_embedding
• get_embedding function now uses either ada or sBERT embeder based on memory_embeder config option
• New config option called memory_embeder that defaults to ada, which can also be configured by setting the MEMORY_EMBEDER environment variable.
• Added sentence_transformers as a requirement

Documentation

• In-code comments that explain the logic and functionality of the code.

Test Plan

• Created a unit test module to test if everything works as intended.

PR Quality Checklist

• My pull request is atomic and focuses on a single change.
• I have thoroughly tested my changes with multiple different prompts.
• I have considered potential risks and mitigations for my changes.
• I have documented my changes clearly and comprehensively.
• I have not snuck in any "extra" small tweaks changes

[FlyWieEinAirplane]

Sounds useful.
Looking at the code, it would be easy to add a third option but it seems to be a bit more complicated to add a third embeddings dimension option. Maybe this could be improved a bit.

[Tymec]

Thanks for the feedback. I agree that it's not a good approach and only did so as a temporary solution which I overlooked.

To improve it, I created a variable that holds a dictionary with the embeder name as the key and the dimension as the value. This variable is dynamically assigned to the dimension based on the embeder used at runtime. Then, this variable is imported and used in memory backends.

[nponeccop]

There is a crazy EOF error now too. I'm approving.

[Tymec]

Just noticed and patched some issues with the test unit, as well as resolved issues with the new EMBED_DIM variable in memory.base module.

[nponeccop]

@Tymec There are conflicts now

[nponeccop]

@Tymec There are conflicts now

[richbeales]

Run flake8 autogpt/ tests/ --select E303,W293,W291,W292,E305,E231,E302
autogpt/config/config.py:85:1: W293 blank line contains whitespace
autogpt/memory/base.py:46:1: W293 blank line contains whitespace
autogpt/memory/base.py:48:1: W293 blank line contains whitespace