Separate fast and smart llm providers

[kesamet]

Different LLM sources for "SMART" and "FAST".
For issues #702, #598

[assafelovic]

Hey @kesamet sorry it took me long to reply here. First of all this is super super valuable!! We've been getting a lot of requests to use different LLMs for different actions. A bit of feedback:

1. resolve conflicts based on latest changes
2. I think it's in the right direction but I'd challenge a solution a bit more complex. Check out how retrievers is built. You can add as many retrievers in comma seperated and they're all populated. Check out here: https://github.com/assafelovic/gpt-researcher/blob/master/gpt_researcher/config/config.py#L54C9-L54C25
3. I think it might be better to rewrite the config param llm_provider to support multiple llms.
4. Then we can use the already existing params fast_llm_model and smart_llm_model automatically.

This approach is probably in the right direction and would be optimal. Lmk if you're up for it or I can take it from here!

[kesamet]

Hey @assafelovic, thanks for the feedback and sorry for the late reply.

Taking inspiration from AWS Bedrock where the model name is something like "anthropic.claude-3-sonnet-20240229-v1:0", I think it is best to combine LLM_PROVIDER and LLM_MODEL in env vars into a combined name "<llm_provider>:<llm_model>" using semi-colon. I call it "FAST_LLM_NAME" and "SMART_LLM_NAME".

Eg

FAST_LLM_NAME = openai:gpt-4o-mini
SMART_LLM_NAME = openai:gpt-4o

config/config.py will then split the names into smart_llm_provider, smart_llm_model, etc params

self.fast_llm_provider = "openai"
self.smart_llm_model = "gpt-4o-mini"
self.smart_llm_provider = "openai"
self.smart_llm_model = "gpt-4o"

What do you think?

[assafelovic]

This actually sounds great! Will require some refactoring but love it

[kesamet]

Please help to review the PR.
We can also do the same simplification for embedding env vars, which I can do in another PR.

[assafelovic]

Embeddings would be amazing @kesamet as well. Thank you for your contributions, I will dive into this PR in the coming days

[assafelovic]

Hey @kesamet is this ready for review?

[kesamet]

@assafelovic yes

[kesamet]

But I noticed that fast_llm_model seems to be no longer in use, right?

[assafelovic]

@kesamet it was used to summarize retrieved articles but since we're using embedding retrieval it's in no use right now. Still worth having the configs in case we find other use cases for it

[assafelovic]

@kesamet everything looks great, may i ask for one last revision and remove the _NAME from the vars? I think simply FAST_LLM is enough. Thank you and I'll merge!

[kesamet]

@assafelovic Done!

[assafelovic]

Thank you @kesamet this is huge and will open many oppurtunities! I'll send an update on this to the community in a few days