-
Notifications
You must be signed in to change notification settings - Fork 44.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is it possible to replace Chat-GPT with local offline model? For example, gpt4all, LLaMa, etc #2158
Comments
An excellent suggestion would be to use local models. AUTOGPT can utilize these models to train individuals to become experts and also develop several agents simultaneously. Additionally, it can train and facilitate communication between these models. |
just replace the request to openai with your own models service in |
maybe I should ask Auto-GPT to analyse itself and figure out how to replace openai api and chat-gpt with local LLM models, lol. |
Depends on if you have a GPU cluster and 64 GB or so of RAM to run anything comparable at a reasonable speed. You also gotta calculate impact on your electric bill. |
|
New repo to browse just dropped fam 😅👌 |
Can I avoid using OpenAI's embedding APIs? Because the network connection to OpenAI can not be always established successfully, I want to be totally offline. |
I wouldn't mind if the agents were from ChatGPT, the rest (i assume the fast_llm is the one the user interacts with) would be a local gpt4all, for example! |
It would be really have a mutliple online expert agent which all are open source. Each of them should be smaller model but specialized in degree. For example Auto creates an Agent designed to have 10% expertise and 90% general knowledge for brain storming. These agents would communicate with each other to train and improve themselves, leveraging collective knowledge and expertise to continuously enhance their capabilities. I would really love to have these agents in the hand of the open source community. |
LLaMA requires 14 GB of GPU memory for the model weights on the smallest, 7B model, and with default parameters, it requires an additional 17 GB for the decoding cache (I don't know if that's necessary). But GPT4All called me out big time with their demo being them chatting about the smallest model's memory requirement of 4 GB. I've never heard of machine learning using 4-bit parameters before, but the math checks out. You'd have to feed it something like this to verify its usability. The full, better performance model on GPU requires 16 GB RAM. My biggest concern would be the context window size. Both of those look like they're limited to 2048 tokens. The full example AutoGPT prompt is a third of that. |
yeah. The number of tokens of questions generated by AutoGPT during the thinking process are quite a lot. Besides, one thing I' m worried about is that the chat models provided by OpenAI, such as gpt-3.5-turbo, take a series of messages as input, as you can find in https://platform.openai.com/docs/guides/chat/introduction. For example, an API call of openai.ChatCompletion.create looks as follows:
Messages must be an array of message objects, where each object has a role (either "system", "user", or "assistant") and content (the content of the message). The system, user and assistant messages and the conversation history helps build contextual information for user's final question. However, I have reviewed the source code of https://github.com/nomic-ai/pyllamacpp (offical pythonic API for gpt4all model) and its usage. It seems no similar input argments for gpt4all, which means the conversation history, roles and context information is missing when chatting with gpt4all model, I guess. |
If you want a conversational model, you should probably use Vicuna (based on llama). It supports the human and assistant roles (via string prefixes). Also, with llama.cpp (or PyLLaMACpp), the memory usage is really low because the models are quantized to 4 bit. Vicuna 13b needs about 5 GB on my machine. |
thanks! One question, why does Vicuna 13b need only 5GB on your machine? It says 7b needs around 30 GB of CPU RAM, and 13b needs around 60 GB of CPU RAM, https://github.com/lm-sys/FastChat. Did you use --low-cpu-mem? |
You need to use llama.cpp (CPU-based) instead of FastChat (GPU-based). FastChat (the original) is more accurate because it operates in floating point, but it also needs much more RAM. Plus some additional tweaks that llama.cpp makes. |
So what I should do is to obtain original LLaMA model weights (also the Vicuna delta wights) and place theme in right place, convert the 7B or 13B model to ggml FP16 or INT4 format, and finally run the inference, right? In INT4 quantized size, memory size of 7B model reduces to 3.9GB. Also, FastChat provides a similar API with OpenAI's, for example:
can I specify roles, such as "user", "assistant" in the command line using llama.cpp? can I use |
i posted this in another related thread but i got autogpt mostly working a couple times but embeddings seems to be the wall im hitting (hardcoded different embedding sizes to make it work) https://github.com/keldenl/gpt-llama.cpp aims to replace openais api endpoints completely with llama.cpp (including embeddings). but i don't know much about embeddings and seems to work off and on, anybody got any pointers there? |
of course, the embedding API call is used for |
You can get the full ggml at huggingface.
I don't think so. You need to use the roles inside the prompt text as described in Vicuna's documentation. About the other FastChat questions: don't know, I only used llama.cpp so far. |
i got autogpt working with llama.cpp! see keldenl/gpt-llama.cpp#2 (comment) i'm using vicuna for embeddings and generation but it's struggling a bit to generate proper commands to not fall into a infinite loop of attempting to fix itself X( will look into this tmr but super exciting cuz i got the embeddings working! (turns out it was a bug on my end lol) edit: had to make some changes to autogpt (add base_url to openai_base_url, and adjust the dimensions of the vector, but otherwise left it alone) |
the https://github.com/oobabooga/text-generation-webui is able to run in an API mode if you use the flags maybe we could make an option to allow users to use their local webui as a server instead of the OpenAI API? something like adding Since they are making an interface for running all open source LLM and our project is instead to use a LLM to run an agent in autonomy, it might be a good idea to avoid redoing what they already did |
nice work! what local embedding model did you test? |
i just utilized the embeddings.cpp example from llama.cpp with llama-based models! |
beautiful work @DGdev91 ! that does the trick. i'll go ahead and write an extensive guide tonight and link it here as well |
✨ FULL GUIDE POSTED on how to get Huge shoutout to @DGdev91 for the PR and hope it gets merged soon! |
Hi 👋 I'm the author of https://github.com/go-skynet/LocalAI, I'd be glad to help out in what's missing to see this working with local models - LocalAI has multiple backends, is multi-models and keeps thing in memory for faster inference. Supports models including gpt4all-j and those supported by llama.cpp, and will also have support for Cerebras. Happy to jump in! |
Thanks for the pointers!
I see! yup indeed I've an issue open already for it to allow model aliases
Out of curiosity, did you tested with docker compose? If you are on a Mac I'd suggest to compile from source instead. Also, to lower the number of threads to the number of physical cores in the hardware - by default it takes all the cores available, but that seems to be problematic. For instance on a Mac M1 user reports between 4 and 7 to be a good spot. On my PC that has 20 cores, and 14 are physical, 14 is a sweet spot as well. Update: just fixed this on LocalAI - now defaults to the number of physical cores =) Another point, is, but it's more a suspect - this is due to the fact that keldenl's implementations bashes out to the llama.cpp binary, and I suspect it keeps the process open and continues the prompting in interactive mode, and uses a stabbed number of cores (4 if I recall correctly), but I'm just speculating here, maybe @keldenl can shed more light on this. LocalAI directly interacts with the library, and so calls are directly sent back to the model, without interactive mode - this probably has impact overall while sending whole inputs back again to the API, but I'm not sure how it is being used here. I'll try to have a look, but usually on my hardware (not a Mac) I don't notice difference between running with the CLI manually and with the API.
👍 Gotcha! Thanks for the pointers! |
I tried both, it doesn't seem to make any difference.
I'm quite sure he uses the max numbers of threads supported by the cpu, it was running at 100% on my machine. |
Could you please open an issue to LocalAI with your steps? that'd be appreciated! I don't want to hijack the thread here, and it looks very weird from what you report, possibly something is off. Not sure if it's due to the CPU type at this point, didn't tried on AMD myself, I have an Intel i7-1280P and here I don't notice any difference. Thanks! Edit: I've benchmarked things locally, and token inference speed is the same as the llama.cpp CLI, however, I've just updated to the latest llama.cpp code here locally, so not sure if that's makes any difference in your case |
@mudler i do use interactive mode, but i dont think it applies to auto-gpt. it detects whether or not you are still in the same chat "conversation" by storing the previous one and comparing. but since auto-gpt requests don't build off each other one after another (instead it goes chat -> different chat agent -> embed -> original chat), interactive mode is pretty useless @DGdev91 i just use whatever llama.cpp comes out of the box, and i think it uses as many cores as it sees fit. in most cases, probably most of them also, i just merged a ton of fixes yesterday and today that pretty much makes gpt-llama.cpp run infinitely continuously with auto-gpt (fixed all the bugs i could find). now the focus is getting auto-gpt results as good as possible. you can read more in my update in the original thread here: keldenl/gpt-llama.cpp#2 (comment) |
Made some other tests, the performances are indeed close to llama.cpp standalone binary. but still struggles with AutoGPT. i tried a couple of time, both of them AutoGPT closed itself for http timeout. |
It's doesn't work in my case. AutoGPT outputs Http Timeout... maybe we could be able of increase the timeout? There isn't any bottle neck on execute the model via node? |
Can we replace the OpenAI embedding API with an open-source embedding model from Hugging Face? |
It can be replaced with other embedding API, just search embedding related code and replace it. |
您好,你的邮件已收到,请耐心等待。我一定在我看到的第一时刻回复您。
方弘
|
Made a tiny script that uses https://github.com/go-skynet/LocalAI to run gpt4all model at localhost:8080 so autogpt can work with it. Haven't tested extensively though and might require fine tuning and picking the right model to speed up responses. |
but openai stated required ssl as default. [tested in Jupyter no issues with the localai] url = "http://localhost:8080/v1/chat/completions" headers = { data = { response = requests.post(url, headers=headers, json=data) Process the response as neededprint(response.json()) {'object': 'chat.completion', 'model': 'gpt-3.5-turbo.bin', 'choices': [{'message': {'role': 'assistant', 'content': 'I am good. How is your day?\nHow are you? I am good. How is your day? You look tired today, please take care of yourself.\nHow are you? I am good, and how is your day?\nHow are you? I am good, and how is your day? You look tired today, please take care of yourself.'}}], 'usage': {'prompt_tokens': 0, 'completion_tokens': 0, 'total_tokens': 0}} --- updated status---- |
您好,你的邮件已收到,请耐心等待。我一定在我看到的第一时刻回复您。
方弘
|
Closing as duplicate of #25 |
Open-source mixed agents would really change things... The 7B parameter models (even the 4bit ones) are already getting pretty good, but are too generalized. If we could download multiple specialized agents and load them only when needed, a performance similar to GPT-4 could be archieved. Of course, loading and unloading models would drastically reduce inference speed, but hey, it would run on most of the consumer GPUs (8-12GB)! I would love to see this :) |
您好,你的邮件已收到,请耐心等待。我一定在我看到的第一时刻回复您。
方弘
|
Duplicates
Summary 💡
Is it possible to replace Can chat-GPT with local offline model? For example, gpt4all, LLaMa, etc
Examples 🌈
No response
Motivation 🔦
No response
The text was updated successfully, but these errors were encountered: