Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: model parameter threads doesn't work #114

Closed
3 tasks
pafik13 opened this issue Dec 7, 2023 · 5 comments
Closed
3 tasks

bug: model parameter threads doesn't work #114

pafik13 opened this issue Dec 7, 2023 · 5 comments
Assignees
Labels
bug Something isn't working good first issue Good for newcomers released on @beta released
Milestone

Comments

@pafik13
Copy link

pafik13 commented Dec 7, 2023

Issue description

It seems to me that parameter threads doesn't work as expected

Expected Behavior

If I have 24 CPUs and pass threads:24 then all CPUs should be utilized. II tried call original llama.cpp with argument -t 24 and it works normally as expected.

Actual Behavior

I pass parameter thread: 24 or 1 to constructor and nothing is changed: always it starts utilize 4 CPUs upper 80% and sometimes use 1-2 additional with 25-50% utilization.

Steps to reproduce

Try to pass different threads value to model constructor and observe CPUs utilization (for example, htop)

My Environment

Dependency Version
Operating System Ubuntu 22.04.3 LTS
CPU AMD EPYC 7742 64-Core Processor
Node.js version v20.10.0
Typescript version no use
node-llama-cpp version 3.0.0-beta.1 & v2.8.1

Additional Context

./llama.cpp/main -m ./catai/models/phind-codellama-34b-q3_k_s -p "Please, write JavaScript function to sort array" -ins -t 24

Screenshot from 2023-12-07 23-03-02

const model = new LlamaModel({
    modelPath: "/root/catai/models/phind-codellama-34b-q3_k_s",
    threads: 1,
});

Screenshot from 2023-12-07 23-04-41

const model = new LlamaModel({
    modelPath: "/root/catai/models/phind-codellama-34b-q3_k_s",
    threads: 24,
});

Screenshot from 2023-12-07 23-05-59

Relevant Features Used

  • Metal support
  • CUDA support
  • Grammar

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, but I don't know how to start. I would need guidance.

@pafik13 pafik13 added bug Something isn't working requires triage Requires triaging labels Dec 7, 2023
@giladgd
Copy link
Contributor

giladgd commented Dec 7, 2023

@pafik13 Thanks for the detailed issue, it helped me a lot to investigate this problem :)
I found the bug, I'll include the fix for it in the next beta

@giladgd giladgd added good first issue Good for newcomers and removed requires triage Requires triaging labels Dec 7, 2023
@giladgd giladgd added this to the v3.0.0 milestone Dec 16, 2023
@stewartoallen
Copy link

@giladgd can you point me to where you found the problem? if I can fix it locally, I can submit a PR.

@giladgd giladgd mentioned this issue Jan 17, 2024
7 tasks
Copy link

🎉 This issue has been resolved in version 3.0.0-beta.2 🎉

The release is available on:

Your semantic-release bot 📦🚀

@giladgd giladgd self-assigned this Jan 20, 2024
@giladgd giladgd closed this as completed Jan 20, 2024
@giladgd giladgd mentioned this issue Jan 20, 2024
17 tasks
Copy link

🎉 This issue has been resolved in version 3.0.0-beta.4 🎉

The release is available on:

Your semantic-release bot 📦🚀

Copy link

github-actions bot commented Sep 24, 2024

🎉 This PR is included in version 3.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers released on @beta released
Projects
None yet
Development

No branches or pull requests

3 participants