Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: function calling #139

Merged
merged 22 commits into from
Jan 19, 2024
Merged

feat: function calling #139

merged 22 commits into from
Jan 19, 2024

Conversation

giladgd
Copy link
Contributor

@giladgd giladgd commented Jan 17, 2024

Description of change

  • feat: function calling support
  • feat: chat syntax aware context shifting
  • feat: stateless LlamaChat
  • feat: improve chat wrapper
  • feat: LlamaText util
  • test: add basic model-dependent tests
  • fix: threads parameter
  • fix: disable Metal for x64 arch by default

Metal on macOS on Intel Macs isn't supported well by llama.cpp ATM, so disabling it by default will be better for most Intel macOS users.

Resolves #101
Fixes #114
Related: langchain-ai/langchainjs#3588 (LlamaChat)

How to use function calling

Support for function calling is still a work in progress, but as for now, Functionary models are supported.

import {fileURLToPath} from "url";
import path from "path";
import {LlamaModel, LlamaContext, LlamaChatSession} from "node-llama-cpp";

const __dirname = path.dirname(fileURLToPath(import.meta.url));

const model = new LlamaModel({
    modelPath: path.join(__dirname, "models", "functionary-small-v2.2.q4_0.gguf")
});
const context = new LlamaContext({
    model,
    contextSize: Math.min(4096, model.trainContextSize)
});
const functions = {
    getDate: defineChatSessionFunction({
        description: "Retrieve the current date",
        handler() {
            return new Date().toLocaleDateString();
        }
    }),
    getNthWord: defineChatSessionFunction({
        description: "Get an n-th word",
        params: {
            type: "object",
            properties: {
                n: {
                    enum: [1, 2, 3, 4]
                }
            }
        },
        handler(params) {
            return ["very", "secret", "this", "hello"][params.n - 1];
        }
    })
};
const session = new LlamaChatSession({
    contextSequence: context.getSequence()
});


const q1 = "What is the second word?";
console.log("User: " + q1);

const a1 = await session.prompt(q1, {functions});
console.log("AI: " + a1);


const q2 = "What is the date? Also tell me the word I previously asked for";
console.log("User: " + q2);

const a2 = await session.prompt(q2, {functions});
console.log("AI: " + a2);

In this example I used this model

Pull-Request Checklist

  • Code is up-to-date with the master branch
  • npm run format to apply eslint formatting
  • npm run test passes with this change
  • This pull request links relevant issues as Fixes #0000
  • There are new or updated unit tests validating the change - more tests will be added soon in another PR after some more breaking changes
  • Documentation has been updated to reflect this change - documentation will be updated in another PR
  • The new commits and pull request title follow conventions explained in pull request guidelines (PRs that do not follow this convention will not be merged)

@giladgd giladgd added this to the v3.0.0 milestone Jan 17, 2024
@giladgd giladgd requested a review from ido-pluto January 17, 2024 23:32
@giladgd giladgd self-assigned this Jan 17, 2024
Copy link
Contributor

@ido-pluto ido-pluto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

giladgd and others added 6 commits January 19, 2024 00:55
Metal on macOS on `x64` isn't supported well in `llama.cpp` ATM, so disabling it by default will be better for most Intel macOS users.
@giladgd giladgd merged commit 5fcdf9b into beta Jan 19, 2024
12 checks passed
@giladgd giladgd deleted the gilad/functionCalling branch January 19, 2024 22:11
Copy link

🎉 This PR is included in version 3.0.0-beta.2 🎉

The release is available on:

Your semantic-release bot 📦🚀

@giladgd giladgd mentioned this pull request Jan 20, 2024
17 tasks
Copy link

🎉 This PR is included in version 3.0.0-beta.4 🎉

The release is available on:

Your semantic-release bot 📦🚀

@giladgd giladgd mentioned this pull request Mar 16, 2024
7 tasks
Copy link

github-actions bot commented Sep 24, 2024

🎉 This PR is included in version 3.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

Successfully merging this pull request may close these issues.

2 participants