Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: add Intel MKL compilation for building library with server example #99

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

xiangyang-95
Copy link

@xiangyang-95 xiangyang-95 commented Jun 17, 2024

This pull request adds steps to build library with server example with Intel MKL backend.
This will also add support for AVX_VNNI hardware instruction sets in Intel CPU.

@xiangyang-95 xiangyang-95 changed the title DOC: add Intel MKL compilation for building library with server example doc: add Intel MKL compilation for building library with server example Jun 17, 2024
@vansangpfiev
Copy link
Contributor

Thanks for your PR. We will review it soon.
Seems like the PR has some conflicts, could you please help to fix them?

@xiangyang-95
Copy link
Author

@vansangpfiev I have resolved the conflict. Feel free to let me know if need any info or modifications.

@vansangpfiev vansangpfiev requested a review from hiento09 June 20, 2024 01:41
@xiangyang-95
Copy link
Author

Hi @hiento09, may I know are there any changes needed? I am also planned to contribute with a version running on SYCL as well

@dan-homebrew
Copy link

@xiangyang-95 FYI, we will be working on Intel in a couple of sprints - thank you for the patience as we get things in order 🙏

@xiangyang-95
Copy link
Author

Hi @dan-homebrew, I am actually from Intel team. Please let me know if anything I can help.

@dan-homebrew
Copy link

Hi @dan-homebrew, I am actually from Intel team. Please let me know if anything I can help.

Hey yes I saw! We also spoke with a few of your colleagues.

On our part, we need to take a bit of time to refactor our implementation of cortex.llama.cpp in Jan:

Jan

  • We will allow users to download multiple versions of llama.cpp (e.g. compiled with sycl, avx2)
  • We will have a UI which allows user to select which version of llama.cpp they want to use
  • We will pre-bundle several versions of llama.cpp, and make additional ones (e.g. sycl available for pull)

Cortex.cpp

  • For cortex.cpp, most of the work is right now infrastructural as we bootstrap the CLI and basic functions
  • We are implementing cortex engines install, which right now relies on hardware auto-detection
  • cortex engines install llama.cpp will need to expand to support the different llama.cpp variants

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants