Nomic Vulkan backend #4456

cebtenzzre · 2023-12-13T23:29:43Z

This is Nomic's Kompute-based Vulkan backend from the GPT4All project, now available under the MIT license. It can be enabled by building with cmake and passing -DLLAMA_KOMPUTE=ON (make is currently not supported).

Structure

The C++ code for the backend is in ggml-kompute.cpp and ggml-kompute.h. This is MIT licensed.
Shaders are in a folder called kompute-shaders. These are based on the Metal backend and MIT licensed.
Nomic's fork of Kompute is provided as a git submodule. This is Apache-2.0 licensed.

Limitations

There is currently no partial offload support, so it is either -ngl 1 or -ngl 0, like Metal. We plan to implement this eventually, by implementing a split point in the compute graph. We do not plan to implement per-op offload (what most backends, do, including 0cc4am's Vulkan backend). (Partial offloading is now implemented.)
Supported model architectures are currently Falcon and Llama.
Supported model formats are Q4_0, Q4_1, FP16, and FP32, with or without Q6_K output tensors.
~~GPU-accelerated matmul for prompt processing is currently disabled due to a known issue with incorrect output. Token generation runs 100% on the GPU.~~ (GPU prompt processing has been fixed and re-enabled.)
Discrete NVIDIA and AMD GPUs have been tested successfully, but both integrated and discrete Intel GPUs are currently known not to work (garbage output).
This backend currently requires shaderFloat16, which e.g. not available in Pascal (GTX 10-series) and older GPUs from NVIDIA. You can check if your GPU is currently supported by going to vulkan.gpuinfo.org, finding your device, and searching for shaderFloat16 under "Features > Core 1.2" - it must be supported.
Windows and Linux are fully supported. macOS has Metal so we haven't attempted to build this backend there.
~~This PR still needs to be updated for the per-layer KV cache.~~ Done a while ago.

Contributions are welcome! This backend currently works well enough for Nomic's purposes so at the moment we are not focused on adding new features, only maintenance - our team is quite small right now. The goal of this PR is to reduce the maintenance burden for us, as llama.cpp frequently introduces changes that affect all backends.

…nse (SOM), version 1.0.

should no longer have new external deps other than libvulkan ``` ubuntu@ip-172-31-1-24:~/repo/gpt4all/gpt4all-backend/build$ ldd ./libllamamodel-mainline-avxonly.so linux-vdso.so.1 (0x00007ffcb53bb000) libvulkan.so.1 => /lib/x86_64-linux-gnu/libvulkan.so.1 (0x00007f239dab5000) libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f239d800000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f239d719000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f239da95000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f239d400000) /lib64/ld-linux-x86-64.so.2 (0x00007f239dd1d000) ```

…bvulkan.

… and stop using so many static objects so we can tear down and bring up vulkan on new devices in the same runtime.

There are some warnings in debug builds that are likely to be false positives.

…vely.

… gguff'd llama.

… our branch.

…ax kernel.

… for all kernels.

slaren · 2024-01-28T13:44:31Z

@ggerganov My concern is that a call to ggml_vk_init_device is required before the backend can be used. This creates two problems:

Adds unnecessary backend-specific code to llama.cpp
Forces other ggml applications that want to use this backend to add more backend-specific code

While I understand that this is not very high priority, the fix should be very simple (just move the initialization calls to the backend code), so I don't really see any reason to not fix it before merging.

cebtenzzre · 2024-01-28T15:43:31Z

It does not use GPU for the moment.

Right now only Falcon and Llama are whitelisted. This backend is strict about the model format because it is designed to run a contiguous graph on the GPU (like Metal), instead of only offloading the ops that are implemented (like Occam's Vulkan PR).

0cc4m · 2024-01-28T16:09:52Z

It does not use GPU for the moment.

Right now only Falcon and Llama are whitelisted. This backend is strict about the model format because it is designed to run a contiguous graph on the GPU (like Metal), instead of only offloading the ops that are implemented (like Occam's Vulkan PR).

At this point my backend also runs the full graph on GPU contiguously (if all layers are offloaded).

…ceb/nomic-vulkan

The previous attempt actually broke GPU inference with the 'main' example, which was previously working. deviceName is a vk::ArrayWrapper1D. Be careful when we convert it to a std::string, so we don't get null bytes at the end.

rajeevn1 · 2024-01-29T21:45:53Z

ggml_vk_graph_compute: error: unsupported op 'MUL_MAT'

I get this error using kompute backend, on Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02)

cebtenzzre · 2024-01-29T21:48:39Z

ggml_vk_graph_compute: error: unsupported op 'MUL_MAT'

What model are you using? Maybe the fallback code got broken among the recent changes (edit: yes, it did) - it sounds like you are using an unsupported quantization type.

Also, Intel GPUs are not currently known to work, and I don't know how well integrated GPUs work in general at the moment.

rajeevn1 · 2024-01-29T21:56:25Z

I am using mistral-7b-instruct-v0.2.Q5_K_M.gguf

Titaniumtown · 2024-01-29T22:09:49Z

ggml_vk_graph_compute: error: unsupported op 'MUL_MAT'

I get this error using kompute backend, on Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02)

I can reproduce on Alder Lake (Iris Xe)

rajeevn1 · 2024-01-29T22:42:25Z

I tried with different quantization mistral-7b-instruct-v0.2.Q4_K_M.gguf but get the same error

ggml_vk_graph_compute: error: unsupported op 'MUL_MAT'

cebtenzzre · 2024-01-30T00:10:18Z

I tried with different quantization mistral-7b-instruct-v0.2.Q4_K_M.gguf but get the same error

This backend currently only supports Q4_0, Q4_1, and F16 quantizations. The latest master of llama.cpp will at least fall back to CPU in this case instead of failing.

sorasoras · 2024-01-30T13:35:22Z

I tried with different quantization mistral-7b-instruct-v0.2.Q4_K_M.gguf but get the same error

This backend currently only supports Q4_0, Q4_1, and F16 quantizations. The latest master of llama.cpp will at least fall back to CPU in this case instead of failing.

It would be nice if it can fallback to the other vulkan backend.
Just use two vulkan backend together lol.

Signed-off-by: Jared Van Bortel <[email protected]> Co-authored-by: niansa <[email protected]> Co-authored-by: Adam Treat <[email protected]> Co-authored-by: Aaron Miller <[email protected]> Co-authored-by: ToKiNoBug <[email protected]> Co-authored-by: Georgi Gerganov <[email protected]> Co-authored-by: slaren <[email protected]>

niansa · 2024-02-12T21:36:20Z

I am really happy the Kompute implementation finally made it into mainline llama.cpp!

akingoverlook · 2024-02-25T00:12:02Z

This is Nomic's Kompute-based Vulkan backend from the GPT4All project, now available under the MIT license. It can be enabled by building with cmake and passing -DLLAMA_KOMPUTE=ON (make is currently not supported).

Structure

Nomic appears to claim support for Qualcomm GPUs by "Nomic Vulkan":
"September 18th, 2023: Nomic Vulkan launches supporting local LLM inference on AMD, Intel, Samsung, Qualcomm and NVIDIA GPUs."

Perhaps that is different from "Nomic's Kompute based Vulkan backend", that is not exactly clear. But what is clear is that this backend can't run on Qualcomm GPUs at all because it wants uniformAndStorageBuffer (8 and 16bit) access, which their Vulkan driver does not show as supported (on any of their GPUs).

FYI, it also can't run on ARM GPUs, because they have maximum subgroup size of 16 (even in the top of the line Immortalis chips), and this backend wants 32.

Any chance these dependencies could be worked around, @cebtenzzre ?

Signed-off-by: Jared Van Bortel <[email protected]> Co-authored-by: niansa <[email protected]> Co-authored-by: Adam Treat <[email protected]> Co-authored-by: Aaron Miller <[email protected]> Co-authored-by: ToKiNoBug <[email protected]> Co-authored-by: Georgi Gerganov <[email protected]> Co-authored-by: slaren <[email protected]>

niansa and others added 30 commits October 5, 2023 13:39

Nomic vulkan backend licensed under the Software for Open Models Lice…

ba15dfd

…nse (SOM), version 1.0.

Remove warning which fails on windows.

48a45ea

Switch to a dynamic dispatch table instead of linking hard against li…

45c8778

…bvulkan.

Completely revamp how we do object management with the vulkan backend…

b7e2e69

… and stop using so many static objects so we can tear down and bring up vulkan on new devices in the same runtime.

Make kompute actually include external SDK headers when requested

beee572

Throw an exception when allocation fails for vulkan.

68cf1df

vulkan: disambiguate gpus with the same name

8bea719

Don't try and install kompute artifacts.

bd5f639

Sync from device back to host at begin of new prompt.

4ed25b2

Only use vulkan with known quant that work.

68aca6b

Set the singleton to nullptr here.

addac25

Don't crash on available devices if we can't even create an instance.

2c24d67

Support for gguf.

1b1416d

kompute : don't fail build because of -Warray-bounds

6b6c73a

There are some warnings in debug builds that are likely to be false positives.

Upload immediately to device.

9e4f8b4

Add a common boilerplate code via include and elim copy pasta

77135a3

Consolidate code for mat x vec kernels and use subgroups more extensi…

93306f1

…vely.

Move the subgroups and printf into common.

601905e

Minor cleanup.

5509f74

Refactor getrows to use common code and get ready for q6_k.

4b223ec

Add q6_k getrows and mul*vec kernel.

f1c9bc1

Fix offset into the qh and now we have working vulkan accelerated for…

06d4b21

… gguff'd llama.

Fixes for norm.

32289aa

Fixup the upstream CMakelists.txt so we can build just llama.cpp with…

6ac3975

… our branch.

Change this back to be in agreement with metal and our previous softm…

de589ce

…ax kernel.

Fixes for subgroup size to bring AMD and NVIDIA inline with eachother…

bc4b5ed

… for all kernels.

kompute : only try to use Vulkan for LLaMA itself

24a4a59

kompute : remove Q6_K from list of supported quant types

3d850db

f16 mv broadcasting fix (gqa fix)

9db90cb

cebtenzzre added 4 commits January 29, 2024 12:07

kompute : better device management

be7c055

Merge branch 'master' of https://github.com/ggerganov/llama.cpp into …

da1dc66

…ceb/nomic-vulkan

kompute : fix merge issues

dc08e51

kompute : remove llama_load_model_from_file_internal

7e11fe0

cebtenzzre requested a review from slaren January 29, 2024 17:57

cebtenzzre added 4 commits January 29, 2024 14:07

minor fixup

48db724

fix trailing whitespace

1f98dff

fix incorrect memcpy

2998211

slaren approved these changes Jan 29, 2024

View reviewed changes

cebtenzzre merged commit fbf1dde into master Jan 29, 2024
53 checks passed

cebtenzzre deleted the ceb/nomic-vulkan branch January 29, 2024 20:50

cebtenzzre mentioned this pull request Jan 29, 2024

kompute : fix fallback to CPU #5201

Merged

maxwell-kalin mentioned this pull request Jan 30, 2024

Add Vulkan runner ollama/ollama#2033

Open

mudler mentioned this pull request Jan 31, 2024

feat(llama.cpp): Vulkan, Kompute, SYCL mudler/LocalAI#1647

Open

4 tasks

stduhpf mentioned this pull request Feb 24, 2024

Are Vulkan and Kompute backend supported on AMD 7800 XT in Windows ? #5698

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nomic Vulkan backend #4456

Nomic Vulkan backend #4456

cebtenzzre commented Dec 13, 2023 •

edited

Loading

slaren commented Jan 28, 2024 •

edited

Loading

cebtenzzre commented Jan 28, 2024

0cc4m commented Jan 28, 2024

rajeevn1 commented Jan 29, 2024

cebtenzzre commented Jan 29, 2024 •

edited

Loading

rajeevn1 commented Jan 29, 2024

Titaniumtown commented Jan 29, 2024

rajeevn1 commented Jan 29, 2024

cebtenzzre commented Jan 30, 2024

sorasoras commented Jan 30, 2024

niansa commented Feb 12, 2024

akingoverlook commented Feb 25, 2024 •

edited

Loading

Structure

Nomic Vulkan backend #4456

Nomic Vulkan backend #4456

Conversation

cebtenzzre commented Dec 13, 2023 • edited Loading

Structure

Limitations

slaren commented Jan 28, 2024 • edited Loading

cebtenzzre commented Jan 28, 2024

0cc4m commented Jan 28, 2024

rajeevn1 commented Jan 29, 2024

cebtenzzre commented Jan 29, 2024 • edited Loading

rajeevn1 commented Jan 29, 2024

Titaniumtown commented Jan 29, 2024

rajeevn1 commented Jan 29, 2024

cebtenzzre commented Jan 30, 2024

sorasoras commented Jan 30, 2024

niansa commented Feb 12, 2024

akingoverlook commented Feb 25, 2024 • edited Loading

Structure

cebtenzzre commented Dec 13, 2023 •

edited

Loading

slaren commented Jan 28, 2024 •

edited

Loading

cebtenzzre commented Jan 29, 2024 •

edited

Loading

akingoverlook commented Feb 25, 2024 •

edited

Loading