Dicussion: Recommended best model base on User's hardware

[nguyenhoangthuan99]

Problem
We are currently curating AI models that are optimized specifically for local assistant use cases. The aim is to provide users with models that perform extremely well, even under hardware constraints.

Our app will detect the user's hardware (e.g. VRAM, RAM, GPU/CPU) and recommend the best model for them to use based on their resources. We are considering thresholds like 80% VRAM usage or 50% RAM usage to ensure that the models can run efficiently and smoothly without overwhelming the system.

Basic Idea

• User usage:
  • Tasks: developement, content writing, general, ...
  • Speed: need as fast as posible, can wait model to response, ...
• Hardware:
  • CPU
  • GPU
  • Hybrid for very large models

Solution

• Detect free RAM/ VRAM in system and notify user the best model for each case (All in GPU, All in CPU, hybrid).
• Warning user are not using maximum free resource for running models.
• Calculate/estimate memory requirement for each model (include KV cache reserve for context length) and recommend user the setting (ngl, ...)
• Base on user task (code, content, ...) recommend to modify temperature, top_p.