Using Small Models

AnyLLM CLI supports two types of models, which can be configured in anyllm.json. This allows for greater flexibility in choosing the right tool for your task.

Model Types

  1. Large Models (Default): These are powerful, general-purpose models that support "Function Calling" or "Tools". They can understand complex instructions and interact with the file system through a structured API. This is the default mode.

  2. Small Models (type: "small"): These are lighter models that may not support function calling. Instead, they operate by generating specific text-based commands to edit code. This approach is inspired by the tool Aider and allows anyllm-cli to work with a wider range of open-source models.

Configuration

To use a small model, add the "type": "small" parameter to its configuration in your anyllm.json file:

{
  "provider": {
    "ollama_local": {
      "name": "Ollama (Local)",
      "type": "openai_compatible",
      "options": {
        "baseURL": "http://localhost:11434/v1/",
        "headers": {
          "Authorization": "Bearer ollama"
        }
      },
      "models": {
        "phi3": {
          "name": "phi3:mini",
          "type": "small"
        },
        "deepseek": {
            "name": "deepseek-coder"
        }
      }
    }
  }
}

In this example, phi3 is configured as a small model, while deepseek will be used as a large model by default.

Important Considerations for Small Models

[!WARNING] Use with Caution Small models offer flexibility but come with trade-offs.

  • They do not guarantee high-quality code or a perfect understanding of instructions.
  • They may require several attempts or prompt refinements to achieve the desired result. Be patient and precise in your requests.
  • Unlike large LLMs, small models are often highly specialized. A model trained for coding may not be good at general conversation, and vice-versa.

It is crucial to choose a specialized model that fits your task.

Recommended Models

Here are some examples of specialized models that perform well in specific domains:

Edit this page on GitHub