> For the complete documentation index, see [llms.txt](https://docs.aichatone.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.aichatone.com/product-guides/custom-chatbots/local-llm-models/xinference.md).

# Xinference

Xorbits Inference(Xinference) is an open-source project to run language models on your own machine. You can use it to serve open-source LLMs like Llama-2 locally.

## Preparation[​](https://doc.chathub.gg/custom-chatbots/openai#preparation) <a href="#preparation" id="preparation"></a>

Follow the instructions at [Using Xinference](https://inference.readthedocs.io/en/latest/getting_started/using_xinference.html) to setup Xinference and run the `llama-2-chat` model.

## Configuration <a href="#configuration" id="configuration"></a>

<figure><img src="/files/jZ3P8ed9j8OpbGiyiCWR" alt=""><figcaption></figcaption></figure>

* **API Endpoit**: `http://127.0.0.1:9997/v1/chat/completions`
* **API Key**: random strings
* **Model**: `llama-2-chat`

You can find all the available models at <https://inference.readthedocs.io/en/latest/models/builtin/llm/index.html>

## Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

* Only models with `chat` in their name are supported.