87 lines
2.7 KiB
Markdown
87 lines
2.7 KiB
Markdown
# AIRC (pronounced Eric)
|
|
|
|
AI is Really Cool
|
|
|
|
NOTE: If running on an Intel Arc A series graphics processor, fp64 is not supported and may need to either be emulated or have the model quantized.
|
|
|
|
This project provides container definitions that will provide PyTorch 2.6 with
|
|
Intel's LLM project. In addition, it provides a small local chat server and an IRC client to provide a chat bot.
|
|
|
|
# Installation
|
|
|
|
This project uses docker containers to build. As this was originally
|
|
written to work on an Intel Arc B580 (Battlemage), it requires a
|
|
kernel that supports that hardware, such as the one documented
|
|
at [Intel Graphics Preview](https://github.com/canonical/intel-graphics-preview), which runs in Ubuntu Oracular (24.10)..
|
|
|
|
NOTE: You need 'docker compose' installed. See [Install Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/)
|
|
|
|
## Want to run under WSL2? No can do...
|
|
|
|
https://www.intel.com/content/www/us/en/support/articles/000093216/graphics/processor-graphics.html
|
|
|
|
The A- and B-series discrete GPUs do not support SR-IOV, required for
|
|
the GPU partitioning that Microsoft Windows uses in order to support GPU acceleration in WSL.
|
|
|
|
## Building
|
|
|
|
NOTE: You need 'docker compose' installed. See [Install Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/)
|
|
|
|
|
|
```bash
|
|
git clone https://github.com/jketreno/airc
|
|
cd airc
|
|
docker compose build
|
|
```
|
|
|
|
## Running
|
|
|
|
In order to download the models, you need to have a Hugging Face
|
|
token. See https://huggingface.co/settings/tokens for information
|
|
on obtaining a token.
|
|
|
|
Edit .env to add the following:
|
|
|
|
```.env
|
|
HF_ACCESS_TOKEN=<access token from huggingface>
|
|
```
|
|
|
|
NOTE: Models downloaded by most examples will be placed in the
|
|
./cache directory, which is bind mounted to the container.
|
|
|
|
### AIRC
|
|
|
|
To launch the airc shell interactively, with the pytorch 2.6
|
|
environment loaded, use the default entrypoint to launch a shell:
|
|
|
|
```bash
|
|
docker compose run --rm airc shell
|
|
```
|
|
|
|
Once in the shell, you can then launch the model-server.py and then
|
|
the airc.py client:
|
|
|
|
```bash
|
|
docker compose run --rm airc shell
|
|
src/airc.py --ai-server=http://localhost:5000 &
|
|
src/model-server.py
|
|
```
|
|
|
|
By default, src/airc.py will connect to irc.libera.chat on the airc-test
|
|
channel. See `python src/airc.py --help` for options.
|
|
|
|
By separating the model-server into its own process, you can develop
|
|
and tweak the chat backend without losing the IRC connection established
|
|
by airc.
|
|
|
|
### Jupyter
|
|
|
|
```bash
|
|
docker compose up jupyter -d
|
|
```
|
|
|
|
The default port for inbound connections is 8888 (see docker-compose.yml).
|
|
$(pwd)/jupyter is bind mounted to /opt/juypter in the container, which is where notebooks will be saved by default.
|
|
|
|
To access the jupyter notebook, go to `https://localhost:8888/jupyter`.
|