Inference LLaMA models using CPU only

This repository is intended as a minimal, hackable and readable example to load LLaMA (arXiv) models and run inference by using only CPU. Thus requires no videocard, but 64 (better 128 Gb) of RAM and modern processor is required.

Conda Environment Setup Example for Windows 10+

Download and install Anaconda Python https://www.anaconda.com and run Anaconda Prompt

conda create -n llama python=3.10
conda activate llama
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

Setup

In a conda env with pytorch / cuda available, run

pip install -r requirements.txt

Then in this repository

pip install -e .

Download tokenizer and models

magnet:?xt=urn:btih:ZXXDAUWYLRUXXBHUYEMS6Q5CE5WA3LVA&dn=LLaMA

CPU Inference of 7B model

Place tokenizer.model and tokenizer_checklist.chk into repo's [/tokenizer] folder.

Place consolidated.00.pth and params.json from 7B torrent folder into repo's [/model] folder.

Run the example:

python example-cpu.py

CPU Inference of 13B, 30B and 65B models

A little bit tricky part is that we need to unshard the checkpoints first. In this example, D:\Downloads\LLaMA is a root folder of downloaded torrent with models. Run the following command to create merged weights checkpoint:

python merge-weights.py --input_dir D:\Downloads\LLaMA --model_size 13B

This will create merged.pth file in the repo's root folder. Move this file into [/model] folder.

Place corresponding params.json file (from 13B torrent folder) into repo's [/model] folder.

So, you should end up with two files in [/model] folder: merged.pth and params.json.

Place tokenizer.model and tokenizer_checklist.chk into repo's [/tokenizer] folder.

Run the example:

python example-cpu.py

Model Card

See MODEL_CARD.md

License

See the LICENSE file.