Inference on CPU code for LLaMA models

You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Go to file

randaller f7896c98d5 Update README.md		3 years ago
llama	run on cpu	3 years ago
model	Create .gitkeep	3 years ago
tokenizer	Create .gitkeep	3 years ago
.gitignore	Initial commit	3 years ago
CODE_OF_CONDUCT.md	Initial commit	3 years ago
CONTRIBUTING.md	Initial commit	3 years ago
FAQ.md	Add FAQ.md // add command line options	3 years ago
LICENSE	Initial commit	3 years ago
MODEL_CARD.md	Fix typos in MODEL_CARD.md	3 years ago
README.md	Update README.md	3 years ago
example-cpu.py	Create example-cpu.py	3 years ago
requirements.txt	Update requirements.txt	3 years ago
setup.py	Initial commit	3 years ago

README.md

Inference LLaMA models using CPU only

This repository is intended as a minimal, hackable and readable example to load LLaMA (arXiv) models and run inference by using only CPU. Thus requires no videocard, but 64 (better 128 Gb) of RAM and modern processor is required.

Conda Environment Setup Example for Windows 10+

Download and install Anaconda Python https://www.anaconda.com and run Anaconda Prompt

conda create -n llama python=3.10
conda activate llama
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

Setup

In a conda env with pytorch / cuda available, run

pip install -r requirements.txt

Then in this repository

pip install -e .

Download tokenizer and models

magnet:?xt=urn:btih:ZXXDAUWYLRUXXBHUYEMS6Q5CE5WA3LVA&dn=LLaMA

CPU Inference

Place tokenizer.model and tokenizer_checklist.chk into [/tokenizer] folder

Place three files of 7B model into [/model] folder

Run it:

python example-cpu.py

Model Card

See MODEL_CARD.md

License

See the LICENSE file.