Inference on CPU code for LLaMA models

You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Go to file

randaller 0e2612680f Update README.md		3 years ago
llama	run on cpu	3 years ago
model	Create .gitkeep	3 years ago
tokenizer	Create .gitkeep	3 years ago
.gitignore	Initial commit	3 years ago
CODE_OF_CONDUCT.md	Initial commit	3 years ago
CONTRIBUTING.md	Initial commit	3 years ago
FAQ.md	Add FAQ.md // add command line options	3 years ago
LICENSE	Initial commit	3 years ago
MODEL_CARD.md	Fix typos in MODEL_CARD.md	3 years ago
README.md	Update README.md	3 years ago
example-cpu.py	Update example-cpu.py	3 years ago
requirements.txt	Update requirements.txt	3 years ago
setup.py	Initial commit	3 years ago

README.md

Inference LLaMA models using CPU only

This repository is intended as a minimal, hackable and readable example to load LLaMA (arXiv) models and run inference by using only CPU. Thus requires no videocard, but 64 (better 128 Gb) of RAM and modern processor is required.

Conda Environment Setup Example for Windows 10+

Download and install Anaconda Python https://www.anaconda.com and run Anaconda Prompt

conda create -n llama python=3.10
conda activate llama
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

Setup

In a conda env with pytorch / cuda available, run

pip install -r requirements.txt

Then in this repository

pip install -e .

Download tokenizer and models

magnet:?xt=urn:btih:ZXXDAUWYLRUXXBHUYEMS6Q5CE5WA3LVA&dn=LLaMA

CPU Inference of 7B model

Place tokenizer.model and tokenizer_checklist.chk into repo's [/tokenizer] folder.

Place consolidated.00.pth and params.json from 7B torrent folder into repo's [/model] folder.

Run it:

python example-cpu.py

README.md

Inference LLaMA models using CPU only

Conda Environment Setup Example for Windows 10+

Setup

Download tokenizer and models

CPU Inference of 7B model

CPU Inference of 13B 30B 65B models

Model Card

License