llama.cpp

You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

History

Georgi Gerganov eb34620aec Add tokenizer test + revert to C++11 (#355 ) * Add test-tokenizer-0 to do a few tokenizations - feel free to expand * Added option to convert-pth-to-ggml.py script to dump just the vocabulary * Added ./models/ggml-vocab.bin containing just LLaMA vocab data (used for tests) * Added utility to load vocabulary file from previous point (temporary implementation) * Avoid using std::string_view and drop back to C++11 (hope I didn't break something) * Rename gpt_vocab -> llama_vocab * All CMake binaries go into ./bin/ now	3 years ago
..
build.yml	Add tokenizer test + revert to C++11 (#355 )	3 years ago
docker.yml	Docker - Fix publish docker image in GitHub Registry (#235 )	3 years ago

Georgi Gerganov eb34620aec

Add tokenizer test + revert to C++11 (#355 )

* Add test-tokenizer-0 to do a few tokenizations - feel free to expand
* Added option to convert-pth-to-ggml.py script to dump just the vocabulary
* Added ./models/ggml-vocab.bin containing just LLaMA vocab data (used for tests)
* Added utility to load vocabulary file from previous point (temporary implementation)
* Avoid using std::string_view and drop back to C++11 (hope I didn't break something)
* Rename gpt_vocab -> llama_vocab
* All CMake binaries go into ./bin/ now

build.yml

Add tokenizer test + revert to C++11 (#355 )

docker.yml

Docker - Fix publish docker image in GitHub Registry (#235 )