6 Commits (e0ffc861fae5ac8b40ce973f822d03db02929d36)

Author	SHA1	Message	Date
Georgi Gerganov	eb34620aec	Add tokenizer test + revert to C++11 (#355) ... * Add test-tokenizer-0 to do a few tokenizations - feel free to expand * Added option to convert-pth-to-ggml.py script to dump just the vocabulary * Added ./models/ggml-vocab.bin containing just LLaMA vocab data (used for tests) * Added utility to load vocabulary file from previous point (temporary implementation) * Avoid using std::string_view and drop back to C++11 (hope I didn't break something) * Rename gpt_vocab -> llama_vocab * All CMake binaries go into ./bin/ now	3 years ago
Casey Primozic	2e664f1ff4	Add initial AVX512 support for dot product on Linux (#320) ... * Update Makefile to detect AVX512 support and add compiler flags if it's available * Based on existing AVX2 implementation, dot product on one 32-value block of 4-bit quantized ints at a time * Perform 8 bit -> 16 bit sign extension and multiply+add on 32 values at time instead of 16 * Use built-in AVX512 horizontal reduce add to get sum at the end * Manual unrolling on inner dot product loop to reduce loop counter overhead	3 years ago
Mack Straight	074bea2eb1	sentencepiece bpe compatible tokenizer (#252) ... * potential out of bounds read * fix quantize * style * Update convert-pth-to-ggml.py * mild cleanup * don't need the space-prefixing here rn since main.cpp already does it * new file magic + version header field * readme notice * missing newlines Co-authored-by: slaren <2141330+slaren@users.noreply.github.com>	3 years ago
Thomas Klausner	41be0a3b3d	Add NetBSD support. (#90)	3 years ago
Georgi Gerganov	7211862c94	Update Makefile var + add comment	3 years ago
Georgi Gerganov	26c0846629	Initial release	3 years ago