6 Commits (3ab3e6582f7320c2b6568c892fdfc8215caf7e6c)

Author	SHA1	Message	Date
Georgi Gerganov	eb34620aec	Add tokenizer test + revert to C++11 (#355) ... * Add test-tokenizer-0 to do a few tokenizations - feel free to expand * Added option to convert-pth-to-ggml.py script to dump just the vocabulary * Added ./models/ggml-vocab.bin containing just LLaMA vocab data (used for tests) * Added utility to load vocabulary file from previous point (temporary implementation) * Avoid using std::string_view and drop back to C++11 (hope I didn't break something) * Rename gpt_vocab -> llama_vocab * All CMake binaries go into ./bin/ now	3 years ago
Mack Straight	a791a68b61	move file magic/version to header, print expected version (#319)	3 years ago
Mack Straight	074bea2eb1	sentencepiece bpe compatible tokenizer (#252) ... * potential out of bounds read * fix quantize * style * Update convert-pth-to-ggml.py * mild cleanup * don't need the space-prefixing here rn since main.cpp already does it * new file magic + version header field * readme notice * missing newlines Co-authored-by: slaren <2141330+slaren@users.noreply.github.com>	3 years ago
Sebastián A	eb062bb012	Windows fixes (#31) ... * Apply fixes suggested to build on windows Issue: https://github.com/ggerganov/llama.cpp/issues/22 * Remove unsupported VLAs * MSVC: Remove features that are only available on MSVC C++20. * Fix zero initialization of the other fields. * Change the use of vector for stack allocations.	3 years ago
Georgi Gerganov	a9e58529ea	Fix un-initialized FP16 tables on x86 (#15, #2)	3 years ago
Georgi Gerganov	26c0846629	Initial release	3 years ago