8 Commits (721311070e31464ac12bef9a4444093eb3eaebf7)

Author	SHA1	Message	Date
Ronsor	956dfda8ad	Use `tokenizer.vocab_size()` instead of hardcoding 32000 in convert-pth-to-ggml.py (#142) ... There are ways that special tokens or other new tokens could be added to the tokenizer; therefore it's probably best not to assume the vocabulary is only 32000 tokens.	3 years ago
Val Kharitonov	2a20f48efa	Fix UTF-8 handling (including colors) (#79)	3 years ago
Georgi Gerganov	7c9e54e55e	Revert "weights_only" arg - this causing more trouble than help	3 years ago
Oleksandr Nikitin	b9bd1d0141	python/pytorch compat notes (#44)	3 years ago
deepdiffuser	a93120236f	use weights_only in conversion script (#32) ... this restricts malicious weights from executing arbitrary code by restricting the unpickler to only loading tensors, primitive types, and dictionaries	3 years ago
Georgi Gerganov	007a8f6f45	Support all LLaMA models + change Q4_0 quantization storage	3 years ago
Georgi Gerganov	70bc0b8b15	Fix a bug in the rope calculation	3 years ago
Georgi Gerganov	26c0846629	Initial release	3 years ago