Update README.md

3 years ago · f3cf7a5ad1
parent 5c29eb779c
commit f3cf7a5ad1
1 changed files with 2 additions and 0 deletions
--- a/README.md
+++ b/README.md
@ -10,6 +10,8 @@ RWKV-3 1.5B on A40 (tf32) = always 0.015 sec/token, tested using simple pytorch

 GPT2-XL 1.3B on A40 (tf32) = 0.032 sec/token (for ctxlen 1000), tested using HF, GPU utilization 45% too (interesting), VRAM 9655M

+Training speed: RWKV-4 1.5B BF16 ctxlen1024 = 106K tokens/s on 8xA100 40G.
+
 ## Join our Discord: https://discord.gg/bDSBUMeFpc :)

 You are welcome to join the RWKV discord https://discord.gg/bDSBUMeFpc to build upon it. We have plenty of potential compute (A100 40Gs) now (thanks to CoreWeave), so if you have interesting ideas I can run them.