Update README.md

3 years ago · 819f2730b2
parent 66c1dabb94
commit 819f2730b2
1 changed files with 3 additions and 0 deletions
--- a/README.md
+++ b/README.md
@ -24,6 +24,9 @@ Training speed: RWKV-4 1.5B BF16 ctxlen1024 = 106K tokens/s on 8xA100 40G.

 I am doing image experiments too (For example: https://huggingface.co/BlinkDL/clip-guided-binary-autoencoder) and RWKV will be able to do txt2img diffusion :) My idea: 256x256 rgb image -> 32x32x13bit latents -> apply RWKV to compute transition probability for each of the 32x32 grid -> pretend the grids are independent and "diffuse" using these probabilities.

+Smooth training - no loss spikes! (lr & bsz change around 15G tokens)
+![RWKV-loss](RWKV-loss.png)
+
 ## Join our Discord: https://discord.gg/bDSBUMeFpc :)

 You are welcome to join the RWKV discord https://discord.gg/bDSBUMeFpc to build upon it. We have plenty of potential compute (A100 40Gs) now (thanks to Stability and EleutherAI), so if you have interesting ideas I can run them.