novarobot
/
RWKV-LM
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'main' of https://github.com/BlinkDL/RWKV-LM into main

Browse Source
main
BlinkDL 4 years ago
parent 8aec414db2 9e959d0b8a
commit 9b903db103
1 changed files with 4 additions and 4 deletions
Show Stats Download Patch File Download Diff File

8
README.md
Unescape Escape View File

@ -68,13 +68,13 @@ Character-level loss on simplebooks-92 dataset https://dldata-public.s3.us-east-
![RWKV-vs-MHA](RWKV-vs-MHA.png) ![RWKV-vs-MHA](RWKV-vs-MHA.png)
Gray: usual MHA+Rotary+GeGLU - performance not as good. Gray: usual MHA+Rotary+GeGLU - performance not as good. 17.2M params.
Red: RWKV ("linear" attention) - VRAM friendly - quite faster when ctx window is long - good performance. Red: RWKV ("linear" attention) - VRAM friendly - quite faster when ctx window is long - good performance. 16.6M params.
Black: MHA_pro (MHA with various tweaks & RWKV-type-FFN) - slow - needs more VRAM - good performance. Green: MHA+Rotary+GeGLU+Token_shift. 17.2M params.
parameters count: 17.2 vs 18.5 vs 18.5. Blue: MHA_pro (MHA with various tweaks & RWKV-type-FFN) - slow - needs more VRAM - good performance. 16.6M params.
``` ```
@software{peng_bo_2021_5196578, @software{peng_bo_2021_5196578,

Write Preview
Loading…
Cancel
Save