novarobot
/
RWKV-LM
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md

Browse Source
main
PENG Bo 4 years ago committed by GitHub
parent 4ffd8f1b76
commit 9e959d0b8a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 4 deletions
Show Stats Download Patch File Download Diff File

8
README.md
Unescape Escape View File

@ -68,13 +68,13 @@ Character-level loss on simplebooks-92 dataset https://dldata-public.s3.us-east-
![RWKV-vs-MHA](RWKV-vs-MHA.png) ![RWKV-vs-MHA](RWKV-vs-MHA.png)
Gray: usual MHA+Rotary+GeGLU - performance not as good. Gray: usual MHA+Rotary+GeGLU - performance not as good. 17.2M params.
Red: RWKV ("linear" attention) - VRAM friendly - quite faster when ctx window is long - good performance. Red: RWKV ("linear" attention) - VRAM friendly - quite faster when ctx window is long - good performance. 16.6M params.
Black: MHA_pro (MHA with various tweaks & RWKV-type-FFN) - slow - needs more VRAM - good performance. Green: MHA+Rotary+GeGLU+Token_shift. 17.2M params.
parameters count: 17.2 vs 18.5 vs 18.5. Blue: MHA_pro (MHA with various tweaks & RWKV-type-FFN) - slow - needs more VRAM - good performance. 16.6M params.
``` ```
@software{peng_bo_2021_5196578, @software{peng_bo_2021_5196578,

Write Preview
Loading…
Cancel
Save