novarobot
/
RWKV-LM
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md

Browse Source
main
PENG Bo 4 years ago committed by GitHub
parent 4b1df60e94
commit 9ca62d3a1e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File

4
README.md
Unescape Escape View File

@ -1,6 +1,6 @@
# The RWKV Language Model # The RWKV Language Model
## v2 ## RWKV v2 RNN
RWKV v2 is a RNN which can also be directly trained like a GPT transformer. RWKV v2 is a RNN which can also be directly trained like a GPT transformer.
@ -8,6 +8,8 @@ You only need x_t, a_t, b_t of position t to compute the vectors for position t+
Hence it can be 100x faster than GPT, and 100x more VRAM friendly. Hence it can be 100x faster than GPT, and 100x more VRAM friendly.
## How it works
The a b c d factors work together to build a time-decay curve: u, 1, w, w^2, w^3, ... The a b c d factors work together to build a time-decay curve: u, 1, w, w^2, w^3, ...
Write out the formulas for "token at pos 2" and "token at pos 3" and you will get the idea: Write out the formulas for "token at pos 2" and "token at pos 3" and you will get the idea:

Write Preview
Loading…
Cancel
Save