novarobot
/
RWKV-LM
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md

Browse Source
main
PENG Bo 3 years ago committed by GitHub
parent 0c77cfbbee
commit fc047a20b1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File

4
README.md
Unescape Escape View File

@ -194,9 +194,9 @@ out.write(ss + "\n")
1. Now time decay is like 0.999^T (0.999 is learnable). Change it to something like (0.999^T + 0.1) where 0.1 is learnable too. The 0.1 part will be kept forever. 1. Now time decay is like 0.999^T (0.999 is learnable). Change it to something like (0.999^T + 0.1) where 0.1 is learnable too. The 0.1 part will be kept forever.
2. Use complex number (so, rotation instead of decay) in some channels. 2. Use complex-valued decay (so, rotation instead of decay) in some channels.
3. Inject some trainable and interpolable positional encoding? 3. Inject some trainable and extrapolatable positional encoding?
4. Aside from 2d rotation, we can try other Lie groups such as 3d rotation ( SO(3) ). Non-abelian RWKV lol. 4. Aside from 2d rotation, we can try other Lie groups such as 3d rotation ( SO(3) ). Non-abelian RWKV lol.

Write Preview
Loading…
Cancel
Save