novarobot
/
RWKV-LM
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md

Browse Source
main
PENG Bo 4 years ago committed by GitHub
parent 5676b039ad
commit 8d63423a56
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File

2
README.md
Unescape Escape View File

@ -49,6 +49,8 @@ My LR schedule for the L24-D1024 RWKV-2:
Fixing NaN or loss spikes: load a previous checkpoint, decrease LR a bit. I find you can decrease the LR faster than GPT, and eventually to 1/50 of LR_max. Fixing NaN or loss spikes: load a previous checkpoint, decrease LR a bit. I find you can decrease the LR faster than GPT, and eventually to 1/50 of LR_max.
**UPDATE: Search for "RWKV v2+" here and change RWKV-2 to PreLN to make it more stable.**
Fine-tuning: see https://github.com/BlinkDL/RWKV-v2-RNN-Pile. Fine-tuning: see https://github.com/BlinkDL/RWKV-v2-RNN-Pile.
## How it works ## How it works

Write Preview
Loading…
Cancel
Save