From 16655d369d8b29b05503e29f4c7015d947cf47c3 Mon Sep 17 00:00:00 2001
From: PENG Bo <33809201+BlinkDL@users.noreply.github.com>
Date: Sat, 21 May 2022 02:38:27 +0800
Subject: [PATCH] Update README.md

---
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 940ee59..0079e50 100644
--- a/README.md
+++ b/README.md
@@ -45,7 +45,7 @@ Fine-tuning: for a small model, try 4e-5 lr, and decay to 1e-5 when it plateaus.
 
 **Important**: For fine-tuning the Pile model, change K_EPS from 1e-16 to 1e-9 (to avoid NaN) in https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v2-RNN/src/model.py and https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v2-RNN/src/model_run.py and disable HeadQK (so it's a pure RNN). You can compare the output with the latest code ( https://github.com/BlinkDL/RWKV-v2-RNN-Pile ) to verify it.
 
-**Fixing NaN or loss spikes**: load a previous checkpoint, decrease LR a bit.
+**Fixing NaN or loss spikes**: load a previous checkpoint, decrease LR a bit. I find you can decrease the LR faster than GPT, and eventually to 1/50 of max LR (instead of 1/10).
 
 ## How it works