novarobot
/
RWKV-LM
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

no message

Browse Source
main
BlinkDL 3 years ago
parent 13bb641007
commit 7cdc8d3164
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File

6
RWKV-v4/train.py
Unescape Escape View File

@ -97,10 +97,10 @@ MODEL_NAME = epoch_save_path + str(EPOCH_BEGIN)
######################################################################################################## ########################################################################################################
if LOAD_MODEL and EPOCH_BEGIN > 0: # we are not saving gradients. so let's have some warmup if we load a model if LOAD_MODEL and EPOCH_BEGIN > 0: # we are not saving gradients, so let's have some warmup if we load a model
warmup_tokens = ctx_len * batch_size * 50 warmup_tokens = 50 * ctx_len * batch_size // NUM_GPUS
else: else:
warmup_tokens = ctx_len * batch_size * 0 warmup_tokens = 0
betas = (0.9, 0.99) betas = (0.9, 0.99)
eps = 1e-8 eps = 1e-8

Write Preview
Loading…
Cancel
Save