Update README.md

main
PENG Bo 4 years ago committed by GitHub
parent 959115a7e6
commit 64fdb61056
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -10,7 +10,7 @@ alt="\begin{align*}
\end{align*} \end{align*}
"> ">
* Here R, K, V is generated by linear transforms of input. Basically RWKV decomposes attention into R(target) * W(src -> target) * K(src). So I call R "receptance", and sigmoid means it's in 0~1 range. * Here R, K, V are generated by linear transforms of input, and W is parameter. Basically RWKV decomposes attention into R(target) * W(src, target) * K(src). So we can call R "receptance", and sigmoid means it's in 0~1 range.
* The Time-mix is similar to AFT (https://arxiv.org/abs/2105.14103). There are two differences. * The Time-mix is similar to AFT (https://arxiv.org/abs/2105.14103). There are two differences.

Loading…
Cancel
Save