diff --git a/README.md b/README.md index 03a936f..42fa7e1 100644 --- a/README.md +++ b/README.md @@ -26,8 +26,6 @@ alt="\text{softmax}_t(\text{K}_{u,c}) = \frac{\exp(\text{K}_{u,c})}{\sum_{v \leq "https://render.githubusercontent.com/render/math?math=%5Cdisplaystyle+W_%7Bt%2Cu%2Cc%7D%3Df_h%28t-u%29%5Ccdot+%5Calpha_h%28u%29+%5Ccdot+%5Cbeta_h%28t%29" alt="W_{t,u,c}=f_h(t-u)\cdot \alpha_h(u) \cdot \beta_h(t)"> -(3) You don't need LayerNorm for Time-mix. In fact, the model converges faster when LayerNorm is removed. - Moreover we multiply the final output of Time-mix layer by γ(t). The reason for the α β γ factors, is because the context size is smaller when t is small, and this can be compensated using the α β γ factors. * The Channel-mix is similar to GeGLU (https://arxiv.org/abs/2002.05202) with an extra R factor.