1. Time decay changed to something like (0.9999^T + 0.1) where 0.9999 and 0.1 are learnable. The 0.1 part will be kept forever.
1. Now time decay is like 0.999^T (0.999 is learnable). Change it to something like (0.999^T + 0.1) where 0.1 is learnable too. The 0.1 part will be kept forever.
2. Use complex number (so, rotation instead of decay) in some channels.