diff --git a/README.md b/README.md index 971e504..f8d8803 100644 --- a/README.md +++ b/README.md @@ -93,6 +93,25 @@ https://github.com/josephrocca/rwkv-v4-web For the old RWKV-2: see the release here for a 27M params model on enwik8 with 0.72 BPC(dev). Run run.py in https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v2-RNN. You can even run it in your browser: https://github.com/BlinkDL/AI-Writer/tree/main/docs/eng https://blinkdl.github.io/AI-Writer/eng/ (this is using tf.js WASM single-thread mode). +I have an idea to quantize a matrix with outliers: +```python +import numpy as np + +# the original M, with outliers +M = np.array([[1, 2, 1, 2],[2, 100, 2, 10],[1, 2, 1, 2],[2, 1, 20, 1]]) + +# the scaled M, without outliers +Q = np.array([[1, 0.2, 0.1, 2],[0.4, 2, 0.04, 2], [1, 0.2, 0.1, 2],[2, 0.1, 2, 1]]) +# we can find optimal a & b to minimize inference error after quantization +a = np.array([1, 10, 10, 1]) +b = np.array([1, 5, 1, 1]) + +# test M.v with random v - the results will be the same +v = np.array([1.23, 5.44, 9.75, 2.98]) +print(M.dot(v)) +print(Q.dot(v * a) * b) +``` + ### Training / Fine-tuning **Training RWKV-4 from scratch:** run train.py, which by default is using the enwik8 dataset (unzip https://data.deepai.org/enwik8.zip).