Update README.md

main
randaller 3 years ago committed by GitHub
parent 6d13fb894a
commit 6f71f26921
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -61,7 +61,7 @@ Running model with single prompt on Windows computer equipped with 12700k, fast
| 13B | 77 Gb, peak 100 Gb | 38 Gb | 380 seconds | can't handle to wait |
### RAM usage optimization
By default, torch uses Float32 precision while running on CPU, that leads, for example, to using 44 GB of RAM for 7B model. We may use Bfloat16 precision on CPU too, which decreases RAM consumption/2, down to 22 GB for 7B model, but inference processing much slower.
By default, torch uses Float32 precision while running on CPU, which leads, for example, to use 44 GB of RAM for 7B model. We may use Bfloat16 precision on CPU too, which decreases RAM consumption/2, down to 22 GB for 7B model, but inference processing much slower.
Uncomment this line in the example-cpu.py to enable Bfloat16 and save memory.
```

Loading…
Cancel
Save