Bits-per-Byte measures how many bits a compression program needs to know to guess the next symbol on average. For example, if the compression program is perfect, then the next symbol is obvious to it, and it needs 0 bits, so \( bpb = 0 \). Or, if it the worst possible, it needs to be given the exact next symbol from the vocabulary, so it needs as \( bpb = log_2(\mathrm{vocabularySize}) \).
How BPB relates to compression ratio or cross-entropy?
Bits-per-Byte (BPB) and Bits-per-Character (BPC) are metrics related to compression ratio and cross-entropy, used in compression and language modeling, with BPC equaling BPB for ASCII Extended characters, and cross-entropy loss using log2 in character-level models equating to BPC.
- compression ratio is defined as \( \mathrm{cmpRatio} = \mathrm{unCompressedBytes} / \mathrm{compressedBytes} \)
- Bits-per-byte is defined as \( \mathrm{compressedBits} / \mathrm{unCompressedBytes} \)
- Bits-per-byte (bpb) metric is inverse compression ratio divided by 8: \( 1 bpb = 1 / (8 \mathrm{cmpRatio}) \).
- Bits-per-character (bpc) metric for ASCII Extended characters equals bits-per-byte (bpb).
- Cross-entropy loss (log2) for a character-level language model averaged over a dataset equals bpc.
- Perplexity is cross-entropy (log2) to the second power \( PP = 2^{\mathrm{crossEnropy}} \)
- Gzip compresses enwik8 2.92 bpb, Morse code approximately 10.8 bpc
- SRU++ model achieves 1.02 bpc - approximately compression ratio of 8
BPC corresponds to BPB for extended ASCII characters, and when using log2 in character-level models, the cross-entropy loss is equivalent to BPB.
Neural Data Compression
Data compression relies on ability to predict next symbol. Read more on neural data compression and its applications in machine learning here.
