Bits-Per-Byte and Bits-Per-Character

BPB and BPC are metrics used in compression and language modelling related to compression ratio.

Bits-per-Byte measures how many bits a compression program needs to know to guess the next symbol on average. For example, if the compression program is perfect, then the next symbol is obvious to it, and it needs 0 bits, so \( bpb = 0 \). Or, if it the worst possible, it needs to be given the exact next symbol from the vocabulary, so it needs as \( bpb = log_2(\mathrm{vocabularySize}) \).

How BPB relates to compression ratio or cross-entropy?

Bits-per-Byte (BPB) and Bits-per-Character (BPC) are metrics related to compression ratio and cross-entropy, used in compression and language modeling, with BPC equaling BPB for ASCII Extended characters, and cross-entropy loss using log2 in character-level models equating to BPC.

  • compression ratio is defined as \( \mathrm{cmpRatio} = \mathrm{unCompressedBytes} / \mathrm{compressedBytes} \)
  • Bits-per-byte is defined as \( \mathrm{compressedBits} / \mathrm{unCompressedBytes} \)
  • Bits-per-byte (bpb) metric is inverse compression ratio divided by 8: \( 1 bpb = 1 / (8 \mathrm{cmpRatio}) \).
  • Bits-per-character (bpc) metric for ASCII Extended characters equals bits-per-byte (bpb).
  • Cross-entropy loss (log2) for a character-level language model averaged over a dataset equals bpc.
  • Perplexity is cross-entropy (log2) to the second power \( PP = 2^{\mathrm{crossEnropy}} \)
  • Gzip compresses enwik8 2.92 bpb, Morse code approximately 10.8 bpc
  • SRU++ model achieves 1.02 bpc - approximately compression ratio of 8

BPC corresponds to BPB for extended ASCII characters, and when using log2 in character-level models, the cross-entropy loss is equivalent to BPB.

Deflate algorithm illustration with LZ77 and Huffman coding
Deflate algorithm illustration with LZ77 and Huffman coding

Neural Data Compression

Data compression relies on ability to predict next symbol. Read more on neural data compression and its applications in machine learning here.

Created on 20 May 2022. Updated on: 25 May 2023.
Thank you

About Vaclav Kosar How many days left in this quarter? Twitter Bullet Points to Copy & Paste Averaging Stopwatch Privacy Policy
Copyright © Vaclav Kosar. All rights reserved. Not investment, financial, medical, or any other advice. No guarantee of information accuracy.