News Summary

• Strawberry contains the letter ‘r’ three times, but when asked, many large language models (LLMs) answer that the letter appears twice.
• This happens because most LLMs encode words as ‘tokens’ that represent sequences of letters.
• report an approach called byteification that retrofits token-based LLMs to operate at the byte level.
• The authors show that byteified models can achieve competitive performance while retaining the ability to read individual characters.