Confronting the Enigma of AI Memorization - How Much Can Advanced Language Models Remember?

The cognitive prowess of Large Language Models (LLMs) drives the leading-edge research in artificial intelligence. Key players in this sphere- Meta, Google, Nvidia, and Cornell University, offer invaluable insights into the computation aspects of LLMs, including their memory power. Nurtured on colossal datasets, LLMs, like those behind ChatGPT, Anthropic’s Claude, and Google’s Gemini, have developed a unique understanding of linguistic patterns and broader world concepts. This understanding, encoded in the form of billions of parameters, inherently influences their output generation based on the training input received.

In a groundbreaking study released recently by researchers at Meta, Google DeepMind, Cornell University, and NVIDIA, it has been determined that GPT-leaning models possess a fixed memorization capacity, which approximates to about 3.6 bits per parameter. To comprehend this seemingly oblique statistic, a bit, the smallest fragment of digital data, holds either a 0 or a 1. Storing 3.6 bits allows for approximately 12.13 distinct values. Even though 3.6 bits are not sufficient to store one English letter or even a single ASCII character, it cleverly encodes a character from a reduced set of 10 common English letters.

A staggering revelation affirmed by the study underscores that the memorization capacity remains constant across different architectural variations, including various depths, widths, and precisions, regardless of model sizes or precision levels. Additionally, another remarkable fact accentuated by the research is that adding more training data does not necessarily amplify the memorization power. Rather, it seems that a model is less likely to memorize a single data point with an increase in training data.