The Second Father of MP3

Introduction

Every time you listen to music, a podcast, or download a track, you're using technology based on perceptual audio encoding — a method rooted in understanding which frequencies the human ear perceives, and compressing audio accordingly.

One of the key people behind this technology is James D. Johnston. IEEE calls him "the father of perceptual audio coding." He holds over 100 patents. Ken Thompson, who developed Unix, personally rewrote his codec, saying it significantly surpassed MP3. The U.S. Federal Appeals Court confirmed Johnston invented MP3 alongside Karlheinz Brandenburg. Yet his name rarely appears in digital audio history.

This is his story.

The Codec They Delayed Publishing

In 1988 at an IEEE conference in New York, two researchers realized they had invented the same technology. Johnston from Bell Labs stood at his booth explaining how to mathematically model human hearing for efficient audio compression. Nearby, Heinz Gerhauser presented identical research conducted by German doctoral student Karlheinz Brandenburg.

"We agreed we could have switched booths and nobody would notice. That's how I met Karlheinz," Johnston recalls.

Johnston began audio compression work in 1984, years before Brandenburg started his research. At Bell Labs, working on Alliant computers, he wrote test software asking a deceptively simple question: how many bits does music actually need?

"The goal was determining the bits needed to send music over a 20 kHz channel," he explains. "I got interesting results. I encoded material using a primitive PXFM codec that worked beautifully at 4 bits per sample."

His breakthrough was perceptual coding — discarding audio data the ear cannot perceive anyway, based on hearing science. Johnston solved a fundamental problem that enabled the advancement of digital music.

But AT&T delayed the work.

"AT&T refused to patent or release this technology. Internal business politics. The work was finished, and I was ready to publish in 1986, for God's sake."

This delay proved critical. By 1988, when AT&T finally approved publication, Brandenburg was finishing his dissertation on the same subject at Erlangen-Nuremberg University. Had Bell Labs management acted decisively, Johnston would have clearly preceded him.

The Collaboration Everyone Forgot

What followed is rarely documented. After that conference encounter, Brandenburg came to Bell Labs.

"Soon after, Karlheinz came to Bell Labs for a pre-doctoral internship, and we worked together on what we called a 'hybrid codec,'" James says.

From April 1989 through 1990, Brandenburg worked directly with Johnston in New Jersey developing what became MP3. This isn't speculation — it's documented in patents, court records, and Brandenburg's own public statements.

Brandenburg himself acknowledged this collaboration. In a LiveVideoStack interview: "At that Hannover meeting I first connected with AT&T Bell Laboratories, where James D. Johnston did essentially the same work as me. Later we combined efforts and worked together... we, together with AT&T and others, submitted ASPEC. Later this algorithm became MPEG Audio Layer III."

Layer III. That's MP3.

But their collaboration didn't end there. In 1995, Jurgen Herr, today one of audio coding's most respected figures, came to Bell Labs to work with Johnston on AAC — the format that eventually became iTunes' foundation and the leading audio codec.

"During that period Jurgen worked with me at Bell Labs," Johnston emphasizes. "Not at Fraunhofer Institute. No. Exactly as Karlheinz did when we standardized MP3. Yet nobody ever mentions this."

Ken Thompson's Epic Code Rewriting

One of audio coding's strangest historical episodes involved Ken Thompson — yes, the co-developer of Unix and C.

Johnston developed PAC (Perceptual Audio Coder), a different format that outperformed all competitors in blind tests. PAC never became a standard, but AT&T used it internally. Johnston's team struggled implementing it efficiently.

"Our guys tried splitting PAC into DSP sections — terrible idea. They converted vector Fortran to standard, then used F2C to translate to C. Four thousand lines became twenty-five thousand unreadable generated lines that wouldn't even compile," Johnston recalls.

Sandy Frazer, Bell Labs' legendary Unix group manager, reported this problem to Thompson, who later called Johnston.

"Ken called around noon saying 'send me that goddamn Fortran code.' So I did. Twenty minutes later he walked into my office at the corridor's other end saying 'I can't read this trash. Go to the Unix section and explain what it does.'"

An epic live-coding session followed.

"He rewrote everything as I explained the functions. Within a week the C code worked and was bit-for-bit identical to the original. Then we fixed several bugs — and presto! A real-time PAC encoder, ready to go."

Thompson wasn't merely helping — he became PAC's evangelist. In 1999, he told Computer magazine that PAC "significantly surpasses MP3" and admitted using this codec for his personal music collection.

The "iPod" That Never Happened

By 1998, three years before Steve Jobs announced the iPod, AT&T already had everything Apple later developed — a portable music player, a music download service, and the corresponding codec.

Their player was called FlashPAC. A deck-sized device that stored compressed audio in flash memory. "It existed. I held it in 1998," Johnston says. "And it worked perfectly."

The music download service, launched with BMG, was called a2b. It let users download tracks from the internet and play them on computers or upload to FlashPAC. In AT&T's 1997 annual report, the company proudly named Johnston and colleague Jack Lacy as creators of this technology.

Today, the FlashPAC prototype is stored at the Computer History Museum.

Management Failed AT&T

"AT&T killed this project because 'nobody will ever sell music over the internet. It's only for stealing tracks,'" Johnston says, frustration evident decades later. "iPod. 1998. Product ready, servers configured — essentially done. But internal marketing pressure ultimately destroyed it."

A devastating blow came in May 1999 when Larry Miller, Howie Singer, and 12 other senior managers and technologists — comprising more than half the a2b team — left for competitor Reciprocal. The New York Times covered this exodus. The service limped along until AT&T quietly shut it down in 2002.

By then, Apple was preparing the technological leap using technology Bell Labs had created and then abandoned.

Recognition That Never Came

Johnston's accomplishments are genuinely extensive. He's an IEEE and Audio Engineering Society fellow, received the "New Jersey Inventor of the Year" designation, plus the "AT&T Technical Medalist Award," "IEEE James L. Flanagan Signal Processing Award," and "IEEE Industrial Innovation Award." He holds over 100 patents.

IEEE calls him "the father of perceptual audio coding" — the theoretical framework underlying all modern audio codecs.

In audio coding circles — IEEE working groups, AES conferences, MPEG standardization meetings — Johnston is well-known and respected. Yet popular sources covering MP3's invention history rarely mention him. Fraunhofer Institute's marketing proved relentless and effective, portraying Brandenburg as the sole genius behind digital audio development.

In 2002, when AT&T dissolved Shannon Labs, Fraunhofer offered Johnston a five-year contract. He declined — the offer excluded health insurance, and his wife had just been laid off.

"Bernhard Grill turned purple," Johnston remembers. "He literally yelled at me in rage when I refused. And that became 'payback' ever since."

The $1.5 Billion Patent War

Eventually, the question of who truly invented MP3 became a legal matter.

In 2003, Lucent Technologies sued Dell and Gateway over MP3 patents developed at Bell Labs. Microsoft voluntarily joined the proceedings. Johnston was central to these disputes.

The claims involved two Johnston patents: U.S. Patent 5,341,457 ("Perceptual Coding of Audio Signals," co-developed with Joseph Hall) and U.S. Patent RE39,080 ("Rate loop processor for perceptual encoder/decoder").

Johnston testified under oath that he had working code years before collaborating with Brandenburg. Court records state: "Johnston testified he completed developing his perceptual transform decoder (PXFM) on April 2, 1987." Sworn testimony legally confirmed Johnston had working audio-coding software a full year before Brandenburg completed his doctoral dissertation.

In February 2007, jurors ruled that Microsoft must pay Alcatel-Lucent $1.52 billion in damages — $769 million per Johnston patent. This was among the costliest patent verdicts ever.

Later, Judge Rudy Brewster overturned the verdict, and the Federal Circuit Appeals Court upheld the judge. But the appellate decision contained one critical sentence: "Working collaboratively in 1989, Johnston and Brandenburg implemented and helped standardize the ISO 11172-3 Audio Layer 3 industrial standard."

Audio Layer 3. That's MP3. The Federal Appeals Court officially confirmed Johnston and Brandenburg created the MP3 standard together.

A Story Worth Telling

Johnston is now over 70, living on Washington's Olympic Peninsula. He left an industry that largely developed from his work, philosophically accepting his lack of proper recognition.

Perhaps Fraunhofer proved more aggressive in marketing and PR. Perhaps American corporations are less willing to credit their specialists than German research institutes. Maybe a "lone genius invents revolutionary technology" narrative sounds better than "international collaboration involving multiple institutions over a decade."

But the more dramatic story isn't always the truest one. Sometimes the real story is better — two researchers separated by an ocean independently solving the same problem, then combining efforts to create something neither could alone. A Unix pioneer personally rewriting code on the spot. A portable music player ready in 1998, destroyed by corporate myopia. World-changing technology with credit given only to one side.

James D. Johnston deserves to be remembered as MP3's second father, equally. The historical records support this.