A developer found a security flaw that could have brought down systems worldwide: the story of Andres Freund

Late last March, the tech world was rattled by an unsettling disclosure: a developer had discovered a security vulnerability capable of bringing down systems worldwide. More alarming still, if the flaw was not fixed in time, hundreds of millions of computers around the world could have fallen into hackers' hands.

Beneath the surface of the Internet lies a tangled mass of aging technology that still runs remarkably well. Diligent people take on the responsibility of maintaining it. Much of this foundation is open-source software, kept running by an “army” of developers of varying skill levels who fix bugs and patch vulnerabilities to keep the Internet's enormous data flows moving smoothly.

At the end of March, one soldier in this army averted a disaster that was already under way. According to The New York Times (NYT), his name is Andres Freund, a 38-year-old developer living in San Francisco and working for Microsoft.

Freund's job was developing part of PostgreSQL, the open-source database software. During routine maintenance he discovered a backdoor hidden in a piece of software that ships with Linux. In all likelihood, that backdoor could have opened the way to a cyberattack on an enormous scale. Fortunately, it never happened.

The NYT approached developer Andres Freund for an interview, but he declined to be photographed, saying that becoming an “internet hero” had left him uncomfortable.

Although Freund is uneasy about the informal title, his achievement has been widely welcomed by influential voices in the technology and security community.

Satya Nadella praised Freund's "curiosity and craftsmanship", while developers shared a comic strip by Randall Munroe in which the American cartoonist joked that the foundation of modern digital infrastructure rests on "a project some random person in Nebraska has been thanklessly maintaining since 2003".

In the context of this story, the “accidental figure” being referred to is the developer Andres Freund. “I find all of this very strange,” he told the NYT. “I am a private person who just sits in front of a computer and tinkers with code.”

Freund's story begins in January 2024, shortly after he returned to the US from a visit to his native Germany. While reviewing a batch of automated tests, he noticed some errors he had not seen before. But worn out from the long flight, and figuring the error messages were not especially serious, he filed the memory away and moved on.

But a few weeks later, while running more tests at home, he noticed that SSH — the tool used for remote computer access — was using more resources than normal. Tracing the issue, he found it stemmed from a data compression tool called xz Utils, and began to suspect a connection to the error messages he had seen not long before.

A note from the NYT reporter: readers do not need a deep understanding of all the proper names mentioned here — it is enough to know that these are small components of the Linux operating system, widely regarded as the most important open-source software in the world. Most of the world's servers — including those run by banks, hospitals and governments — run on Linux, so a Linux security vulnerability carries global risk.

Like other open-source software, Linux is updated continuously, and most bugs originate in mistakes made without malicious intent. But when Freund dug into the xz Utils source code, he found traces of deliberate modification by someone — or some organization.

Specifically, he found that someone had planted malicious code in the latest version of xz Utils. The backdoor would let whoever planted it use SSH connections to run software remotely in secret.

The malicious code Freund discovered could have given attackers a way into computers worldwide.

New York Times (NYT) reporter Kevin Roose compared the data engineer's discovery of malicious code in the Linux system to a bakery owner noticing an odd smell in a fresh batch of bread and tracing the problem back to the yeast supply chain. A discovery like this takes years of experience, an eye for even the smallest detail, and a great deal of luck.

“A surreal feeling,” Freund said of the moment he discovered the backdoor planted in the Linux operating system. “At times I felt this was the result of a sleepless night, or a dream born of a delirious fever.”

The deeper he looked, the more evidence he found. In late March, Freund sent what he had gathered to a group of open-source developers. The news traveled through the community immediately, and the flaw was patched within hours. Some developers have said that without Freund, a cyberattack that would have gone down in history could well have taken place.

“It could have been the most widespread and most effective backdoor ever planted in any piece of software,” said Alex Stamos, a senior executive at SentinelOne, a cybersecurity research firm. According to Stamos, the vulnerability could have “handed its creator a master key to hundreds of millions of computers running SSH worldwide.” An attacker could have stolen sensitive information, planted malware and disrupted global infrastructure — without ever being caught.

Mr. Freund declined to comment on who, or which organization, may have planted the malicious code in the system. He did say that whoever was behind it was skilled and cunning enough to erase their tracks, and even used several techniques to make the malicious code harder to find.

Since discovering the vulnerability, Freund has continued to help a group of developers reverse-engineer the code in order to analyze it and identify who was behind it. He is also developing and finalizing a version of PostgreSQL of his own, expected to be released later this year.

“I really do not have time to raise a glass and celebrate,” said the German-born coder.

Andres Freund's story is not only about finding a security vulnerability; it is about one person's determination and effort to protect the online community. Although he declined the recognition offered to him, Freund has continued to contribute to the developer community, helping identify and fix latent vulnerabilities in software. It demonstrates the power of community in defending global cybersecurity.