Paul Olson is a software engineer with a background in platform engineering and observability, and is working on an engineer’s history of web architecture. Here’s how he describes the project:
The years spanning from 1993 to 2015 were extraordinary. Linux and FreeBSD became mainstream server operating systems, the NCSA httpd project was released, it begat the Apache HTTP server, which in turn inspired projects like nginx. The world saw the birth of a global network for information exchange and its evolution into the substrate of our everyday lives. We went from editing files and placing them in directories on a server to shipping fully containerized applications through complex pipelines. Client-server architecture became three-tiered architecture, eventually morphing into the microservice death star diagrams shared in conference talks as humble brags about the amount of infrastructure required to run large sites. Most of this innovation was done in the open, often by people collaborating across organizational and geographic boundaries. I owe my career to this time period and to the people who shaped it. The amount of knowledge sharing and learning, between practitioners figuring things out the hard way, that has happened during this time period is staggering.
I’m writing a book detailing the major shifts in web architecture that happened during this time period. Things continued to happen, of course, and 2015 is admittedly a bit arbitrary — but I chose this timeline because we had containerization and orchestration cemented. Much work that has happened since has been focused on ML / AI, which is out of scope for this project. I’m writing the book because it’s important to illustrate the connective tissue between these developments. It’s also important to appreciate how these contributions were made by practitioners working in specific contexts under specific constraints, and how their innovations compounded — sometimes unexpectedly. I want to dive into how the industry evolved from stateless web servers serving static documents through the C10K problem, through developments in caching reverse proxies, improvements to OS kernels, experiments with new programming models, infrastructure like CDNs, and beyond. I’ll treat security as a common thread that runs through each chapter, with each innovation addressing some challenges and introducing new attack surface area for others.
I think these kinds of projects are immensely valuable. Building an account of why things are the way they are, from the perspectives of people working in the trenches can leverage insights that are valuable for future projects. For those of us working in software engineering today, understanding this perspective is a form of architectural forensics. You can’t truly understand why a system is complex until you understand the problem it was originally designed to solve.
It’s also valuable for people outside of the world of software — while Osman’s intended audience is engineers, academics really benefit from this sort of undertaking, we can better understand the shape the web is taking when we understand that technologies aren’t inevitable, but responses to specific problems.
The book will be released progressively at buildingtheweb.dev, and you can following along by signing up at the site, or using your RSS reader of choice.
Quick aside — I’m planning to ramp up the frequency of these posts. There are some big (and very positive) changes in the works professionally. Expect a bit of an adjustment period, followed by a lot more technical deep-dives and interesting links!