ŷ

Jump to ratings and reviews
Rate this book

Database Internals: A deep-dive into how distributed data systems work

Rate this book
When it comes to choosing, using, and maintaining a database, understanding its internals is essential. But with so many distributed databases and tools available today, it’s often difficult to understand what each one offers and how they differ. With this practical guide, Alex Petrov guides developers through the concepts behind modern database and storage engine internals.

Throughout the book, you’ll explore relevant material gleaned from numerous books, papers, blog posts, and the source code of several open source databases. These resources are listed at the end of parts one and two. You’ll discover that the most significant distinctions among many modern databases reside in subsystems that determine how storage is organized and how data is distributed.

This book examines:

Storage engines: Explore storage classification and taxonomy, and dive into B-Tree-based and immutable log structured storage engines, with differences and use-cases for each
Distributed systems: Learn step-by-step how nodes and processes connect and build complex communication patterns, from UDP to reliable consensus protocols
Database clusters: Discover how to achieve consistent models for replicated data

376 pages, Paperback

Published November 4, 2019

645 people are currently reading
3,480 people want to read

About the author

Alex Petrov

1book55followers

Ratings & Reviews

What do you think?
Rate this book

Friends & Following

Create a free account to discover what your friends think of this book!

Community Reviews

5 stars
239 (47%)
4 stars
171 (34%)
3 stars
74 (14%)
2 stars
14 (2%)
1 star
3 (<1%)
Displaying 1 - 30 of 60 reviews
Profile Image for Sebastian Gebski.
1,148 reviews1,256 followers
February 9, 2020
One of the best tech books I've read in the last 12 months.
It consists of 2 parts: DB internals & DB distribution internals.

The 1st part is pure gold - one can learn about B*-trees, LSM-trees, differences between locks and latches, memory VS disk optimizations, rebalancing, concurrency models for transactions and much, much more. I can't recall any single book that covers as much deep-level knowledge on these topics.

The 2nd part is less unique - there are other good resources on distributed systems. What I liked was a solid description of Paxos algorithm (including variants: multi- and fast-). A chapter about anti-entropy was very solid as well (it's a pity I didn't have any comparable resource on the topic before I've started working with Cassandra).

No point in extending this review further - it's a great book, just grab it (if you're keen on the topic) - you won't find anything better.
Profile Image for Bugzmanov.
230 reviews89 followers
December 13, 2019
I liked this one a lot.

It complements nicely "Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems" by Martin Kleppmann.
While Kleppmanns' book provides pretty solid overview of data processing landscape, this book goes deeper into implementation details, data structures and algorithms.
It's a bit more dry and more technical, but still it's a relatively easy read.
Profile Image for Emre Sevinç.
171 reviews422 followers
May 29, 2020
“Database Internals: A Deep Dive Into How Distributed Data Systems Work� by Alex Petrov, belongs to a very special category of O’Reilly books such as “Designing Data-Intensive Applications� and “Cassandra: The Definitive Guide�, in the sense that it is a serious deep dive into the most fundamental and challenging aspects of big and distributed data systems that we rely on daily basis.

Today there’s an unprecedented proliferation of distributed database technologies, combined with an ever-growing multitude of cloud computing services offering them, as well as rapid advances in physical storage systems such as NVMe SSDs that force considering different trade-offs and algorithms. A working software developer, system engineer, solution architect, or a CTO can easily be overwhelmed with so many distributed NoSQL, newSQL, time-series, graph, document, key-value, embedded databases in addition to typical, traditional, enterprise RDBMS variants. Luckily, all of these fancy distributed database technologies are built on a limited number of concepts, techniques, and algorithms which are concisely introduced and surveyed in “Database Internals� book.

Profile Image for Adrian.
153 reviews28 followers
September 18, 2020
Unfortunately i have read this book after Martin Kelppman's Designing Data Intensive apps and probably this is the reason i rated it poorly.

The book starts really good by describing the intetnal systems that are encompassed in any dbms ( Connection Listener layer , Query parser+ optimizer layer , execution layer and of course the storage layer).

I wanted the book to explore more on this subject , how the components are drsigned and how they deal with concurrency etc.

The book then took a deep dive into tree-based data structures ..as in really deep.
I found myself at times wondering why am i reading this.It was way too terse.

Now after this part the book became interesting once again by the time it started tackling distributed transactions,consensus ,replication,byzantine faults, paxos algorhitms.

I was already familiar with all of those which in my humble opinion were tackled slighty better in the first book i mentioned.

All in all a good book , but i wouldn't classify it as complementing Martin Klepmann's
Profile Image for Mikhail Filatov.
335 reviews13 followers
April 2, 2020
The book is a strange mix - the second part is really about distributed data systems and it's ok - while "Designing Data Intensive applications" is better in this part.
The first part contains a lot of descriptions of different implementations of B*-Trees (replace * with any other symbol(s)) - most of them unreadable.
Profile Image for Bilal.
113 reviews9 followers
April 28, 2020
The book is divided into two parts: The first part deals with storage on hard disk and solid state storage but in the context of a singular system; while the second part deals with distributed systems. In this sense it differs from most other books on distributed storage that typically do not discuss the topics in the first part of this book.

I found the book informative, but not very effective in building a solid understanding of concepts. I felt the author jumps from idea to (related) idea too frequently in the manner of short paragraphs, and in so doing doesn't see an idea through to the end in enough detail for it to be learned properly. Perhaps the first part was better presented; the second was not.
Profile Image for dantelk.
191 reviews18 followers
October 6, 2024
This book deserves five stars, with its deeper sea-exploration of distributed system design patterns and explanation of database tree structures. I should confess that I couldn't help myself thinking "how's this going to help me in real life", especially file systems etc, but overall, great deep level concepts explained in an understandable way. Some concepts such as Paxos etc were above my tech knowledge, and I have to supplement the context with youtube videos. Still, i'll give it 5 stars, just the reading list is absolutely great, a lot of efforts have been given into this book.
Profile Image for Povilas Balzaravičius.
28 reviews
November 5, 2020
Good and interesting content. But some chapters are scattered with missing transitioning between topics or algorithms. Some other parts are developed well. Diagrams are missing where I have expected better explanation or are present for obvious things. The writing is 3/5, but the content is 5. So it is 4/5. I'm glad I've read the book and definitely will get back to some chapters to refresh some details.
Profile Image for Eric.
4 reviews
February 1, 2021
This book really feels like two incomplete books in one. The first half of the book focuses on database internals, file formats, caching strategies etc. The second half of the book switches gears and dives into the components (algorithms, and strategies) used by distributed systems.

The problem is that there is nothing to tie the first and second parts of the book together. You could be reading entirely different books. The second issue is that even within each part, you are presented with a lot of great information, but there is no guidance (imo) on how you may want to logically put the pieces together to build a complete system.

True to the book's title, it does a great job of exploring the internals of database systems. If you are looking for how a specific component is built (ie you want to learn more about RAFT consensus), this is a great resource. If you are looking for a book that ties these concepts together however, I would suggest looking elsewhere.
Profile Image for Vishwanath.
44 reviews6 followers
May 3, 2020
Informative but would have preferred more examples with practical scenarios. No code and this all mostly conceptual. Some good references to papers for subsequent reading. The first part of the book deals primarily with storage and covers an in-depth discussion of b-trees and types. The second half is focused on distributed systems and has useful sections on consensus protocols. Concepts like "2-phase commits" are explained well with figures. However, the lack of practical examples/code and overall dry subject matter made this a laborious read. Good book to reference theoretical concepts.
Profile Image for Łukasz Słonina.
124 reviews25 followers
April 7, 2020
I like this book for the content, if you would like to know more about databases and distributed systems plus get long list of further reads then go for this book. What I don't like is actually that this material does not read like book (e.g. DDIA), it's more like compendium of algorithms, data structures and theories. Some of the algorithms could be better presented (more diagrams).
Profile Image for Szymon Kulec.
206 reviews114 followers
October 21, 2023
If you're looking for a book that answers the question How all these dbs work?, this is the book. The broadness of this question can also trick you into doubting that one book can cover it all. This book tries to do its best. How it is done?

The book is split into two parts where the first covers the local storage and the second covers the distributed aspects. I must admit that for such a short book the author does an amazing job into squeezing a lot of knowledge into it. You'll find information about B+ trees, logs, ARIES, including most recent developments like RAMP transactions etc. Paxos, Raft? It got you covered.

All the descriptions are supported with drawings that are coherent and show exactly what they need to. Whether it's a log in Raft or messages sent in a 2PC or 3PC, they are clear and easy to follow (not in a sense that Paxos is easy to follow:P but rather that they do their job).

Why not 5 stars then? I think that it should be either two books or the book should be longer. I don't think that a reader that was not familiar with any of these can chew through it in a reasonable time manner and grasp these concepts. At the same time, if you heard about Raft, or what B+ tree CosmosDb uses, and want to understand what you heard, this is a book definitely for you.
Profile Image for Marcin Golenia.
39 reviews7 followers
April 25, 2021
The book is organized into 2 parts and let me review the book in two parts.

1. Storage Engines (5/5)
I didn't expect that we will get so much into internals. Hats off! The knowledge of the author is extensive and nicely presented by the book text and illustrations. Everything you need to know - hardware (HDD, SSD) and the relation between them and data storage algorithms, transactions, slotted pages, b-trees variants, LSM. Great stuff.

2. Distributed Systems (4/5)
There is a gap between complexity level in introduction of topics and the actual "meat" that is described. It starts nice and easy then boom! "Percolator transaction execution", "Calvin" and "Spanner". Hard stuff and this part may keep you reading the page few times so you can understand it. I failed to do so in few places but I am fine with this - The author provides many references so you can learn the intermediate knowledge from there.

I know I will forget some of the advanced things that Alex tried to explain to me, but I will know where should I look for it ;) All in all the book will help you to build nice end2end understanding of how the database really work in both places - your computer and a big cluster.

Profile Image for Ahmad hosseini.
313 reviews69 followers
February 20, 2021
I part 1, book explains internal database structure in details and examines its parts like storage engine very well.
“The storage engine (or database engine) is a software component of a database management system responsible for storing, retrieving, and managing data in memory and on disk, designed to capture a persistent, long-term memory of each node.�
Part 2 explains distributed systems characteristics in general and examine some specific topics related to distributed databases.
Book also introduces good sources for further reading
Profile Image for Bartosz Sypytkowski.
45 reviews11 followers
August 9, 2020
This book comes along nicely together with "Designing Data-Intensive Applications" by Martin Klepmann: they both focus on core, fundamental concepts of persistent, distributed systems, providing wide variety of known algorithms and protocols for common problems in that area, including rationale behind each one, which helps to build intuition about their trade offs. It's also full of references for anyone, who wants to continue more in-depth exploration for a given topic.
Profile Image for Leo.
301 reviews25 followers
April 5, 2022
I've found some similarities between this one and my favourite "wild board book" ("Designing Data-Intensive Applications" by Martin Kleppmann), though in the first part, dedicated to DB internals, it goes much more in-depth, and there are tons of awesome stuff there. Second part, dedicated to Distributed systems, is a bit less unique, but still very good, and worth reading.
So, wholeheartedly recommend to anyone working with DBs, or just distributed systems in general.
Profile Image for Tiago  Caxias .
14 reviews
April 22, 2024
This is a mandatory book for the data engineering, database and architects worlds.
It has (literally) two parts, covered in great detail:
1. Storage Engines
2. Distributed Systems

For these two matters, this book is outstanding.

Some other database topics, such as query optimizers, were left out intentionally.
Profile Image for Rafał Łasocha.
4 reviews2 followers
September 6, 2023
The book was very uneven when it comes to level of detail it describes and the scope of the book is way too big, author should've focus on databases, instead of pulling distributed systems into it.

In the first part, I really liked how the author went deep and into the level of bit-masks of specific data structures when it comes to explaining how the data is stored, how pages work, how variable-sized data is moved and how DBMS approach storing different versions of the same data records for the sake of transactions.

What I did not like however is variety of B-Trees optimizations being described. I have no idea why did I read these chapters, I am sure I completely forgot them by now. It may be useful when you're actually, right now, implementing a database and want to have some options given to you on the table, but except that, it's just better to skim over these chapters (except the practical implementations like B*-Trees of course). I found it confusing why the author explains some B-Trees optimizations with such detail, but then when there is a subchapter about data recovery after database crash, there is just short, four-step, incredibly high-level description of the recovery process with the statement that it can be read in some paper. That's how it is, B-Trees optimizations described in very detailed, many more interesting things have only an overview.

When it comes to second part -- when it comes to majority of the content, it's just better to read Martin Kleppman's book. If someone has already read Martin's book, many chapters will be boring, they may be useful if someone didn't. When it comes to consensus algorithms, I have flashbacks from beginning of the book, because now, all different varieties of the Paxos algorithm are described. I've forgot these variations next day. Raft is nicely explained, but raft is elegant algorithm :) In retrospective, I have no idea why the author added majority of the second part into the book. Although of course databases are inherently distributed systems, if you want to "know more about databases", you probably don't mean "I want to know a lot about different consensus algorithms".

Not a bad book at all if someone doesn't know much neither about databases nor about distributed systems, though I imagine it may be incredibly hard to grasp all of the definitions described in the book.
Profile Image for Lauro Caetano.
8 reviews6 followers
April 2, 2020
Excellent book! It goes a bit in the direction of what Design Data-intensive applications goes when it talks about distributed systems, dist transactions and so on.
But this book goes some steps further: explaining how the db represents data internally, and also explaining distributed systems algorithms.

Excellent read!
Profile Image for Ivan.
223 reviews10 followers
December 30, 2020
Детальное, но без больших подробностей (это искупается большим числом ссылок и рекомендаций для дальнейшего изучения) описание структур и алгоритмов для современных систем.
16 reviews1 follower
December 31, 2021
Great, content-wise. Although appeared to have lost the flow/transitions in describing concepts.
Had to continuously take notes and cross-refer them so as to prevent myself from loosing the flow
Profile Image for Orkhan Huseynli.
9 reviews
April 28, 2025
The book does a great job on the Part I, explaining internals of databases (as the name suggests). You are going to learn a lot about why B-Trees are important, and why you can't just use AVL or RedBlack tree instead of B-Tree. It talks about architecture of databases and storage engines and how some databases use existing storage engines to not to start from scratch. It also talks about how database pages are managed and evicted, as well as page layout on disk as well as finally answered my question about OS pages and database pages. Moreover, thanks to this book I finally know what is a latch. At the end of Part I, you will learn about LSM trees as well.

However, I found LSM tree explanations a bit confusing and for me DDIA book did a better job explaining LSM tree than this one. Overall, most of the chapters the author talks about B-Trees, so he could name the book B-Tree internals as well.

What I did not like about the book is Part II. Part II was about distributed systems and somehow it turned out to be DDIA in a nutshell but more confusing. I would expect this book to talk about real applications of replication, leader election and consensus, but even in the chapter called "Replication and Consistency" it does not talk about replication (e.g. it could have mentioned types of replications as DDIA book did). Also on the chapter about "Distributed Transactions" the author feels remorse and decides to write about Database Partitioning in a small paragraph. I would expect this book to talk about partitioning in a dedicated chapter and show real life examples, how MongoDB, Postgres or MySQL does it.

So if you don't want to get confused about the distributed systems, I would recommend reading DDIA and just pick some topics from this book to learn more about them where DDIA does not deep-dive (e.g. Paxos and Raft).
Profile Image for Phil Eaton.
113 reviews288 followers
January 30, 2024
A solid guide told from a unique perspective. It is clearly told from the perspective of Cassandra which is interesting since most of what I find online is from the perspective of OLTP databases. Of course this perspective makes sense since the author is a Cassandra contributor. This perspective, however, does not mean that the book does not do a good job covering aspects that are not specific, or not related at all, to Cassandra.

The second half was stronger than the first. My primary critique is that parts of the book could have been better edited for clarity. Some ideas were discussed in an order that was difficult to follow until reaching the end of a section. I can imagine the difficulty though of finding editors who can help make sense of such complicated topics.

All in all, a solid introduction to BTrees, LSM trees, replication, partitioning, and consensus. Partnered with a good editor, a future edition could be 5 stars.
Profile Image for Peter Aronson.
392 reviews17 followers
February 19, 2025
Four-and-a-half stars. What's in the book is solid, mostly well explained, and useful. But it's a bit of a miscellanea -- a bit of this and a bit of that -- all organized around the subject of databases. The first half of the book, Storage Engines, seems more complete. The second half, Distributed Systems, seems more like a survey with references. Which is odd, given that the first sentence of the Preface is "Distributed database systems are an integral part of most businesses and the vast majority of software applications". (Rather an extreme over statement in my opinion.) With that opening, I would expect a more complete coverage of Distributed Systems, even if it is a bit topic that would have required a bigger book. But this is still a useful book on both topics.
143 reviews4 followers
December 26, 2024
This book is an excellent treatment of how database system internals work. The first section of the book covers practical items like file systems and other local-related challenges - there is a wealth of information to dive into here, and I already have a large collection of books to dig into just from the references here. Then, in the second half, you dive into concurrency and distributed systems. While I've absorbed the info like a sponge, I am already looking forward to coming back to this after spending some additional time with some other distributed systems readings.

A truly excellent book that is truly a wealth of knowledge.
112 reviews5 followers
June 24, 2020
A book that I really appreciated to expand and have a 360 degree refresh on Database essentials. The progression of the book is built in an organic way, parting from basic concepts at low level implementation to modern distributed challenges. This helps you build comprehension naturally, in constructs with some other technical books that jump from topic to topic without any respect to cognitive challenges.

Totally recommended to get a solid understanding on databases to help solve contemporary problems.
22 reviews
July 31, 2022
Solid coverage of concepts related to database technologies. I felt some concepts weren't super intriguing and I didn't think connected well. The distributed systems portion was not as well-done. I did learn some core ideas, but I don't feel as well-read on them compared to the database concepts. I also think a real-world section of what services are commonly used in the real world to handle these problems as well as their trade-offs would be a nice addition.
Profile Image for Diego Gomez.
31 reviews
October 21, 2024
My first resource diving into the world of database internals, and it's an incredible introduction to the subject.

The book provides a comprehensive overview of most aspects of storage engines, offering a detailed look at how distributed data systems function under the hood.

We often underestimate the complexities behind the technology we use daily. We are truly standing on the shoulders of giants.

If you’re looking to deepen your understanding of database systems, this is a must-read.
Displaying 1 - 30 of 60 reviews

Can't find what you're looking for?

Get help and learn more about the design.