TigerBeetle is a most interesting database
amplifypartners.com259 points by todsacerdoti 7 hours ago
259 points by todsacerdoti 7 hours ago
As nice as TigerBeetle is, do keep in mind that this article is written by an investment company that has invested in TigerBeetle: https://www.amplifypartners.com/blog-posts/our-investment-in...
I have a few of these posts I've written coming out over the next few months that I want people to discuss. Would you prefer I add a disclaimer at the top? Easy enough to add
Personally, I'd say you should definitely add a disclaimer, otherwise a discussion that would likely gain traction is that the information might be being intentionally concealed.
I will add them in subsequent posts. The information is all over our site (they are on our portfolio page, and we wrote about them when we invested) but I am happy to reiterate on each post for each company
If you care about ethics of journalism then yes you should obviously disclose a financial interest in what you write.
Yeah I'm not going to say what I think of the way it's written but suffice to say that was obvious from how it was written.
While I am a fan of TigerBeetle’s general stance on correctness, coding practices, and their desire to hyper-specialize, I have some critiques on the post.
I think the paragraph about multi-node is a bit misleading. Contrary to what cloud native folk will tell you, a single beefy DB, well-tuned and with a connection pooler, can serve a dizzying amount of QPS just fine. At a former employer, during a maintenance period, I once accidentally had all traffic pointed to our single MySQL 8 RDS instance, instead of sharing it between its read replicas. That was somewhere around 80-90K QPS, and it didn’t care at all. It wasn’t even a giant instance - r6i.12xlarge - we just had a decent schema, mostly sane queries, and good tuning on both ProxySQL and MySQL. At peak, that writer and two .8xlarge read replicas handled 120K QPS without blinking.
A DB hosted on a server with node-local NVMe (you know, what used to be normal) will likely hit CPU limits before you saturate its I/O capabilities.
For redundancy, all RDBMS designed for networked activity have some form of failover / hot standby capability.
My other mild criticism is in the discussion on TigerBeetle’s consensus: yes, it seems quite clever and has no other dependencies, but it’s also not trying to deal with large rows. When you can fit 8,190 transactions into a 1 MiB packet that takes a single trip to be delivered, you can probably manage what would be impossible for a traditional RDBMS.
None of this should be taken as belittling their accomplishment; I remain extremely impressed by their product.
> My other mild criticism is in the discussion on TigerBeetle’s consensus: yes, it seems quite clever and has no other dependencies, but it’s also not trying to deal with large rows. When you can fit 8,190 transactions into a 1 MiB packet that takes a single trip to be delivered, you can probably manage what would be impossible for a traditional RDBMS.
Isn't that the point? They're saying to separate out transactions workload from other workloads. They're not saying they'll replace your OLGP db, you remove transactionally important data into another db.
It's something similar that we see with another db: https://turbopuffer.com/
That a traditional modern RDBMS is very fast is not really useful here because the use case of Tigerbeetle includes work loads with high contention. Under that kind of load, in their own tests you can see in a recent conference talk, you can see the throughput drops dramatically in clusters where many transactions affect a single account.
edit: from the horses mouth is better https://news.ycombinator.com/item?id=45437046
Huh, that as not been my experience at all.
Too be fair, its been something like 10 years iirc.
The database in question was MySQL 8, running on plain old enterprise ssds (RAID 10)
The workload was processing transactions (financial payments)
The database schema was ... Let's call it questionable, with pretty much no normalization because "it's easier when we look at it for debugging", hence extremely long rows with countless updates to the same row throughout the processing, roughly 250-500 writes per row per request/transaction from what I recall. And the application was a unholy combination of a PHP+Java monolith, linked via RPC and transparent class sharing
DB IO was _never_ the problem, no matter how high qps got. I can't quote an exact number, but it was definitely a lot higher then what this claims (something like 40-50k on average "load" days like pre Christmas etc)
Not sure how they're getting this down to ~250qps, it sounds completely implausible.
Heck, I can do single row non-stop updates with >1k qpm on my desktop on a single nvme drive - and that's not even using raid.
Using your numbers and speaking roughly, if you're doing 50k rps and 500 typically to the same row wasn't your contention around 1%? TigerBeetle clais to be able to handle work loads with very high contention, for example 80 or 90% of transactions filtering out a fee to a single account to make up a use case.
Ah, you might be right!
Contention is what can be parallelized, right?
So with roughly 100-200 requests/s you end up with 1-0.5 contention if I understood that right.
That moves me even further towards agarlands points, though - if I plug that into the equation, I end up with >50k qps.
The used numbers create an insanely distorted idea wrt real world performance
I really like the work Joran and his team have done around DST, distributed systems awareness, and performance practices. I especially love the craziness around no dependencies (well, could you consider an underlying OS a dependecy?).
But I've always felt the way they treat normal OLTP (they call OLGP) seems unfair. For example, comparisons using clear sub-optimal interactive SQL transactions for financial workloads, like locking rows rather than using condition checks at commit time, because "that's how OLTP was intended to be used when it was designed ~50(?) years ago".
In their cited https://tigerbeetle.com/#performance the lowest the slider can go is 1% contention. Do you think Stripe has 1% contention directly on an OLTP DB? Definitely not.
You can build systems that _expect contention_, and elegantly handle it at REALLY high throughput. These systems protect the DB from contention, so you can continue to scale. From talking to folks working on these systems, I roughly know the kinds of transactional (financial) throughput of DBs like Stripe's and other systems - they have _many_ more zeros behind them than their performance comparison page proposes they could possibly have at even 0.01% contention.
Their marketing largely ignores this fact, and treats everyone like they just slam the DB with junior engineer-designed interactive transactions. Most developers (I hope) are smarter than that if they're working at a payments company. There's even the title "payments engineer" for the kind of person that's thinks about scaling contention and correctness all day.
TigerBeetle is great, but I find the pattern of being quite misleading about other OLTPs off putting.
Hey Dan, appreciate the compliment.
Do you think it would be more fair to suggest that OLTP workloads have 0% contention? That debit/credit sets never intersect?
In practice, we're seeing that our customers are already some of the largest brokerages, wealth managements or exchanges, in their jurisdictions, even national projects, and all of them with decades of Postgres etc. experience, some even running installations with 200 Postgres/MySQL machines backing their sharded core ledger.
They're not junior engineers. For example, they know about stored procedures, but the problem of concurrency runs deeper, into the storage engine itself.
At least for our customers, for the seasoned payments engineers, OLTP contention is a killer. They're tired of scaling outside the DBMS, and the complexity of expensive reconciliation systems. Several of them are even leaving their Chief Architect positions and starting TB-related startups—I know of at least 4 personally, from fintech brands you will recognize, and probably use.
I hope the spirit of our talks, trying to get the big ideas across, to make people aware of contention and Amdahl's Law, is clear, and that you take it in good faith, to the extent that we show it ourselves.
> Do you think Stripe has 1% contention directly on an OLTP DB? Definitely not.
Stripe runs on top of MongoDB, which is horrifying in its own right, but in any case comparing them to a shop running an RDBMS is apples to oranges.
Stripe delivers incredible uptime. This type of comment is beneath this platform.
I didn’t say they’re bad at their job, I said I find the technology choice horrifying. That’s a personal take, not a value statement.
>well, could you consider an underlying OS a dependecy?
I've worked on systems that ran entirely in memory, and continued running during kexec. You can't do syscalls. So yeah :)
We were considering TigerBeetle, but found blockers:
* We use Cloudflare Workers. TigerBeetle client app is not supported. It might work using Cloudflare Containers, but then the reason we use Cloudflare is for the Workers. --> https://github.com/tigerbeetle/tigerbeetle/issues/3177
* TigerBeetle doesn't support any auth. It means the containing server (e.g. a VPS) must restrict by IP. Problem is, serverless doesn't have fixed IP. --> https://github.com/tigerbeetle/tigerbeetle/issues/3073
But Cloudflare Workers or AWS Lambda setup would not work anyway with any db?
* spawning 1000 workers all opening a connection to a db,
* solved by service/proxy in front of db,
* proxy knows how to reach db anyway, let's do private network and not care about auth
DBs like DynamoDB work great with these kinds of runtimes, they don't need a connection pool in front
Cloudflare has its own DB (D1, Sqlite-derived), but you can also connect with PostgreSQL using their adapter (Hyperdrive). I have used both, they're okay.
I wouldn't recommend D1 for now due to its harsh storage limitations (10 GB).
You should chat to our solutions team, who'd be able to help you solve auth etc. at the logical level with end-to-end encryption, something we typically do for our customers.
I might not be able to afford paid support yet since it's just PoC, but if it's okay to be public, could they respond to the GitHub issue instead so it can benefit others? https://github.com/tigerbeetle/tigerbeetle/issues/3073
Wait, what? A database in 2025 doesn't support any kind of auth? A financial database? WTF?
C'mon folks, the least you can do is put a guide for adding an auth proxy or auth layer on your site.
Particularly since you don't use HTTP (cant easily tell from the docs, I'm assuming), then folks are going to be left wondering: "well how the hell do I add an auth proxy without HTTP" and just put it on the open internet...
Joran from TigerBeetle here!
TigerBeetle is our open source contribution. We want to make a technical contribution to the world. And we have priorities on the list of things we want to support, and support properly, with high quality, in time.
At the same time, it's important I think that we encourage ourselves and each other here, you and I, to show respect to projects that care about craftsmanship and doing things properly, so that we don't become entitled and take open source projects and maintainers in general for granted.
Let's keep it positive!
> TigerBeetle is our open source contribution.
Wait, is it open source?? Since when? I always thought it was proprietary
Apache 2.0 since day 0, since it came out of a non-profit central bank switch by the Gates Foundation, which is also Apache 2.0.
Our view is that this kind of infrastructure is simply too valuable, too critical, not to be open source.
Exactly my thoughts, and yes, it's not HTTP. Not a big deal, I guess, if they explain how to work around it...
> In less than a decade, the world has become at least three orders of magnitude more transactional. And yet the SQL databases we still use to power this are 20-30 years old. Can they hold up?
Errr yes. Without much sweat really.
Just because something started ~30 years ago doesn't mean it hasn't updated with the times, and doesn't mean it was built on bad foundations.
Joran from TigerBeetle!
Without much sweat for general purpose workloads.
But transaction processing tends to have power law contention that kills SQL row locks (cf. Amdahl’s Law).
We put a contention calculator on our homepage to show the theoretical best case limits and they’re lower than one might think: https://tigerbeetle.com/#general-purpose-databases-have-an-o...
>Traditional SQL databases hold locks across the network; under Amdahl's Law, even modest contention caps write throughput at ≈100–1,000 TPS
In fact large real world systems are not limited to 100-1000 TPS, or even 10 kTPS as the calculator tries to suggest. That's not because Amdahl's law is wrong, the numbers you're plugging in are just wildly off, so the conclusions are equally nonsensical.
There might be some specific workloads where you saw those numbers, and your DB might be a good fit for this particular niche, but you shouldn't misrepresent general purpose workloads to try to prop up your DB. Claiming that SQL databases are limited to "100-1000 TPS" is unserious, it is not conductive to your cause.
They’re talking about 100-1000 TPS for transactions all locking a single row, which is not wrong, just not reflective of most workloads. They’re not talking about TPS of the entire database with simultaneous operations on many independent rows. This should be reasonably clear in context, but of course when you publish grandiose claims (when viewed in isolation) and very vague graphs backing said claims people won’t be happy.
TFA contextualizes this better:
> This gets even worse when you consider the problem of hot rows, where many transactions often need to touch the same set of “house accounts”.
This whole page, and their response in this thread, is about tigerbeetle as a transaction processing database - e.g. financial transaction processing
I think this is very clear, I don't know why you're saying that tigerbeetle is trying to make a generic claim about general workloads
The comment you're replying to explicitly states that this isn't true for general workloads
Its a financial database built for use cases where this invariant holds and built for enabling new use cases where this invariant prevented businesses from expanding into new industries. The creator says as much:
> Without much sweat for general purpose workloads.
Having a niche and expanding into new industries are all fine, there is no problem with having a DB filling a particular sub-segment of the market.
But writing that traditional SQL databases cannot go above these "100-1000 TPS" numbers due to Amdahl's law is going to raise some eyebrows.
He clearly says this is in the context of "transaction processing" in the comment you're responding to.
> But writing that traditional SQL databases cannot go above these "100-1000 TPS" numbers due to Amdahl's law is going to raise some eyebrows.
I don't think that's controversial. Amdahl's law applies to all software. Its not a peculiar feature of SQL databases. The comment is well-contextualized, in my view, but reasonable minds may disagree.
You’re changing the subject to the company’s mission when the concern was about a specific claim made.
A specific claim about OLTP processing under contention. Or how would you characterize the specific claim, specifically?
So, the pitch for TigerBeetle is... "you can do database schemas wrong and we're performant enough"? (and also we don't have auth)
DNS still runs strong and it was ~~designed~~ released* in the November 1983.
It still holds up basically whole of internet.
in most cases SQL is good enough for 90% of workloads.
Isn't DNS a catastrophic clusterfuck which also had to be patched up multiple times to not allow people burning down the whole world? I mean, yes, it's doing its job, but mainly because of people doing their job.
People who are malicious or see an opportunity for gain will always abuse a system. Just because the DNS system has had to be patched to handle the abuse doesn't mean its badly designed. If it were as badly designed as your comment seems to imply it wouldn't be able to operate even half as well as it does.
The combustion engine's fundamental design is pretty damn good and that it had to be updated to handle unleaded gasoline isn't a knock (pun... intended?) against that design.
Quite the opposite: as it designed in a way that it could be patched and extended, while also being performant.
What would DNS look like if it were released in 2025?
It would be completely centralized with a micropayment rent seeking solution to update records.
I think it would actually be (pseudo-)decentralized and you'd have to mine a blockchain node containing each domain, where its creators will have reserved at least 10% of its node pool to make sure they became billionaires if it took off.
Plus micropayments, of course :-)
Nono, it would run decentralized in blockchain with a mircropayment rent seeking solution to update records.
You’re right that it would run on a block chain, but that fact would primarily exist to power some marketing. Everybody would end up interacting with it through a single centralized web site and API because it’s the only usable way to get it to work.
Owned by GoogleFlare with public spec, but most of it would be run as closed source application that uses extended spec akin to EEE - making it impossible to use without relying on GoogleFlare.
config defined in YAML.
>config defined in YAML.
when I woke up today, I didn't really expect to be convinced that we live in a relatively good timeline, but...
Certainly a JSON-based REST API that is not really RESTful just so that people can complain about it
Certainly some AI features
And it would make it really painful.
But we couldn't get rid of it because it papered over something important.
Are you telling me that new shiny things are not always better that established, time tested and boring tech?
Sometimes I feel like we software engineers have the worst memory of any engineers.
I don’t even think it’s memory, it’s lack of experience. A lot of the battle-tested infrastructure is now abstracted away to the point that a lot of devs are hard-pressed to explain the difference between AWS’ ALB and NLB, much less name OSS versions of them.
Actually, load balancers are a great example. The number of times I’ve seen a team re-implementing HAProxy, but poorly, is entirely too high. The reasoning is always something along the lines of “we need this extremely specific feature.” OK, and have you even read the manual for HAProxy? Or if that feature truly doesn’t exist, did you consider implementing that separately?
True, the only right answer for a question like "Should we use X here?" is "It depends". Maybe NIH is the right answer in some specific case but not on another that seems quite similar to the first.
Some months ago I was re-enlightened when Anders Hejlsberg (creator of C# and TypeScript) explained why they chose Go for reimplementing the TypeScript compiler, instead of using any of those languages or something like Rust.
The way they defined the problem, the tests they did and how they justify their pick is how these kind of decisions should be made if we want to call ourselves engineers.
I disagree. If we are talking about distributed systems in which we have N different databases, then distributed transactions are left as an exercise to the reader (that’s why we have things like Sagas).
Within a single machine, yeah, relational dbs still work like a charm.
The other interesting thing to consider is that machines are a lot more powerful in all dimensions than they were 30 years ago - so many tasks that would have required distributed systems no longer do.
The caveat is that the hardware constraints have fundamentally changed, so database architectures from 30 years ago are effectively incapable of properly using modern hardware environments.
Databases have not been bottlenecked on storage bandwidth in a long time but most databases are designed as if this was still the case. Optimizing for memory bandwidth, the current bottleneck, leads to substantially different architectures than are commonly deployed.
DBMS design has also changed. The bottleneck for a DBMS has moved from spinning disk to CPU and per core memory bandwidth. That's also key to TigerBeetle's design. For example, with static allocation and zero-copy techniques such as zero-deserialization.
If you were to design an OLGP DBMS like Postgres today, it would look radically different. Same is true for OLTP.
I was part of a project that successfully deployed them into production using Oracle, sometimes those license costs are actually worth it.
Agree.
But for anyone tempted by Oracle, do remember that the upfront, agreed licence costs are only a fraction of the true price:
You’ll need someone who actually knows Oracle - either already in place or willing to invest a serious amount of time learning it. Without that, you’re almost certainly better off choosing a simpler database that people are comfortable with.
There’s always a non-zero risk of an Oracle rep trying to “upsell” you. And by upsell, I mean threatening legal action unless you cough up for additional bits a new salesperson has suddenly decided you’re using. (A company I worked with sold Oracle licences and had a long, happy relationship with them - until one day they went after us over some development databases. Someone higher up at Oracle smoothed it over, but the whole experience was unnerving enough.)
Incidental and accidental complexity: I’ve worked with Windows from 3.1 through to Server 2008, with Linux from early Red Hat in 2001 through to the latest production distros, plus a fair share of weird and wonderful applications, everything from 1980s/90s radar acquisition running on 2010 operating systems through a wide range of in house, commercial and open source software and up to modern microservices — and none of it comes close to Oracle’s level of pain.
Edit: Installating Delphi 6 with 14 packages came close, I used 3 days when I had to find every package scattered on disks in shelves and drawers and across ancient web paces + posted as abandonware on source forge but I guess I could learn to do that in a day if I had to do it twice a month. Oracle consistently took me 3 days - if I did everything correct on first try and didn't have to start from scratch.
If it's gonna fit on one machine I'm picking Postgres, Mongo or Redis before TigerBeetle and let's be honest it's not the difference between the 4 that will make it not fit on one machine.
And to be fair, those are all great general purpose solutions. For example, you could also use Postgres or Redis as queue, instead of say Redpanda or Kafka.
But our customers need separation of concerns in their architecture. While they could put the cash in the general purpose filing cabinet, they actually want the separation of concerns between OLGP system of reference (string database, i.e. PG) in the control plane, and OLTP system of record (integer/counting database, i.e. TB) in the data plane.
True. The older databases run very well even on hardware which was far less powerful than what is available now.
Yeah, SQL is not the issue. At least not mostly.
The relational model has shown itself to be exactly the flexible and powerful model that Codd said it was, even in its relatively-debased from in SQL.
In fact the potential of it as a universal and flexible data model that abstracts away storage still remains to be fully unlocked.
Now as for existing SQL databases, yes, many of them were built on foundational assumptions about the nature of memory and secondary storage that no longer hold true.
Many of us in this industry still have our heads stuck in the past world of spinny spinny magnetic disks on platters with heads and cylinders and grinding noises. Real world hardware has moved on. We have a couple orders of magnitude higher IOPS generally available than we did just 10-15 years ago.
So I'm excited about products like CedarDB and others (Umbra before it, etc) that are being built on a foundation from day one of managing with hybrid in-memory/disk.
Throwing out SQL is not really the recipe for "performance" and the lessons of separating the storage system from the data model is still key, since the 1960s.
I am willing to grant that a specialized transaction/ledger system like TigerBeetle might have its place as an optimization strategy in a specific industry/domain, but we should not make the mistake of characterizing this as a general problem with the relational model and data storage broadly.
> Throwing out SQL is not really the recipe for "performance"
We've discovered hacks to work around the limitations of SQL, so you can maintain performance with sufficient hackiness, but you have to give up "purity" to get there. Worse is better applies, I suppose, but if we were starting over today the design would be very different.
Throwing out SQL isn't strictly required, but like Rust isn't strictly required when you already have C, sometimes it is nice to take a step back and look at how you can improve upon things. Unfortunately, NoSQL turning into a story about "document databases" instead of "A better SQL" killed any momentum on that front.
No, I'm arguing in fact the opposite and you'll have to prove the point wrong instead of drive-by insisting on this. The point of Codd's original paper was to separate concerns about data storage from its representation and to propose an elegant representational & transformation & query model based on first order logic and I believe he succeeded.
SQL was the first serious attempt to translate this into a working system, and it's full of warts, but the underlying model it is reaching towards retains its elegance.
But most importantly the principle of not tying the structure of data to its storage model is absolutely key. Not least because we can iteratively and flexibly improve the storage model over time, and not be stuck with decisions that tie us to one particular structure which is the problem that 1960s "network" and "hierarchical" databases (and modern day "key value" or "NoSQL" databases) cause.
The opposite of what? There is no contention with Codd's original paper. I'd suggest that Codd's Alpha paper was lacking in the same way SQL struggles, but that's a bit orthogonal to what you seem to be trying to say. I know its a bit confusing as SQL tries to name many different, independent things with the same name, but when confused it is best to become unconfused, not go off to la-la land.
Notably, you edited your comment after my reply :-)
Strangly off-topic, but if that's where you want to go, what's notable about that? Given the confusion, there should have been no rush to reply before some minor adjustments were made. That is how you ended up here in the first place. Understand first, the consider replying.
All of these apply to FoundationDB as well.
- Slow code writing.
- DST
- No dependencies
- Distributed by default in prod
- Clock fault tolerance with optimistic locking
- Jepsen claimed that FDB has more rigorous testing than they could do.
- New programming language, Flow, for testing.
You probably could solve the same problems with FDB, but TigerBeetle I imagine is more optimized for its use case (I would hope...).
AFAIK - the only reason FDB isn't massively popular is because no one has bothered to write good layers on top. I do know of a few folks writing a SQS, DynamoDB and SQLite layers.
> the only reason FDB isn't massively popular
The only reason is Apple. They liked the product that was released in 2013 so much they bought the whole company, and all other FoundationDB users were abandoned and were forced to drop it.
Who would trust a database that can be yanked out of you at any moment? Though a lot of products have license terms like this only a handful were ever discontinued so abruptly. It's under Apache license now but the trust is not coming back.
I thought they got bought by Apple and what happened to it after?
Apple bought them, took it down from the web, then quietly open-sourced it a few years later. They tried to make it popular, ran a conference for it, but the adoption was too minor for Apple to care afterwards.
It's still maintained by a sizable team at Apple, GH stats show that the activity is much lower now than it was 3 years ago, but there're about 10 people that contribute on a steady regular basis, which is honestly better than 99% of open source projects out there.
> AFAIK - the only reason FDB isn't massively popular is because no one has bothered to write good layers on top. I do know of a few folks writing a SQS, DynamoDB and SQLite layers.
I started writing this comment:
> It seems interesting, but considering what it's for, why aren't the hyperscalers using it?
And while writing it I started searching for FoundationDB and found this:
> https://github.com/<<apple>>/foundationdb
Ah, all right :-p
And a more serious comment, to separate it from the silly one below:
Interesting that they didn't release it with an SQL client, is there no way to make it compatible? Even with extensions to SQL, I imagine it would be great for a lot of use cases.
Edit: ah, it's more of a key-value store.
I've been applying a lot of principles and suggestions from TigerBeetle style lately, mainly in Rust and Go and I can’t recommend it enough.
- single entry point, near-zero deps
- ci locally and tested, one command to runs tests, coverage, lint etc
- property/snapshot/swarm testing, I love writing simulations now and letting the assertions crash
- fast/slow split + everything is deterministic with a seed
- explicit upper bounds + pool of resources. I still dynamically allocate but it makes code simpler to reason about
Thanks to the TB team for the videos and docs they been putting out lately.
Such a pleasure to read this, that TigerStyle is making a difference for you. Appreciate the kind words!
> They keep assertions enabled in production.
Never understood why we turn those off. An assert failing in prod is an assert that I desperately want to know about.
(That "never understood" was rhetorical).
It's historical: turning off assertions made the code run faster.
But nowadays an extra comparison is no biggie, especially if the compiler can hint at which way it's likely to go.
On paper it's because assertions are only checks for developers to make sure they don't misuse an API. They still have values if it's user input, but at that stage, it would be better to change it to business logic like a proper error message returned to the end-user.
I like to write assertions that aren't always easy to check. Like asserting that a list is sorted.
Yeah, there's actually trickiness here. Another curveball is that, with simulation testing, you generally want more assertions to catch more bugs, but slow assertions can _reduce_ testing efficiency by requiring more CPU time per iteration! But then, if you know, at comptime, that the list is going to be short, you might as well do O(N^2) verification!
We captured these consideration in the internal docs here: https://github.com/tigerbeetle/tigerbeetle/blob/0.16.60/src/...
Then you want Design By Contract, dependent types, or formal verification.
However, several factors have to be taken into account regarding performance impact when they get cleverly written, thus in manys cases they can only be fully turned on during debug builds.
Out of curiosity, have you looked at Cue before?
It seems to be used for stuff like this, though I'm yet to really look into it properly.
That sounds easy to check. Can you expand on this, because I don't understand.
Meaning computationally. It would cost a lot of cycles to keep that enabled in production.
It’s only O(n), but if I check that assertion in my binary search function then it might as well have been linear search.
Basically this could be something like this, D based example:
void process_list(List aList)
in(aList.isSorted, "List must be sorted!")
do
{
// do something
}
its easy as in "simple to implement and execute" but not cheap, because it may require scanning large amounts of memory. You have to visit every list entry.
Whats trivial for a very small list, may be a no-go for gigabyte-sized lists.
Ah I see. That's the bit of the conversation I was trying to head off with "rhetorical" :)
Because the very raison d'etre of asserts is to be a compile time/test time check. If you wanted them to run in production, you wouldn't use an assert, you would just run an if. Otherwise what's an assert to you? A mildly convenient sugar syntax alternative to if statements?
That a semantic we've all just sort of inherited from C - I don't know we're required to be bound by it forever. The word itself doesn't imply that assertions would be contextually disabled.
right, it is just syntactic sugar, but if that wasn't helpful then why have it in dev either? I find it more confusing to have asserts be stripped, which creates an implicit dev/prod discrepancy
Pretty much agree, I honestly think that TB team should invest some more effort on evangelizing the double-entry model for non-financial scenarios, like managing stocks, booking a show ticket etc. The 1000x API improvement has been done, now people must know how to use it
It sounds awesome. I'm an a analytics guy using a lot of SQL - not a coder. Though I understand the OP writeup and the purported performance advantages. Can someone explain:
a) what Tigerbeetle data looks like in practice? Assuming it doesn't look like a regular table
b) how you use it, if you can't write sql queries?
c) separately, curious what double-entry would look like for stocks, tickets etc. E.g. I'm a venue and I have 1000 tickets in inventory & deferred revenue.. each time I sell a ticket I turn that inventory to cash and the deferred into a performance liability? Or no entries at all until a ticket is sold? Something else?
As one example, Rene Schallner made a pretty cool ticketing demo called TigerFans: https://tigerfans.io
You could probably do something similar in Postgres.
Similar, yes. But not near the same performance, TB is entirely built around this concept so they became pretty good at it. It's like comparing a lambo and a beetle (no pun intended).
But honestly, if double-entry really become a thing I foresee traditional DBMS agglutinating it just like they did with vector and object databases, getting the long tail of the market.
You could, but TB brings a whole lot of debit/credit primitives out of the box (making it dead simple for e.g. 2PC money movement across third party systems, or transactions of thousands of transactions, to express complex financial contracts)… that are Jepsen-tested, which run multi-AZ, and which would probably take months if not years to replicate on PG. Even then, the internal concurrency control of PG’s storage engine would just get hammered by the contention, and would struggle to approach TB’s replicated performance, even if PG were single node.
Nothing takes “months or years” anymore, if you really wanted this it could be done fairly quick by someone who knows what they’re doing and you don’t layer it in bureaucracy.
It would be nice. I can think of things for which I want to process a financial transaction and something else in one atomic transaction (e.g. selling a ticket - something I am working on right now) and have both content (e.g. product descriptions) and finance in a single store.
>Most teams write code fast. TigerBeetle tries to write code slow. Most teams treat testing as a necessary evil. TigerBeetle is built entirely on Deterministic Simulation Testing (DST). Most teams build their software on top of loads of other software. TigerBeetle has zero dependencies.
I believe that was more like the norm 25+ years ago. Before Google and Facebook brought in the move fast and break things mentality across software industry.
I hope TigerBeetle gets more recognition. Worth reading its Jepsen report as well. https://news.ycombinator.com/item?id=44199592
Let's wait 25 years and we'll see if TigerBeetle becomes the next google or if it's engulfed by a less perfect (but faster) rival.
Move fast and break things was the Facebook motto. I've never really felt it was a Google embraced concept. What Google's values have always lied with investing heavily on testing to ensure folks can be productive by knowing that if they would break something a test would catch it.
That said it's good to make sure you're building for requirements that exist. Engineers have a habit of inventing requirements and causing delays unnecessarily. Building something and placing it in the hands of users so they can give you feedback which you can react to is far more valuable than building products in a bubble.
Looking at the Zig vs. Rust angle — an interesting takeaway from the article is that Zig makes it considerably easier to build statically allocated applications than Rust.
For all of Rust’s talk about “safety”, assuming the above assertion is true, than perhaps Zig is a better choice to augment or replace C in embedded, safety-critical software engineering, where static memory allocation is a core principle.
Because I'm sure other people will ask - no, it does not support SQL.
Joran from TigerBeetle here!
Yes, this is by design. SQL is a great general purpose query language for read-heavy variable-length string workloads, but TigerBeetle optimizes for write-heavy transaction processing workloads (essentially debit/credit with fixed-size integers) and specifically with power law contention, which kills SQL row locks.
I spoke about this specific design decision in depth at Systems Distributed this year:
> which kills SQL row locks.
What's it like compared to MVCC?
Depending on contention and RTT, as a specialized OLTP DBMS, TB can do roughly 1000-2000x more performance than a single node OLGP DBMS (cf. the live demo in the talk above)… but also with strict serializability. You don’t need to sacrifice correctness or real-time resolution, and that’s important. For example, if you need to do real-time balance checks.
Hmmm, I guess it sounds weird to me to be talking "RTT" (round trip time) when the example is "a single node".
I'll watch your talk properly at some point and see if it makes sense to me after that. :)
Node count doesn't matter. You could use an embedded database and encounter the same problem. There is some time T between when you acquire a lock and release it. Depending on the amount of contention in your system, this will have some affect on total throughput (i.e. Amdahl's law).
How familiar are you with MVCC?
https://www.postgresql.org/docs/current/mvcc-intro.html
Asking because needing a lock for changing a row isn't the only approach that can be taken.
Almost all commercial MVCC implementations (including Postgres) use row locks. Very few use OCC even though it's arguably a more natural fit for MVCC. Pessimistic locking is simply more robust over varied workloads.
TB seems really awesome, but is there non-DebitCredit use cases where it can be applied effectively? I like trying to find off-label uses for cool technology
It helps to know what kind of data TigerBeetle handles. The data committed by its transactions is an immutable Transfer of id:128-bit, debit_account_id:128-bit, credit_account_id:128-bit, amount:128-bit, ledger:128-bit, code:16-bit, flags:bitfield, timestamp:64-bit, user_data_128, user_data_64, user_data_32.
Transactions atomically process one or more Transfers, keeping Account balances correct. Accounts are also records, their core fields (debits_posted, credits_posted, etc).
This gives a good idea of what TigerBeetle might be good for and what it might not be. For anything where latency/throughput and accuracy really, really matters, it could be worth the effort to make your problem fit.
I’m somewhat confused by the distributed story for tiger beetle. In the docs here: https://docs.tigerbeetle.com/concepts/performance/#single-th...
It says: “TigerBeetle uses a single core by design and uses a single leader node to process events. Adding more nodes can therefore increase reliability, but not throughput.”
How does this work with multi-region? Does this mean that regardless of where in the world your users might live, they need to make a request to the leader node for a write to occur?
Is distribution purely for redundancy?
The way I understand it yes. Latency will be dictated by going to the leader node and replication to enough followers. The CAP theorem dictates that if you want strong consistency the way they offer you have to talk to a quorum of nodes IIRC N/2 + 1. Paxos and Raft also work that way.
Apparently there is a game about it https://tigerbeetle.com/blog/2023-07-11-we-put-a-distributed...
Anybody know of simple way/tool (local, single node) to have strict Debit/Credit consistency ?
Is there an extension for sqlite or simple command line tool that do that?
I know about the ledger cli tool but it a bit much since it is a full fledge double-entry accounting system.
It's worth noting that the breathless worship of everything about TigerBeetle in this post is likely related to the fact that the author is an investor in TigerBeetle.[0]
Why hasn't anyone mentioned the game they made? It looks fantastic. The concept of letting players try to crash a system to demonstrate its robustness is brilliant.
Thanks!
For anyone who wants to play the walking sim (excuse the pun!): https://sim.tigerbeetle.com
I work at a bank with old old monstrous sql queries. I thought I can make a use of tigerbeetle to simplify the system. But sadly, I just couldn't figure out how to make it work. Transaction requires lot of business logics but I couldn't convert that to make it work with combination of RDBMS + tigerbeetle. I wish there were some realworld example that I can get insight using tigerbeetle.
this has such great marketing zingers, I probably didn't know I needed TigerBeetle for something. I don't know yet, but I promise you, I'll find something to use it for.
I wonder if the 'debits and credits' example is still realistic. Is it not that banks record all individual money transfers and keep them on record for years (often due to national laws that require this). Then the balance on a certain account is a simply a selection/view on the set of all transactions. For efficiency reasons you may want to store the result in some table. If there is a limit on withdrawals on a certain account, you do need to calculate the balance first and then a transaction mechanism is needed to assure that there are not several money transfer that individually do not surpass the limit but together do.
Sounds awesome but can I use it like a regular database for my Laravel/Rails/Django app?
While TigerBeetle way is very interesting I find lack of multicore processing support (or any kind of horizontal scaling) a blocker in terms real world deployment.
It is just too risky as there is a hard limit to scalability and while it might look it is high enough for foreseeable future, what am I supposed to do once I reach this limit? Financial database has to be planned with at least 15-20 years of growth in mind.
sorry if the question seems dumb, but are the clustering options not good for scaling?
TigerBeetle is plenty interesting, though if half of SpacetimeDB's [1] [2] claims are true, it's even more interesting for me. (The title of this post has "_a_ most interesting database", while the article has "_the_", hence this comment)
[1] - https://spacetimedb.com/ [2] - https://www.youtube.com/watch?v=kzDnA_EVhTU&
hope you all like this post as much as I enjoyed writing it!
Just to be a little pedantic for a second, isn't TigerBeetle a database management system (DBMS) whereas a TigerBeetle database would be actual data and metadata managed by TigerBeetle?
> Often the term "database" is also used loosely to refer to any of the DBMS, the database system or an application associated with the database.
We did an episode with Joran on SEDaily which addresses some of the points raised in the discussions
https://softwareengineeringdaily.com/2024/09/12/building-a-f...
What does TigerBeetle stand on the CAP theorem? Sounds like it's AP?
CAP defines availability physically: “requests to a non-failing node receive a response”. TigerBeetle ensures logical availability: as long as clients can reach a majority, requests complete with strict serializability, preserving safety and liveness under partitions.
cf. Durability and the Art of Consensus: https://www.youtube.com/watch?v=tRgvaqpQPwE
> And yet some of the most popular OLTP databases in the world today are still highly dependent on a single node architecture.
Which databases? SQLite is the one I can think of, but it's designed for that use-case. Others start as single node but will replicate to other nodes, either as master-slave or master-master.
I am quite sure Oracle and MS SQL server do just fine in multi-node cluster based architectures, but maybe that isn't their target audience.
MSSQL has a full-blown Death Star deployment option now:
https://learn.microsoft.com/en-us/azure/azure-sql/database/h...
The HN crowd does not talk much about those, but every single place I have worked used them or Postgres.
Same here, I happily ignored all the NoSQL movement, and little by little, they got all relevant features out of it.
Postgres, MySQL
They both have replication, which allows them to be multi-node.
With the bounds capped by a single writer. Unless you can shard the data and create a distributed database with manual sharding.
But yes. Postgres remains an amazing choice, especially with modern hardware, until you also have the money available to tackle said write throughput issue.
I think the point is that sharding won't really help that much since transactions will happen across all or most shards, and then you have certain accounts that will be more active than others.
if your sharding schema is designed properly you will avoid cross shard transactions.
How so? Any account may transact with any other account - regardless which shard it resides in.
OMG as I read this I became hyper-skeptical by the end.
Some points:
In general, in very tangible terms, what are the real benefits for choosing TigerBeetle over another distributed database? What are the target use cases? Most of the article is pontificating about academic details in a way that's putting the cart before the horse. (IE, all these details don't matter when a traditional database with traditional backups is "good enough" and comes with no technology risk.)
Who's using TigerBeetle? For what kind of applications?
https://news.ycombinator.com/item?id=45436926 states, "TigerBeetle doesn't support any auth". Poor security is an unprofessional oversight. For all the ambitious things TFA describes, it's shocking that they overlooked "secure by design."
The above-linked post is the only top-level post in this thread that claims real-world experience with TigerBeetle.
But, here are some quotes from the article that don't pass scrutiny:
> Traditional databases assume that if disks fail, they do so predictably with a nice error message. For example, even SQLite’s docs are clear that:
SQLite is a file on disk. It's not something that would work in the same space as TigerDB. Furthermore, this statement needs proof: Do you mean to say that Oracle, MSSQL, MariaDB, Postgress, ect, can't detect when a file is corrupted?
> All in all, you’re looking at 10-20 SQL queries back and forth, while holding row locks across the network roundtrip time, for each transaction.
These can often be solved with stored procedures. In this case, the problem lies somewhere between programmers implementing suboptimal solutions, and databases being too hard to work with.
> Instead of investing in the technology of 30 years ago – when the most popular relational databases today were built
Don't assume that because technology is old, that it's bad. New databases come with higher, not lower, risk.
> They say databases take a decade to build.
Prove it. No really, prove it.
> Biodigital jazz.
It seems like someone is getting so obsessed with the code that they're forgetting the purpose and use case.
---
IMO:
Figure out the niche where TigerBeetle excels and traditional databases fall flat. Is it because your query model is easier to program with? Is it because, in a particular situation, TigerBeetle is more performant or cheaper? Is there a niche where existing databases corrupt data more than is tolerable.
Once TigerBeetle excels in a niche, expand outward. (This is the basic plan in "Crossing the Chasm." https://en.wikipedia.org/wiki/Crossing_the_Chasm)
Otherwise, this smells like an academic exercise to experiment with different programming techniques, but with no tangible deliverable that has a demonstrated market.
TigerBeetle looks interesting. A whole networked database using statically allocated memory. I usually use that technique for small keyboard-shortcut triggered applications.
One question in case folks who work there see this:
This is the most technically impressive zig project I've seen so far. Do you have a blog post detailing your perspective on zig? i.e what design decisions of ziglang helped you in massive way, what were the warts, and in general any other thoughts too?
> Deterministic Simulation Testing (DST) feels like the most transformational technology that the fewest developers know about. It’s a novel testing technique made popular by the FoundationDB team
It's not novel. That's how hardware (ASIC) testing has been done forever. The novelty is applying it to software.
> TigerBeetle’s VOPR is the single largest DST cluster on the planet. It runs on 1,000 CPU cores
Only if you exclude hardware, otherwise basically every chip design company has a cluster bigger than this.
Leading edge photo mask synthesis is easily in the top 5 most computationally demanding things out there. You need an actual supercomputer to solve these problems on meaningful (competitive) timescales.
The most interesting database is the most boring one.
Consider re-titling it "A very cool & interesting article about a very boring & reliable database"?
Since "interesting" is the very last thing that anyone sane wants in their accounting/financial/critical-stuff database.
I really liked this article. I met the new kid on the block, the DBMS neighborhood. I also didn't know that Zig programming language existed. So many new things. Congratulations to TigerBeetle! I'm going to tell my team about it and try it out on an interesting project.
hiiiiiiiiiii owo pwp qwq
[dead]
hiiiiiiiiii owo qwq pwp