Bitcoin Magazine

Is your bitcoin site secure?
Show Menu
blkfork

Bitcoin Network Shaken by Blockchain Fork

Yesterday, the Bitcoin network experienced one of the most serious hiccups that we have seen in the past four years. Starting from block 225430, the blockchain literally split into two, with one half of the network adding blocks to one version of the chain, and the other half adding to the other. For the next six hours, there were effectively two Bitcoin networks operating at the same time, each with its own version of the transaction history. The split lasted for 24 blocks or 6 hours, finally resolving itself when one version of the chain conclusively pulled ahead of the other at block 225454, leaving the other chain largely abandoned, with only a small number of miners that are incapable of recognizing what has now become the main chain still mining it, while the bulk of the network quickly returned to normal.

The fork was first noticed at about 23:30 GMT on Monday, March 11, when “thermoman” on the bitcoin-dev IRC channel mentioned that “some client told my client it (the other host) had 225431 blocks, but blockexplorer says that currently the block count is at 225430″. Some other blockchain resources also showed 225430 blocks. Over the course of the next thirty minutes, other users started reporting more strange reports from Bitcoin client logs. Bitcoin developer Peter Wuille (“sipa” on IRC) claimed that he was on block 225435, and then soon 225439, while other sources were still reporting 225431. At 00:00 GMT March 12, sipa posted “I wonder if there’s something that triggered it on the network, a large reorganization or so”. It turned out that a blockchain reorganization, an event that happens when a client discovers a new blockchain longer (and therefore more likely to be valid) than the one it was working with before, and switches to it, was indeed what happened, and over the next few minutes everyone realized what was going on: a blockchain fork.

What had happened was the following. The latest version 0.8 release of bitcoind, by far the most popular implementation of Bitcoin used by miners, switched the database that it used to store blocks and transactions from BerkeleyDB to the more efficient LevelDB as part of an effort to reduce blockchain synchronization time. However, what the developers did not realize at the time was that by doing so they also accidentally introduced a change to the rules of the Bitcoin protocol. In order to make an update to the database, the database process must make a “lock” on the part of the database which stores that particular item of information, a mechanism implemented to prevent two changes from occurring simultaneously and accidentally corrupting the database. In a b-tree, the data structure used by BerkeleyDB to store objects, two locks are required per update. However, BerkeleyDB requires its users to set a limit to the number of locks that can be made at the same time; “If the values are too small,” the FAQ page warns, “requests for locks in an application will fail. If the values are too large, the locking subsystem will consume more resources than is necessary.” In the case of Bitcoin, the limit was 10,000. What happened in block 225430 was that a single block simultaneously affected the status of over 5,000 transactions, requiring more than 10,000 locks on the b-tree to be made at the same time. As a result, the BerkeleyDB failed, and so the older bitcoind 0.7 (and earlier versions) could not read the block. In the case of bitcoind 0.8, LevelDB has no such restrictions, so it could accept such blocks just fine. Because the Bitcoin protocol builds up the transaction history, used primarily to calculate and agree on everyone’s account balances, by creating new blocks representing roughly ten-minute time intervals’ worth of transactions on top of existing valid blocks in a chain (hence, “blockchain”), miners using bitcoind 0.8 started building up a version of the blockchain that included the offending block, while miners using bitcoind 0.7 rejected it and started working on a another blockchain of their own. Ordinary users using BitcoinQt 0.7 or platforms that rely on bitcoind 0.7 as a server saw the 0.7 fork, and everyone else saw the 0.8 fork.

With the fork in progress, the Bitcoin developers had a choice: do they support the 0.8 fork or the 0.7 fork? Ultimately, there could only be one; a monetary system cannot function if there are two different databases of how much money each person has. The 0.8 fork had much more computing power behind it, and was already eight blocks ahead by the time the community could muster any effort toward fixing the problem, and upgrading to 0.8 is something that will have to be done eventually. On the other hand, if the 0.8 fork took over, thousands of users on 0.7 would be forced to upgrade in order to use Bitcoin at all, something which would not happen if the 0.7 fork took over since both versions of bitcoind can read it. The developers quickly settled on 0.7, and the community set to work on the next task: notifying major miners and mining pool operators of what they need to do.

Over the next few hours, nearly every major Bitcoin developer and mining pool operator joined the bitcoin-dev IRC channel. Major mining pools that were using bitcoind 0.8 shut down, downgraded to 0.7, and switched back on. Merchants were also notified; most large businesses, including BitcoinStore, BitPay, SatoshiDice and MtGox, shut down deposits to protect themselves from double spend attacks. BitPay quickly turned themselves back on once their servers were on the 0.7 fork; “safe mode alerted us there’s a problem,” BitPay’s Tony Gallippi writes. “That’s when Steve jumped on IRC to see where the consensus was going, and we were back in business very quickly.” Progress on switching hash power to 0.8 appeared to be slow at first, and at block 225451, the 0.8 chain was 13 blocks longer than 0.7. But that was the furthest that the 0.8 chain would get ahead. By then, the two chains were growing roughly in lockstep, and at about 03:30 the tipping point came. The 0.7 chain quickly caught up to being only 10 blocks behind, then 8 blocks, and at 06:19 both chains converged to the same length at block 225454, leading to nearly all remaining miners abandoning the other.

This incident will go down in history as one of the closest moments that we have come to the underlying Bitcoin protocol actually failing. But it is not the most serious breach ever made. In August 2010, a transaction in block 74638 contained two outputs summing to over 184 billion – just over 2^64 satoshis. The result was an integer overflow bug, the digital equivalent of a mechanical odometer wrapping around to zero after the car drives 999,999 kilometers. The overflow caused the software to think that the transaction contained only a small amount of BTC while in reality the outputs together had thousands of times more than the 21 million that should ever exist. A new version of the Bitcoin software had to be published, the blockchain was forked, and a new, valid, chain overtook the old one at block 74691 – 53 blocks after the original fork. This time, it only took 24 blocks, and it was not even a life-critical threat to the system – if the developers had done nothing, then Bitcoin would have carried on nonetheless, only causing inconvenience to those bitcoind and BitcoinQt users who were on 0.7 and would have had to upgrade. The economic damage was significant, but fairly small; the only monetary losses that have been reported are the $26,000 USD worth of mining block rewards from the 24 mined blocks of 25 BTC that are now forever lost in the now abandoned chain, as well as a $10,000 double spend against OKPay. Aside from the lost mining revenue and this double spend, transactions were not affected and no bitcoins were “lost”; any transaction that was included in the now abandoneded chain was included in the new chain as well, so any bitcoins that were spent during the fork are now at their proper destinations.

In a way, this was the best possible time for such a thing to happen. The Bitcoin price was on a steady uptrend, and so the 24% drop in price that occurred at the time of the incident was quickly reversed, and as of the time of this writing Bitcoin stands at $44-$46, down from $48 the day earlier but up from $36 one week before. Public media attention on Bitcoin is very much positive, and rather than attacking Bitcoin as they would have in 2011 many journalists actually praised the Bitcoin development team on their rapid response. Ars Technica’s Timothy B Lee wrote a neutral piece on the event, writing that “the incident will be an important test of the cryptocurrency’s decentralized governance structure”, and an article at ecurrency.ec on the subject was entitled “Bitcoin software bug has been rapidly resolved”.

However, the incident opens up serious questions about the nature of the Bitcoin protocol, and puts into the spotlight some uncomfortable facts about Bitcoin’s notion of “decentralization”. Most security protocols, including encryption algorithms, hash algorithms and full-scale protocols, have dozens of implementations in many different programming languages, and the protocol specification is determined by a clear standard against which any individual implementation can be checked for compliance. In the case of Bitcoin, however, things are different. Although there is technically a standard on the Bitcoin wiki pages, it has at times been poorly updates, and the reality is that the bitcoind implementation is the standard, and nearly all miners on the Bitcoin network are using some version of it. There are a few alternative implementations, the most complete one being Amir Taaki’s libbitcoin, with Mike Hearn’s BitcoinJ (written in Java) close behind, but so far they have gained very little traction in use with mining, and, what’s more, there is a small portion of the Bitcoin development community which is actively against the idea of using multiple codebases.

Fortunately, most Bitcoin developers do not support this viewpoint, although many have come out in favor of keeping a healthy level of prudence. Mike Hearn wrote the following on the Bitcointalk forums in June 2011:

Gavin wrote to me only days after the BitCoinJ release to tell me how happy he was to see an alternative implementation. Satoshi expressed very similar sentiments. Nobody is against alternative implementations.
What some people, especially Satoshi, have said is that there’s an unusual amount of risk involved with reimplementing the full system and using that reimplementation to mine. Bitcoin is very complex and if you aren’t skilled and very thorough you are likely to diverge from its behavior in small, hard to detect ways. This can fork the chain and split the economy. It’s one of the few things that could instantly kill Bitcoin beyond legal harassment of its users.

Lead Bitcoin developer Gavin Andresen replied to another poster in the same thread: “Really? I’ve been encouraging alternative implementations, who is the power-hungry core developer?”, and in November 2012 he wrote in a Bitcoin Foundation update that “part of the solution is to encourage alternative implementations that make different trust/convenience tradeoffs than the reference implementation. There has been a lot of behind-the-scenes work on cross-implementation testing (the “testnet3″ blockchain contains hundreds of transaction validation test cases, for example), and new features are being added to the protocol to support alternative implementations”

But alternative implementations are not just useful for supporting different trust/convenience tradeoffs. They are also crucial in making Bitcoin’s claim of decentralization a reality. If there had instead been five distinct Bitcoin implementation in use at the time of the fork, what would have happened is that one of the five would have recognized the wrong blockchain and forked off, leading to a loss of revenue for a small number of miners and requiring the users of clients using that implementation to upgrade. The aberrant implementation’s fork of the blockchain would end up much weaker than the others right from the start, so the risk of double spend attacks would be minimal. One can argue that there will be a greater number of forking incidents with more implementations, but each one will be smaller in effect, and testing all implementations together on the testnet before release would reduce the number of bugs that slip into production software to about the same frequency as we see today.

The other aspect of Bitcoin’s decentralization that this incident calls into question is that of mining pools. The reason why the controlled switch to the 0.7 fork was even possible was that over 70% of the Bitcoin network’s hash power was controlled by a small number of mining pools and ASIC miners, and so the miners could all be individually contacted and convinced to immediately downgrade. Another article on the fork reads [Russian]: “the real problem is not even in the code supporting the Bitcoin network; bugs are everywhere. Rather, it’s the matter of who controls it. This event clearly showed that even such a well thought-out system is controlled by the will of a very small number of people – particularly, the operators of mining pools. Over 70% of new blocks right now are being found on pools, and not on individual solo miners. The underlying idea of the system was that the benevolent majority can stop a small number of attackers, but in the present time it is simply not working. The winner in a possible takeover will be the one with greater computing power, and no one else.” Bitcoin is clearly not at all the direct democracy that many of its early adherents imagined, and, some worry, if a centralized core of the Bitcoin community is powerful enough to successfully undertake these emergency measures to set right the Bitcoin blockchain, what else is it powerful enough to do? Force double spends to reverse million-dollar thefts? Block or even redirect transactions known to originate from Silk Road? Perhaps even modify Bitcoin’s sacred 21 million currency supply limit?

However, a strong argument can be made that such fears are very unlikely to materialize. The reason why has nothing to do with the specific identities of the Bitcoin mining pool operators or the cohesiveness of the Bitcoin mining community; rather, it’s the fact that Bitcoin mining is still in fact quite decentralized; it simply is decentralized in a different way. Taking a political analogy, the closest equivalent would be a liquid democracy: a hybrid of direct and representative democracy where people can either vote for individual bills by themselves or assign politicians – with the proviso that if they do not like what a given politician is doing they can switch to assigning their voting power to someone else at any time. Back in the world of Bitcoin, although much of the Bitcoin network’s hash power is concentrated with mining pool operators in practice, every individual miner can switch from one pool to another almost instantly, so if a coalition of mining pool operators decides to start violating the Bitcoin protocol miners can simply switch to any pool that remains honest, instantly depriving the miscreants of their power. Although no mining pool has attempted to actively subvert the Bitcoin protocol so far, this kind of “voting” has been done before; in 2011, there were several incidents where the mining pool Deepbit pushed above 50% of the total network hash power, and in each case there was a mass exodus of miners toward other pools to balance things out. Although the nominal power may rest with the mining pool operators, the feedback of the community is always only one step away.

Altogether, the incident was handled very well, and all parts of the Bitcoin community should congratulate themselves for their speedy resolution of the problem and their unconditional cooperation. The Bitcoin community is not always in perfect harmony; Bitcoin gambling site SatoshiDice and a number of Bitcoin developers, notably Luke Dashjr, are usually at odds over concerns that SatoshiDice’s large transaction count is bloating the Bitcoin blockchain, but yesterday differences were laid aside as the community worked together to solve the problem. We also learned a lot, and merchants are likely to be much more prepared for such incidents in the future, perhaps implementing techniques like automatic fork detection to handle forks and avoid double spends without immediate manual intervention. Before today, many people knew that some test for the Bitcoin network would come, whether at the 1 MB block size limit or else where, but just how the community would handle such a thing was an unknown. Now, the test has come and gone, and how the Bitcoin community handled the test is known to everyone: we passed with flying colors.

BTC: 1FxkfJQLJTXpW6QmxGT6oF43ZH959ns8Cq

LTC: LaBhvWiAP7msku6w8QSQ5G7omVWMF3uxJC

By

Vitalik Buterin is a co-founder of Bitcoin Magazine who has been involved in the Bitcoin community since 2011, and has contributed to Bitcoin both as a writer and the developer of a fork of bitcoinjs-lib, pybitcointools and multisig.info, as well as one of the developers behind Egora. Now, Vitalik's primary job is as the main developer of Ethereum, a project which intends to create a next-generation smart contract and decentralized application platform that allows people to create any kind of decentralized application on top of a blockchain that can be imagined.

Get Top Stories Weekly

We respect your email privacy

  • http://twitter.com/ErikVoorhees Erik Voorhees

    “Although the nominal power may rest with the mining pool operators, the feedback of the community is always only one step away.” Great point, this. And excellent article, as per usual, Vitalik.

  • http://https://self-evident.org/ Nemo

    So basically, a small group of Mandarins were able to revert perfectly legitimate transactions because some buggy implementation of the protocol could not handle them. And this is a shining example of Bitcoin’s “decentralized governance structure”.

    Have I got that right?

    • Maciej Mikosik

      Correct me if I’m wrong but I don’t think any transactions were reversed because both blockchains in the end contained same transactions. (unless explicit doublespend attack occured)

      • http://twitter.com/bitcoinmoney Bitcoin Money

        It (double spend attack) did occur. An OKPay customer had already been credited with confirmed funds, and an hour later broadcast a conflicting transaction which got mined, depriving OKPay of their $10K USD worth of bitcoins they already had six confirmations for.

        OKPay may not believe Bitcoin passed this incident with flying colors.

        • Maciej Mikosik

          Sorry, my fault. I meant “except double spend attack”.
          I was addressing “a small group of Mandarins were able to revert perfectly legitimate transactions” part, which I still think is incorrect.
          Am I correct now?

        • Vitalik Buterin

          If the devs had let 0.8 win, there may have been even more double spending attacks. The algorithm would be simple: make a transaction to yourself dependent on something that happened in the 0.8 chain already, throw it in the 0.8 chain, and then make another tx to a merchant still on the 0.7 chain. There would have been merchants still on 0.7 for weeks, so unscrupulous Bitcoin users would have had plenty of time to plan such operations, and even without double spends the business disruption may well have cost more than $36k USD.

          Bitcoin also promises 100% uptime, with this decision it pretty much delivered (since barring double spends any transaction sent during this time would eventually make it into the main chain), while if we had gone with the 0.8 chain there would have been problems.

          Yes, the best solution would have been to foresee the problem in the firstplace, but given that it happened I think the team did close to the best they could.

          • http://https://self-evident.org/ Nemo

            The problem is not that the centralized governors of Bitcoin made the wrong choice. The problem is that so few people have the power to make the choice at all… Kind of the opposite of what Bitcoin advertises.

          • abrayoungham

            Ah, I see your issue. You mistakenly think that including every possible brain on the decision will make it more democratic, more “just”, more “fair”. But it won’t be a “better” decision. it’s a GOOD thing that we don’t have to ask permission from every single human being before we decide to take a course of action. Your philosophy can’t work in the real world. Be careful, you may one day get what you are asking for, and it wouldn’t do you any good. There is a difference between “centralized power” and “people who know what to do and have the integrity to do”. In the first, centralized power exists solely to benefit themselves. In the second, individuals are at work making choices only they can (and should) make.

          • Edmund Broadley

            No, this would not have happened because the 0.8 chain was longer than the 0.7 chain, and therefore (I hope!) bitcoind should enter into a safe-mode to not assume that any funds received are safe. It should (but I don’t think currently does) also enter this safe mode if it sees another chain catching up with the current valid chain.

          • http://twitter.com/bitcoinmoney Bitcoin Money

            When miners realized that their coins would never mature, they would jump to v0.8 clients. There still would be a few blocks created on the pre-v0.8 side but probably not the six required to confirm. Plus v0.7 clients had the automatic warning whereas v0.8 clients were already way past more than six confirmations after the fork.

            I’m not saying the “friendly 51% attack” wasn’t the right decision, but I definitely wouldn’t say that in this instance the Bitcoin community “passed with flying colors”.

      • http://https://self-evident.org/ Nemo

        What about the “offending transaction” that tickled the 0.7 bug in the first place? Was that replayed, or was it reversed?

        Which reminds me… What will happen if someone now creates a flurry of similar “offending” transactions?

        • Maciej Mikosik

          There was no “offending transaction”. There was offending block 225430.

          • http://https://self-evident.org/ Nemo

            *yawn* whatever. You could correct my spelling, too, and be just as useful. But would it answer my questions?

            Oh, well. Lucky thing this was not a deliberate attack, huh? Else there would have been a lot more than $10,000 of double spending. Only a matter of time?

          • Maciej Mikosik

            You asked whether offending transaction was reversed. I explained that there was no offending transaction so there was nothing to reverse. I do hope that answered your question!

            The possibility of forks and double-spend attack as situation proved are still a possibility but no one is arguing with you about that.

            I was only commenting facts that you misrepresented.

          • http://https://self-evident.org/ Nemo

            OK, for the moment, let’s ignore that this could have been a double-spend attack (and almost certainly will be someday).

            I said “transaction” instead of “block” because I have not yet polluted my brain with all of Bitcoin’s parochial terminology. The questions are the same if you just search&replace terms.

            The _block_ was valid and it was reverted. In this case, it was mined by the same people who actually have the power to revert valid blocks; that is, Bitcoin’s centralized governors.

            It could just as easily have been mined by a third party. Indeed, it could just as easily have been a “transaction”. In either of those scenarios, the reversion would have made it more obvious (but no more true) that Bitcoin has a de facto centralized governing structure.

            The issue is not whether Bitcoin’s central governors made a good decision here. The issue is that they exist at all.

          • Maciej Mikosik

            Again. Nothing was reverted. The majority of the network reached a consensus to mine at 0.7 fork and rest of the network agreed. Those who do not agree are free to stay on 0.8 fork (which contain “reverted” block you’re talking about) or any other.
            How does unanimous decision made by network means centralization?

            You may argue that most of network hashrate is in too few hands (51+% hashing power, too few humans), which many would claim is true. Is this your claim?

          • http://twitter.com/damagedaniel Daniel Damage

            YAY, someone that actually understands how the blockchain works!

            @Nemo, you can’t just /s/transaction/block. They are fundamentally different. A block consists of many transactions. The network keeps track of unprocessed/unconfirmed transactions. If a transaction remains unconfirmed, it is queued for the next block, or the next, or the next. Hence, worst impossible case, if a block disappears out of thin air, transactions will just get moved/confirmed in the next block.

            IIRC.

          • http://https://self-evident.org/ Nemo

            I was using “transaction” in the database theory sense (i.e. atomicity/consistency/isolation/durability), not the “I sent you some money” sense. For Bitcoin, the “database” is the block chain itself and a “database transaction” is a new block

            The integrity of that database — including the atomicity and durability of its transactions — is the foundation of the integrity of the currency.

            If you simply replace “transaction” with “block” in my original comments, you will understand what I was saying… Namely, that this event calls the integrity of that database into question.

          • http://https://self-evident.org/ Nemo

            > Again. Nothing was reverted.

            You seem intelligent, so I assume you do not really believe this.

            Consider the point of view of someone running the 0.8 software. You know, the non-buggy software that _correctly_ implements the documented protocol. They saw a new, perfectly valid block creating some Bitcoins. Then six hours later, they saw that block disappear due to the deliberate actions of a stunningly small group of people.

            Try a little thought experiment. Suppose instead of creating 25 Bitcoins, the block had represented a transfer of 2500 Bitcoins. And suppose instead of six hours, it had been one year. By your definition, “nothing was reverted” and this would also just be “the majority of the network reaching a consensus”. By my definition, this would be “a small cabal reverting the block for reasons of their own”. I suspect my definition far more closely comports with how the vast majority of people would view such an event.

            But the actual event this week differs only in degree, not in nature. That it was possible at all calls several of Bitcoin’s fundamental claims into question.

            Future events will make this extremely clear, I bet.

            OK, we have reached the point where I am not learning anything; and, I suspect, neither are you. Feel free to have the last word.

          • Maciej Mikosik

            >Try a little thought experiment. Suppose instead of creating 25 Bitcoins, the block had represented a transfer of 2500 Bitcoins.

            Ok, let’s try it.
            First, block does not represent transfer, transaction does. So let’s assume that “offending” block contains transaction of 2500 bitcoins. The trick is that your transaction is included in both blockchain forks! If one blockchain gets abandoned your transaction will not get lost. The article clearly says that:
            “any transaction that was included in the now abandoneded chain was included in the new chain as well, so any bitcoins that were spent during the fork are now at their proper destinations”
            Again. Nothing was reverted!

            I can only guess you think that miners voted on which transactions to include and which to abandon. They didn’t. They voted on which fork (containing same transactions*) to join. Your transaction is included whether you were running 0.7 or 0.8.

            *-excluding miner rewards and double-spend attacks.

          • http://https://self-evident.org/ Nemo

            One last time, just in case you are being serious.

            I thought it was obvious from context, but my thought experiment should include: “…and the reverted transaction is not present on the new block chain”.

            Again, _a block was reverted by a small cabal_. (Yes, “reverted” is the right word.) In this case, it was just a block creating a few bitcoins, but it _could_ have been something more interesting.

            You continue to make the same mistake throughout this discussion: Confusing the relatively innocuous outcome of this event (“the cabal did the right thing”) with the problem it demonstrates (“the cabal exists at all”). They imposed a new history, by pure force of computation, that _reverted_ a legitimate block addition from six hours prior. In this case, it was just a coin creation block, and they were kind enough to replay all of the outstanding transactions (save the double-spend). But there was nothing — absolutely nothing — forcing them to do that.

            What this event has demonstrated is that the integrity of the currency depends not on the protocol, but on the benevolence of the cabal. This should not inspire confidence.

            And no, they do not exist by “consensus”. Anyone with sufficient computing power can become the new cabal… Which is arguably the most fundamental flaw in Bitcoin. This is not a new observation, but this week’s event does throw it into sharper focus.

            OK now you really may have the last word.

          • Maciej Mikosik

            I rest my case. I think anyone following discussion has seen enough to make up his mind. Preferably someone else should have last word.

  • http://twitter.com/bitcoinmoney Bitcoin Money

    Not everyone mining v0.8 abandoned mining that, even after converging there still were more blocks on the v0.8 fork mined.

    And those few that did suffer double spends (and a few whose “already confirmed on v0.8 side” transactions that to this day have not been included in the pre-v0.8 side) did not “make a sacrifice”, they instead were the sacrifice made.

  • Wooxer Pt

    “This incident will go down in history as one of the closest moments that we have come to the underlying Bitcoin protocol actually failing.”
    Well, the Bitcoin protocol was doing exactly what is has been programmed to do, have the most of the hashrate producing and following the longest chain, there was no need to tell a lot of people to go back to the 0.7, people that used it would eventually upgrade (but it was better this way in order to mitigate other kinds of problems I reckon.

    Also, this shows that the network, although not “super-decentralized” is still very decently decentralized because miners and pools had to be contacted in order to react to this problem and they’ve chosen to react by downgrading to 0.7 but everyone of the people contacted could have chosen not to do so in case they’ve suspected, after analyzing the problem, that something was fishy, like a true decentralized network should be (people democratically addressing the issues of it in order to protect it).

    Very good article