What happens when your Lightning Network routing node is fed with garbage transactions that never resolve? In short, it causes a lot of grief for routing nodes. What was once a smooth, global payment system can be locked up with trivial effort from a savvy script writer.
Working in a small team of routing nodes, we successfully ran a test of the attack with real funds and demonstrated the “griefing” attack described by Joost Jager. The attack is called a grief attack since it is not a theft of funds, but it causes a victim’s Lightning funds to be frozen: a major upset. What we found is that griefing is a serious threat to large “wumbo” channels expecting to earn a yield on their bitcoin, only to have their funds frozen for a period of time.
[paywall price=”100″]
This is mostly a grief attack: no loss of funds, but the victim may be forced to pay for an expensive channel force close. This is a known vulnerability on mainnet Lightning and it needs to be understood and prioritized, especially at this early market stage of Bitcoin’s Lightning Network.
Thanks to Clark Burkhardt and Phillip Sheppard for their willingness to participate in this test and to Jager for his tireless work to bring attention and priority to this vulnerability. Jager played the role of the attacker for our demonstration, while Burkhardt and Sheppard joined me as connected victim routing nodes.
How The Attack Works
The attacker saturates one (or several) channel(s) with Hashed Time Locked Contracts (HTLCs) that don’t resolve as a finalized payment. These are a special breed of HTLCs known as HODL invoices. Only 483 of these unresolved HTLCs are required to overwhelm a channel per direction. Once those HTLCs are in the channel, any transactions using that same channel direction are impossible, including a transaction to cooperatively close that channel.
In theory, an attacker could contact the victim (perhaps via a keysend message or in an “onion blob”) and demand a ransom be paid to halt the attack. Once the ransom is paid, the attacker could remove the unresolved payments, ending the attack. The attack can be sustained indefinitely, halting all routing and payment activity in that channel. This freezes the funds in the Lightning channel.
Both directions of payments can be stalled in a channel by using 483 HTLCs in each direction, both inbound and outbound.
Why Would An Attacker Do Something Like This?
The first motive that comes to mind is to demand a ransom. This attack causes pain for the victim and paying a ransom may be attractive to a victim, even without assurance that the attack would stop. Contacting the victim might be risky for an attacker, but a ransom payment might not be the only reason someone would do this.
A secondary incentive for launching a griefing attack would be to disrupt routing competition. Jamming a competitor’s route could create more demand for a route owned by an attacker.
As a benchmark, consider that Lightning Labs’ Loop node has an ongoing demand for liquidity for which it will sometimes pay a 2,500 parts per million of the payment (ppm) (0.25 percent) fee rate. In my experience, they would normally exhaust 16 million sats worth of liquidity in about two weeks (5.2 percent annual percentage rate), but that is with competition present.
If an attacker could disable any competing route with lower fee rates, Loop may be willing to pay a higher fee rate (since the supply of liquidity is now reduced). Let’s say Loop would pay 3,000 ppm (0.3 percent), as well as use that liquidity more quickly since no other channels are functioning. Loop might use that liquidity in half the time, say one week. The attacker would more than double their usual yield to 15.6 percent APR in this example. The only cost to the attacker is the cost of running a script on an existing channel and the psychological cost of doing something immoral/damaging to the Lightning Network. With a single attacker channel, a malicious actor could jam about nine channels (see Jager’s tweets about this).
What Would The Victim Of This Attack Experience?
The victim of this attack wouldn’t really know that this attack was happening unless they had some special alerts set for pending HTLCs. For Thunderhub users (a highly recommended tool), the home screen will show a chart of pending HTLCs as well as a warning stating that channels can only hold 483 pending HTLCs.
In practice, my node quickly became unreliable and experienced several app crashes, including Thunderhub, which was the only app to notify me of the problem. Then, thanks to my “Balance of Satoshis” Telegram bot, I got a channel closing notification. The channel under attack force-closed itself! That was not supposed to be part of the experiment. (For more technical information on the involuntary force close, see below for additional force-close data.)
What Can The Victim Do To Stop A Griefing Attack?
Once an attack starts, a victim essentially can’t do anything to stop it. The only alternatives available to halt an ongoing attack would be to force-close the channel being attacked, which means that the terrorists win.
To add insult to injury, force-closing the channel will push the unresolved payments to the on-chain transaction data, triggering secondary on-chain transactions for the initiator of the force close. At 50 sats/vbyte and 483 on-chain transactions, that’s easily a 1 million sat price tag to force close a single channel under attack (a $368 channel close fee at today’s prices). The multiple on-chain transactions only occur if the output is above the minimum payment “dust” limit. (See this example on testnet.)
How To Prevent A Griefing Attack
Jager has been working on a proof-of-concept program to help isolate and fight attackers. He’s calling his program “Circuitbreaker.” The Circuitbreaker works at a network level, which unfortunately means that everyone has to participate for it to be effective.
Beyond that, this issue needs prioritization and attention from dedicated engineers/developers to find better solutions. There have also been some good discussions on modifying the protocol in the Bitcoin Optech newsletter (issue #122 or #126).
This attack can be executed today. It is a miracle that it hasn’t already been used maliciously. It’s a reflection of the incentives for those using Lightning today so that it can become an open, universal payment network. Please share this post as you see fit to encourage and inspire more work to fix this problem before it causes real harm.
Additional Technical Information About The Involuntary Force-Close
Here are the logs from my node running LND 0.11 at the moment that the above mentioned involuntary force-close occured:
2020-11-26 21:24:47.374 [ERR] HSWC: ChannelLink(657759:561:0): failing link: ChannelPoint (c37bec006b18df172698a84739ca47128935e0a8666fecd1a843e49b01db207c:0): received error from peer: chan_id=7c20db019be443a8d1ec6f66a8e035891247ca3947a8982617df186b00ec7bc3, err=rejected commitment: commit_height=455, invalid_commit_sig=3044022076fd65191eb6305b723fa6012be378413b6326e2786c38db58b4c02e1f3999d202207605ca31de8b4c5b1d9cd20dc1581dfa2383e0b4e06c8ad4f718ab5c434d8cf5, commit_tx=02000000017c20db019be443a8d1ec6f66a8e035891247ca3947a8982617df186b00ec7bc300000000008a792e8002210d0000000000002200201031cf10a1efef261edd3d0a1a6a953b27bc25bd7150bb2b07afdc69805e02157213000000000000160014de650929042bef58b71783ae1a44834a902a8f2d542ca720, sig_hash=4e0fb804c74376020e4c44a60969b9206eb0aaa9a89b76017d60f23ad5cf63e5 with error: remote errorThe logs show an “invalid_commit_sig” which is a known issue in LND. Supposedly, this can happen upon reconnecting and isn’t a direct result of the channel jamming. The volume of pending HTLCs unfortunately makes it more likely to happen. Jager helped explain the process as channel jamming –> endless payment loop (bug) –> node down –> reconnect –> invalid commit sig (bug) –> channel force-close.
The “endless” loop bug is a known bug that occurs when the HTLC limit is reached and an additional HTLC is sent. Instead of ending in a payment failure, LND will continue to attempt the payment in a loop. To help with this bug, see LND issue #4656.
[/paywall]
What’s up, just wanted to mention, I enjoyed this post.
It was helpful. Keep on posting!
I really like what you guys tend to be up too. Such clever work and exposure!
Keep up the awesome works guys I’ve you guys to my blogroll.
I believe a node operator could mitigate the issue by reducing the max number of pending htlc. This would essentially make the attack less profitable for the attacker, however would lower the threshold for a channel DoS.
Personally, I’d rather a channel be DoS than channels forced close at my expense or funds being potentially stolen (a different attack disclosed recently)
You can lower the max pending htlc with (lnd):
default-remote-max-htlcs=30