“We live one click away from wreckdom”, once said a wise Bitcoin patriarch. And indeed all it took was a single click to ruin my LND node, with the very real possibility of some funds being gone for good. I eventually took it back, but it was not easy, so I’ll summarize my experience here for brothers that might do the wrong click one day too.
Initial Setup
I have been running a BTCPay server for some times now. It runs on a VPS with a storage pruned to 60GB to keep the price reasonable. BTCPay server runs LND with some specific parameters, including noseedbackup=1 which is at the root of the problem here.
It means that the LND that is running inside BTCPay doesn’t generate a seed when it creates its internal wallet. The rationale behind this decision is that the seed alone is actually not enough to restore all your funds in case your LND directory got totally wiped out (if I got it right the seed won’t restore your channels back, so every funds in channels would be lost). It would have induced a false sense of security to the user, and explain properly what it does and what it does not would be complicated UX wise, so the BTCPay team decided to remove it altogether.
This trade-off makes sense for BTCPay, but it has another less anticipated downside: without seed, a LND internal wallet doesn’t have a “birthday” either. It means LND has no way to tell when the wallet was created. And that takes us to the next chapter of our story.
Wreckdom
On the last days of december, I casually connected to my BTCPay server to check if everything was alright, and figured out I haven’t updated in a while. “Sure, what could go wrong?” I thought, and clicked on the “update” button.
A few hours later, I connected to my VPS for some reasons and casually did a lncli getinfo. I immediately noticed that all my channels were inactive. After a quick research, I found out this issue on the LND repo.
There Roasbeef was making pretty clear what was the issue: on updating from 0.5 to 0.5.1, LND’s wallet needs to rescan the chain. It is supposed to rescan from birthday. Remember, because it has no seed, BTCPay’s LND have no birthday too. So LND will try to rescan from Genesis by default. Oops, my BTCPay node is pruned, Genesis block hasn’t been there for a while. And it just got stuck here.
I must admit I had neglected a bit my BTCPay’s LND, and I wasn’t sure how much money was lying aroung in limbo at the moment. I later figured out that I withdrawed most of the content of the wallet a couple of weeks before that, which made me chill out. But I still had to rescue the channels and some funds I didn’t withdraw.
In the Github issue, it was suggested to copy the .lnd directory and run it against a non-pruned full node to let him do the rescan.
The Rescue
I turned down LND inside my BTCPay, and copied the directory to another machine of mine that run a non-pruned node. I launched LND and indeed it started rescan the whole blockchain from Genesis. My machine is a bit weak, and it took about 24 hours to complete the rescan.
Once complete, I copied the content of the directory back in my BTCPay server and restarted LND. First thing I noticed was that my channels were not stuck in inactive state anymore, but most of them instantly force closed on remote side. I’m not sure why, I guess the erratic behaviour of my node triggered a force close on some of my peers’ side, or maybe it had been inactive for too long.
After a while when all the pending channels got closed, I noticed that while I could see most of the funds again in the wallet, one of them hasn’t showed up. That was strange, because I had a look in an explorer and the transaction was completed. But my LND failed to see it for some reasons.
After having copied my directory back to my full node and done a second complete rescan without success, I thought that I was rekt, even if hopefully only of a relatively small amount. I was thinking that I got lucky, and was prepared to just let it go as some kind of sacrifice to the Lightning’s Gods in case there was no way to take it back.
dropwtxmgr wizardry
That’s when wpaulino suggested using one of btcwallet internal tools called dropwtxmgr before the rescan.
I don’t use btcd, so I wasn’t quite sure how to do it, and got carried away by other stuff until those last days. Here is what I did (I suppose you have already installed Go, if not you can follow instructions on LND install page) :
git clone https://github.com/btcsuite/btcwallet $GOPATH/src/github.com/btcsuite/btcwalletcd $GOPATH/src/github.com/btcsuite/btcwalletGO111MODULE=on go install ./dropwtxmgrcd $GOPATH/bindropwtxmgr --db=/path/to/data/chain/bitcoin/mainnet/wallet.db
(Those instructions are different than the dropwtxmgr README file, that are slightly out of date, see this)
After that, I started up lnd with my BTCPay lnd directory once again, and waited 24 hours for it to do a whole rescan. And… it worked! All funds are SAFU.
Conclusion
- I’m still not very sure what was the issue, but it seems it was about the “look ahead distance of the rescan”, see below :
- We’re all playing with fire, especially running LND on a pruned node inside another application like BTCPay was asking for trouble, so don’t be like me, always carefully read the release note before updating either BTCPay or LND…
- It is okay to copy .lnd directory on another machine, even if I’m not sure if I could have saved the channels, but at least you should be able to take your funds back
- I really need to take the time to understand how I should backup a LND node, because this whole story made me realize how little I know about that. I guess you should make a back-up of your .lnd directory every time you open a new channel. Need to do further research on that.
Thanks to the LND and BTCPay developpers, that are doing a great job and always took the time to help me when I ran into trouble, despite working on pretty badass futuristic software!