Move transaction broadcasting and fee estimation to dedicated modules#205
Merged
tnull merged 5 commits intolightningdevkit:mainfrom Dec 4, 2023
Merged
Move transaction broadcasting and fee estimation to dedicated modules#205tnull merged 5 commits intolightningdevkit:mainfrom
tnull merged 5 commits intolightningdevkit:mainfrom
Conversation
7042e38 to
f8fe15d
Compare
wpaulino
reviewed
Nov 27, 2023
3335b34 to
efea678
Compare
efea678 to
69710b7
Compare
Collaborator
Author
|
Rebased on main to resolve minor conflicts. |
wpaulino
reviewed
Dec 4, 2023
src/tx_broadcaster.rs
Outdated
|
|
||
| pub(crate) async fn process_next_package(&self) { | ||
| let mut receiver = self.queue_receiver.lock().await; | ||
| if let Some(next_package) = receiver.recv().await { |
There was a problem hiding this comment.
Rather than just processing one package, are we able to keep reading until nothing's left?
Collaborator
Author
There was a problem hiding this comment.
Yeah, I originally avoided that since broadcasting might also trigger the Esplora rate limits. So I thought it might make sense to let the server breathe for ~1sec between packages. However, probably doesn't make a big difference and we already have the 500ms retry delay now. So made it a while let Some(..) now.
We introduce a separate `TransactionBroadcaster` which will regularly process a message queue of transactions to be broadcast. This allows us to a) decouple broadcasting from the BDK wallet and b) add transactions to the queue from a blocking context, without the need to schlep around a separate runtime or mess with it.
Due to the rate-limiting applied by many Esplora servers we often
receive HTTP 429 ('too many requests') errors during syncing. Here, we
simply give broadcasting transactions a second chance after a slight
delay of 500ms to ease the pain of immediate failures. Generally,
rebroadcasting will then be initiated by the `OuputSweeper`.
We also decouple fee estimation from the BDK on-chain wallet and BDK's corresponding `EsploraBlockchain`. Instead we use the esplora client directly in a dedicated `OnchainFeeEstimator` object. For one, this change is nice as it allows us to move more things out of the `Wallet` thread and corresponding locks. Moreover, it makes things more modular in general, which makes future upgrades and testing easier.
69710b7 to
ea31d2d
Compare
Collaborator
Author
|
Added new tranaction-bytes-logging commit and included minor requested changes: > git diff-tree -U2 69710b7 ea31d2d
diff --git a/src/lib.rs b/src/lib.rs
index 8adfe55..ae0fda1 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -686,5 +686,5 @@ impl<K: KVStore + Sync + Send + 'static> Node<K> {
}
_ = interval.tick() => {
- tx_bcaster.process_next_package().await;
+ tx_bcaster.process_queue().await;
}
}
diff --git a/src/logger.rs b/src/logger.rs
index cab62c6..b0504a3 100644
--- a/src/logger.rs
+++ b/src/logger.rs
@@ -1,4 +1,4 @@
pub(crate) use lightning::util::logger::Logger;
-pub(crate) use lightning::{log_debug, log_error, log_info, log_trace};
+pub(crate) use lightning::{log_bytes, log_debug, log_error, log_info, log_trace};
use lightning::util::logger::{Level, Record};
diff --git a/src/tx_broadcaster.rs b/src/tx_broadcaster.rs
index ae98373..4e56cba 100644
--- a/src/tx_broadcaster.rs
+++ b/src/tx_broadcaster.rs
@@ -1,5 +1,6 @@
-use crate::logger::{log_debug, log_error, log_trace, Logger};
+use crate::logger::{log_bytes, log_debug, log_error, log_trace, Logger};
use lightning::chain::chaininterface::BroadcasterInterface;
+use lightning::util::ser::Writeable;
use esplora_client::AsyncClient as EsploraClient;
@@ -34,7 +35,7 @@ where
}
- pub(crate) async fn process_next_package(&self) {
+ pub(crate) async fn process_queue(&self) {
let mut receiver = self.queue_receiver.lock().await;
- if let Some(next_package) = receiver.recv().await {
+ while let Some(next_package) = receiver.recv().await {
for tx in &next_package {
match self.esplora_client.broadcast(tx).await {
@@ -67,4 +68,9 @@ where
e
);
+ log_trace!(
+ self.logger,
+ "Failed broadcast transaction bytes: {}",
+ log_bytes!(tx.encode())
+ );
}
}
@@ -77,4 +83,9 @@ where
e
);
+ log_trace!(
+ self.logger,
+ "Failed broadcast transaction bytes: {}",
+ log_bytes!(tx.encode())
+ );
}
}, |
wpaulino
approved these changes
Dec 4, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #22.
Previously, transaction broadcasting and fee estimation was the concern of our BDK-based onchain wallet and its
EsploraBlockchain. However, this for one meant these functions would also be subject to BDK's locking requirements, pontentially even requiring additional runtime threads. Given that theBlockchaintrait is going away in BDK 1.0 anyways, the little convenience it gave us wasn't worth the additional complexity. Moreover, the prior design was not particularly modular, which made creating mock objects for tests hard, e.g., in #152.Here, we therefore move transaction broadcasting and fee estimation to two dedicated modules which take a cloned
EsploraClientinstance. This also allows us to make the calls in an async fashion while providing the blocking interface to LDK objects in a more reasonable manner that doesn't lean on tokio'sblock_in_place.While general rebroadcasting of transactions will be initiated via the
OutputSweeperto come in #152, we now also immediately retry rebroadcasts after a small delay if they fail due to connection (or HTTP) errors.