Add interop integration test harness for LND, CLN, and Eclair by febyeji · Pull Request #839 · lightningdevkit/ldk-node

febyeji · 2026-03-24T15:43:01Z

Summary

Closes #766

Add a interop test harness for testing LDK-Node against external Lightning implementations (LND, CLN, Eclair).

Test Coverage

Each implementation runs the same shared scenarios:

Implementation	Image	Version	bitcoind
LND	`lightninglabs/lnd`	`v0.18.5-beta`	29.1
CLN	`elementsproject/lightningd`	`v24.08.2`	30.2
Eclair	`acinq/eclair`	`latest` (currently 0.14.0-SNAPSHOT)	30.2

Combo tests (16 per impl): {payment, idle} × {ldk, ext} × {coop, force} × {ldk, ext} — covers disconnect/reconnect timing, close type, and close initiator
Standalone tests: basic channel cycle, bidirectional payments, concurrent payments, fee-change close, inbound channels, expired invoice
Splice tests: CLN only (Eclair/LND do not yet support splicing)

Design Decisions

Feedback on the design trade-offs would be especially appreciated! Here are the first few decisions I've made in this draft:

bitcoind versions. Eclair requires Bitcoin Core 30+ (bitcoind:30.2); CLN also uses 30.2. LND stays on 29.1 due to intermittent GetInfo gRPC hangs under 30.2.

ExternalNode::wait_for_block_sync(). Eclair's API responds before block indexing completes, so it needs explicit sync-waiting. LND's gRPC hangs if forced to wait. Solution: trait default no-op, Eclair overrides.

Eclair container recreation. Stale channel state causes SIGSEGV in secp256k1-jni (eclair#3275). Each test recreates the container; setup_clients() calls lockunspent to release UTXOs from prior force-closes.

Eclair request timeout + settlement polling. 60s reqwest timeout prevents indefinite hangs. pay_invoice/send_keysend poll /getsentinfo so failures surface immediately.

Issues Found

CLN → LDK keysend fails.

CLN v24.08+ includes payment_secret in keysend HTLCs, which appears to cause LDK to reject the payment during inbound_payment::verify() — likely because no invoice exists for spontaneous payments
test_receive_keysend_payment is skipped for CLN; needs further investigation
send_keysend now polls /getsentinfo for settlement (matching pay_invoice)

ldk-reviews-bot · 2026-03-24T15:43:03Z

🎉 This PR is now ready for review!
Please choose at least one reviewer by assigning them on the right bar.
If no reviewers are assigned within 10 minutes, I'll automatically assign one.
Once the first reviewer has submitted a review, a second will be assigned if required.

ldk-reviews-bot · 2026-03-26T15:55:25Z

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tnull

Thanks, this already looks great on the first pass! I think we should get some minor things out of the way and then move forward with this so we can take advantage of the additional test coverage for our upcoming release already.

Some comments def. can be addressed in a follow-up (e.g. adding BOLT12 test coverage would be great).

docker-compose-cln.yml

.github/workflows/cln-integration.yml

.github/workflows/lnd-integration.yml

Cargo.toml

.github/workflows/eclair-integration.yml

tests/common/mod.rs

tnull · 2026-03-27T09:34:53Z

tests/integration_tests_cln.rs

+/// Upstream LDK fix needed: skip payment_secret verification when a valid
+/// `keysend_preimage` TLV is present.
+#[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+#[ignore = "CLN v24.08+ sends payment_secret in keysend — LDK rejects (upstream fix needed)"]


Have yet to explore this further.

tests/integration_tests_eclair.rs

tests/common/scenarios/mod.rs

…setup Address review feedback on PR lightningdevkit#839: - Replace reqwest dependency with bitreq for Eclair REST client - Replace curl shell calls with bitreq async HTTP requests - Remove per-test docker container recreation (reuse single Eclair instance, unlock UTXOs between tests instead) - Fix chmod -R 755 to u+rwX,go+rX in CLN/LND CI workflows - Add --fail flag to curl readiness check in Eclair CI

- Shared scenarios in tests/common/scenarios/ generic over ExternalNode - Test entry points for LND, CLN, and Eclair - Combo orchestrator (combo.rs) with interop_combo_tests! macro generating 16 tests per implementation (phase × disconnect side × close type × initiator) - Building blocks use no test_ prefix; full scenarios include setup internally - #[ignore] annotations for known interop failures: CLN keysend (payment_secret issue), Eclair keysend (InvalidOnionPayload), CLN/Eclair splice (version requirements)

- Add docker-compose configs for CLN (bitcoind 29.1), LND (bitcoind 29.1), and Eclair (bitcoind 30.2 required by Eclair latest) - Add CI workflows: cln-integration.yml, lnd-integration.yml, eclair-integration.yml - Bump corepc-node feature to 29_0 and update download_bitcoind_electrs.sh

…setup Address review feedback on PR lightningdevkit#839: - Replace reqwest dependency with bitreq for Eclair REST client - Replace curl shell calls with bitreq async HTTP requests - Remove per-test docker container recreation (reuse single Eclair instance, unlock UTXOs between tests instead) - Fix chmod -R 755 to u+rwX,go+rX in CLN/LND CI workflows - Add --fail flag to curl readiness check in Eclair CI

febyeji · 2026-03-28T16:43:09Z

Thanks for the review! Pushed an update addressing the feedback. Will address in follow-ups:

Move docker files to tests/docker/
Add BOLT12 scenario coverage

tnull · 2026-03-30T08:40:42Z

Thanks for the review! Pushed an update addressing the feedback. Will address in follow-ups:

Move docker files to tests/docker/

Add BOLT12 scenario coverage

Thanks! Mind opening tracking issues for these so we don't forget?

tnull

CI is failing right now:

thread 'test_eclair_cooperative_close_after_fee_change' (16127) panicked at tests/common/scenarios/channel.rs:275:55:
called `Result::unwrap()` on an `Err` value: ChannelClosingFailed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    test_eclair_cooperative_close_after_fee_change

…ange test Add a 2s delay after the post-fee-change payment to let LDK's internal monitor updates settle, and use the cooperative_close_by_ldk helper.

febyeji · 2026-03-30T10:13:47Z

@tnull Fixed the test_eclair_cooperative_close_after_fee_change CI failure! The remaining channel_full_cycle_with_vss_store failure is from VSS tests, and I believe a rerun could clear it. Also I opened the tracking issues for the follow-ups.

tnull

Thanks for addressing the feedback! Here are some more comments.

What stands out to me is that:

This PR/the code is very verbose. It would be good to simplify and DRY up where possible to improve maintainability going forward.
We currently ignore a lot of test cases, often for reasons that aren't conclusive to me (or seem outdated). Can we double-check whether we really can't make them work?

tnull · 2026-04-02T11:37:20Z

.github/workflows/cln-integration.yml

-          sudo apt-get install -y socat
+          docker compose -f docker-compose-cln.yml exec bitcoin bitcoin-cli -regtest -rpcuser=user -rpcpassword=pass createwallet miner
+          ADDR=$(docker compose -f docker-compose-cln.yml exec bitcoin bitcoin-cli -regtest -rpcuser=user -rpcpassword=pass -rpcwallet=miner getnewaddress)
+          docker compose -f docker-compose-cln.yml exec bitcoin bitcoin-cli -regtest -rpcuser=user -rpcpassword=pass generatetoaddress 1 "$ADDR"


What are we using the miner wallet for? Note that coinbase funds are only maturing after 100 blocks, so they can only be spent then (which is why you usually see generatetoaddress 101 on setups like this).

tnull · 2026-04-02T11:39:49Z

.github/workflows/rust.yml

        with:
          path: bin/bitcoind-${{ runner.os }}-${{ runner.arch }}
-          key: bitcoind-${{ runner.os }}-${{ runner.arch }}
+          key: bitcoind-29.0-${{ runner.os }}-${{ runner.arch }}


It's a bit odd to only adjust the cache key but not the path here. What's the rationale for this?

tnull · 2026-04-02T11:41:17Z

tests/common/scenarios/channel.rs

+		node.next_event_async(),
+	)
+	.await
+	.unwrap_or_else(|_| panic!("{} timed out waiting for ChannelClosed event", node.node_id()));


nit: Could use a newline

tnull · 2026-04-02T11:56:41Z

tests/common/lnd.rs

+			.try_into()
+			.map_err(|_| self.make_error(format!("capacity_sat overflow: {}", capacity_sat)))?;
+		let push_sat: i64 = push_msat
+			.map(|m| (m / 1000).try_into())


Hmm, seems this could lead to surprises in the future, if we simply forget about any sub-sat amounts and then want to assert them? Maybe this means the entire trait needs to take sat denomination? But I guess we have similar issues with other LND APIs, such as list_channels?

tnull · 2026-04-02T12:01:46Z

tests/common/cln.rs

+		tokio::task::spawn_blocking(move || f(&*client)).await.expect("CLN RPC task panicked")
+	}
+
+	fn make_error(&self, detail: String) -> TestFailure {


nit: It seems this can just be inlined everywhere, rather than duplicating it a bunch of times for each ExternalNode?

tnull · 2026-04-02T12:07:23Z

.github/workflows/rust.yml

        run: |
          source ./scripts/download_bitcoind_electrs.sh
-          mkdir bin
+          mkdir -p bin


Why is this suddenly necessary?

tnull · 2026-04-02T12:17:21Z

tests/integration_tests_cln.rs

+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+#[ignore = "CLN splicing requires --experimental-splicing flag and CLN v25+"]


Yes, why can't we enable this? AFAICT CLN v25+ should be availble? E.g., https://hub.docker.com/layers/elementsproject/lightningd/v25.12.1/images/sha256-fb5c956ff969cf7bb3a9ee46240daa9fdc03a9d7bc35931a2f8a888343fd16d2

tnull · 2026-04-02T12:23:48Z

tests/integration_tests_cln.rs

+}
+
+/// CLN v24.08+ includes a `payment_secret` in outbound keysend HTLCs.
+/// LDK treats any inbound HTLC with `payment_secret` as a BOLT11 payment and


Hmm, I'm not positive this is true? Could you un-ignore this so we can see what the error actually is? Is it related to the checks here?: https://github.com/lightningdevkit/rust-lightning/blob/db42ad6ba053d54f98f360f05afad5fee896ed69/lightning/src/ln/channelmanager.rs#L8479

tnull · 2026-04-02T12:25:23Z

tests/integration_tests_eclair.rs

+
+/// Eclair 0.8.0 rejects LDK keysend with `InvalidOnionPayload(8,0)` — LDK includes
+/// `payment_data` (TLV type 8) in keysend onions, which Eclair considers invalid for
+/// spontaneous payments. Eclair 0.14.0+ may handle this differently.


How do you arrive at this conclusion? Are they aware of this incompatibility? Can you link the releveant issues when including such remarks? Given we have Eclair v0.14-SNAPSHOT, can't we un-ignore this then?

tnull · 2026-04-02T12:26:23Z

tests/integration_tests_eclair.rs

+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+#[ignore = "Eclair 0.8.0 does not support splicing (introduced in v0.10.0+)"]


Why is this ignored, we should have Eclair v0.14-SNAPSHOT available, i.e., can test splicing, no?

…setup Address review feedback on PR lightningdevkit#839: - Replace reqwest dependency with bitreq for Eclair REST client - Replace curl shell calls with bitreq async HTTP requests - Remove per-test docker container recreation (reuse single Eclair instance, unlock UTXOs between tests instead) - Fix chmod -R 755 to u+rwX,go+rX in CLN/LND CI workflows - Add --fail flag to curl readiness check in Eclair CI

Update test infrastructure to Bitcoin Core 29.0: - corepc-node feature: 27_2 → 29_0 - electrsd feature: corepc-node_27_2 → corepc-node_29_0 - download script: bitcoind 27.2 → 29.0 with updated SHA256 hashes Prerequisite for lightningdevkit#839 (interop test harness) which needs bitcoind 29.0 for compatibility with updated Docker images.

Update test infrastructure to Bitcoin Core 29.0: - corepc-node: 0.10.0 → 0.10.1, feature 27_2 → 29_0 - electrsd feature: corepc-node_27_2 → corepc-node_29_0 - download script: bitcoind 27.2 → 29.0 with updated SHA256 hashes Prerequisite for lightningdevkit#839 (interop test harness).

tnull self-requested a review March 24, 2026 15:55

febyeji force-pushed the interop-test-harness branch 4 times, most recently from 8c1e94f to a50c9d0 Compare March 26, 2026 15:49

febyeji force-pushed the interop-test-harness branch from a50c9d0 to 433515c Compare March 26, 2026 17:36

febyeji marked this pull request as ready for review March 26, 2026 17:54

tnull reviewed Mar 27, 2026

View reviewed changes

febyeji force-pushed the interop-test-harness branch from 433515c to 2613a3f Compare March 28, 2026 16:00

febyeji added 3 commits March 29, 2026 01:06

Add ExternalNode trait with LND, CLN, and Eclair implementations

4a1113b

febyeji force-pushed the interop-test-harness branch from 2613a3f to 9c2fbdc Compare March 28, 2026 16:06

febyeji force-pushed the interop-test-harness branch from 9c2fbdc to 7420893 Compare March 28, 2026 16:21

tnull reviewed Mar 30, 2026

View reviewed changes

fix(test): wait for monitor update before cooperative close in fee ch…

ec69789

…ange test Add a 2s delay after the post-fee-change payment to let LDK's internal monitor updates settle, and use the cooperative_close_by_ldk helper.

This was referenced Mar 30, 2026

Move interop test Docker files to tests/docker/ #855

Open

Add BOLT12 scenario coverage to interop test harness #856

Open

tnull reviewed Apr 2, 2026

View reviewed changes

Conversation

febyeji commented Mar 24, 2026

Summary

Test Coverage

Design Decisions

Issues Found

Uh oh!

ldk-reviews-bot commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldk-reviews-bot commented Mar 26, 2026

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

febyeji commented Mar 28, 2026

Uh oh!

tnull commented Mar 30, 2026

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

febyeji commented Mar 30, 2026

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ldk-reviews-bot commented Mar 24, 2026 •

edited

Loading