uhhm/varde
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

Migrate to iroh 1.0 and iroh-blobs 0.103

Browse Source 
The 0.35 pin's rationale ("the post-0.35 rewrite is not yet production
quality") expired when iroh hit 1.0 in June 2026: the rewrite line
(0.103) is now the production line and the only one receiving fixes.

The rewrite replaced wrappable store traits with an irpc-based API, so
the seams moved:

- shaped.rs: the 440-line ShapedStore trait wrapper becomes a provider
  event handler. Trust gating (untrusted peers see only openly-served
  hashes) now intercepts requests before any bytes move; upload rate
  limiting rides the provider's Throttle hook; upload progress events
  come from per-request update streams. The TokenBucket is unchanged.
- store.rs: FsStore's API handle replaces the store traits, and the
  LocalPool machinery for non-Send futures is gone. GC is now the
  store's built-in periodic mark-and-sweep, fed by a pin-roots snapshot
  via the protect callback (new config knob gc_interval_secs, default
  300); the on-demand Gc request answers Unimplemented, and the gc
  conformance test polls the sweep instead.
- transfer.rs: BlobsProtocol + Router replace handle_connection, the
  new multi-provider Downloader replaces the old queue, and mdns
  discovery moved to the iroh-mdns-address-lookup crate (it left iroh
  core in 1.0). Ticket-embedded provider addresses feed a MemoryLookup
  address book. Endpoint presets: Minimal (LAN-only default) or N0
  (wan_upload), preserving the old relay posture.
- Node* became Endpoint* throughout; announcement signatures use iroh's
  own Signature type (ed25519-dalek dep dropped); iroh-io dropped.

Known regression: max_download_bytes_per_sec is currently not enforced
— download shaping rode the old store's batch writer and 0.103's
downloader has no equivalent seam yet.

Announcement wire format note: EndpointAddr serializes differently than
NodeAddr, so pre- and post-migration daemons won't parse each other's
LAN announcements. Announcements are live-only, nothing stored breaks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Bendik Lynghaug
2026-08-16 14:03:15 +02:00
co-authored by Claude Fable 5
parent a2d150225a
commit ad69f7d884
14 changed files with 1807 additions and 2248 deletions
Download Patch File Download Diff File Expand all files Collapse all files
varde-daemon/src/daemon.rs
+40 -26
View File
@@ -51,8 +51,30 @@ impl Daemon {
pub async fn open(config: Config) -> anyhow::Result<Arc<Daemon>> {
std::fs::create_dir_all(&config.store_dir)
.with_context(|| format!("creating store directory {}", config.store_dir.display()))?;
let store = BlobStore::open(&config.store_dir).await?;
let meta = Arc::new(Meta::load(&config.store_dir)?);
// The store's built-in gc keeps whatever this snapshot reports
// (plus tags and in-flight temp tags) on every run.
let gc_roots: crate::store::GcRootsFn = {
let meta = meta.clone();
Arc::new(move || {
meta.pins()
.into_iter()
.filter_map(|(hex, _)| {
let hash = Hash::from_str(&hex).ok()?;
Some(HashAndFormat {
hash,
format: blob_format(meta.format_of(&hex)),
})
})
.collect()
})
};
let store = BlobStore::open(
&config.store_dir,
Duration::from_secs(config.gc_interval_secs.max(1)),
gc_roots,
)
.await?;
// Capacity bounds memory if a subscriber stalls; laggards get a
// Lagged error, not unbounded buffering.
let (events, _) = broadcast::channel(1024);
@@ -68,7 +90,6 @@ impl Daemon {
&store,
meta.clone(),
open_filter,
store.pool_handle(),
events.clone(),
metered,
)
@@ -194,7 +215,7 @@ impl Daemon {
/// Track presence for a discovery sighting of `node_id`. Content
/// sync happens via the announcement path, not here.
fn on_peer_seen(&self, node_id: iroh::NodeId) {
fn on_peer_seen(&self, node_id: iroh::EndpointId) {
let id = node_id.to_string();
let rejoined = {
let mut peers = self.lan_peers.lock().expect("lan peers lock");
@@ -247,10 +268,10 @@ impl Daemon {
if let Ok((_, _, true)) = self.store.presence(root).await {
return; // already complete
}
let providers: Vec<iroh::NodeAddr> = ann
let providers: Vec<iroh::EndpointAddr> = ann
.providers
.iter()
.filter(|p| self.meta.is_trusted(&p.node_id.to_string()))
.filter(|p| self.meta.is_trusted(&p.id.to_string()))
.cloned()
.collect();
if providers.is_empty() {
@@ -269,7 +290,7 @@ impl Daemon {
/// Queue fetches for every incomplete pin from a trusted LAN peer.
/// The downloader dedupes by content, so repeated sightings are
/// cheap; discovery resolves the peer's addresses.
async fn sync_pins_from(&self, node_id: iroh::NodeId) {
async fn sync_pins_from(&self, node_id: iroh::EndpointId) {
for (hex, _record) in self.meta.pins() {
let Ok(hash) = Hash::from_str(&hex) else {
continue;
@@ -283,7 +304,7 @@ impl Daemon {
hash,
format: blob_format(self.meta.format_of(&hex)),
},
vec![iroh::NodeAddr::new(node_id)],
vec![iroh::EndpointAddr::from(node_id)],
);
}
Err(e) => warn!(hash = %hex, error = %e, "checking pin presence"),
@@ -334,11 +355,11 @@ impl Daemon {
// Fetch if absent: try every trusted peer currently
// present on the LAN.
if let Ok((_, _, false)) = self.store.presence(parsed).await {
let present: Vec<iroh::NodeAddr> = self
let present: Vec<iroh::EndpointAddr> = self
.present_trusted_peers()
.into_iter()
.filter(|p| p.connected)
.filter_map(|p| p.node_id.parse().ok().map(iroh::NodeAddr::new))
.filter_map(|p| p.node_id.parse::<iroh::EndpointId>().ok().map(iroh::EndpointAddr::from))
.collect();
if !present.is_empty() {
self.transfer.spawn_fetch(
@@ -414,9 +435,14 @@ impl Daemon {
Ok(ResponseData::Pins { pins })
}
Request::Gc {} => {
let roots = self.pin_roots()?;
let blobs_removed = self.store.gc(roots).await?;
Ok(ResponseData::GcDone { blobs_removed })
// iroh-blobs 0.103 removed on-demand gc: the store runs
// its own periodic mark-and-sweep, protecting pins via
// the roots snapshot handed to BlobStore::open.
Err(OpError::Unimplemented(
"gc now runs continuously inside the blob store; \
on-demand gc has no API in iroh-blobs 0.103"
.to_string(),
))
}
Request::TicketExport { hash } => {
let parsed = parse_hash(&hash)?;
@@ -458,7 +484,7 @@ impl Daemon {
})
}
Request::PeerTrust { node_id } => {
let parsed: iroh::NodeId = node_id
let parsed: iroh::EndpointId = node_id
.parse()
.map_err(|e| OpError::InvalidArgument(format!("invalid node id: {e}")))?;
self.meta.trust_peer(&parsed.to_string())?;
@@ -476,7 +502,7 @@ impl Daemon {
Ok(ResponseData::Done {})
}
Request::PeerUntrust { node_id } => {
let parsed: iroh::NodeId = node_id
let parsed: iroh::EndpointId = node_id
.parse()
.map_err(|e| OpError::InvalidArgument(format!("invalid node id: {e}")))?;
if !self.meta.untrust_peer(&parsed.to_string())? {
@@ -491,18 +517,6 @@ impl Daemon {
}
}
/// The GC roots: every pinned hash with its recorded format.
fn pin_roots(&self) -> Result<Vec<HashAndFormat>, OpError> {
let mut roots = Vec::new();
for (hash, _record) in self.meta.pins() {
let parsed = parse_hash(&hash)?;
roots.push(HashAndFormat {
hash: parsed,
format: blob_format(self.meta.format_of(&hash)),
});
}
Ok(roots)
}
}
fn blob_format(format: StoredFormat) -> BlobFormat {