uhhm/varde
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Migrate to iroh 1.0 and iroh-blobs 0.103

Browse Source 
The 0.35 pin's rationale ("the post-0.35 rewrite is not yet production
quality") expired when iroh hit 1.0 in June 2026: the rewrite line
(0.103) is now the production line and the only one receiving fixes.

The rewrite replaced wrappable store traits with an irpc-based API, so
the seams moved:

- shaped.rs: the 440-line ShapedStore trait wrapper becomes a provider
  event handler. Trust gating (untrusted peers see only openly-served
  hashes) now intercepts requests before any bytes move; upload rate
  limiting rides the provider's Throttle hook; upload progress events
  come from per-request update streams. The TokenBucket is unchanged.
- store.rs: FsStore's API handle replaces the store traits, and the
  LocalPool machinery for non-Send futures is gone. GC is now the
  store's built-in periodic mark-and-sweep, fed by a pin-roots snapshot
  via the protect callback (new config knob gc_interval_secs, default
  300); the on-demand Gc request answers Unimplemented, and the gc
  conformance test polls the sweep instead.
- transfer.rs: BlobsProtocol + Router replace handle_connection, the
  new multi-provider Downloader replaces the old queue, and mdns
  discovery moved to the iroh-mdns-address-lookup crate (it left iroh
  core in 1.0). Ticket-embedded provider addresses feed a MemoryLookup
  address book. Endpoint presets: Minimal (LAN-only default) or N0
  (wan_upload), preserving the old relay posture.
- Node* became Endpoint* throughout; announcement signatures use iroh's
  own Signature type (ed25519-dalek dep dropped); iroh-io dropped.

Known regression: max_download_bytes_per_sec is currently not enforced
— download shaping rode the old store's batch writer and 0.103's
downloader has no equivalent seam yet.

Announcement wire format note: EndpointAddr serializes differently than
NodeAddr, so pre- and post-migration daemons won't parse each other's
LAN announcements. Announcements are live-only, nothing stored breaks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Bendik Lynghaug
2026-08-16 14:03:15 +02:00
co-authored by Claude Fable 5
parent a2d150225a
commit ad69f7d884
14 changed files with 1807 additions and 2248 deletions
Download Patch File Download Diff File Expand all files Collapse all files
varde-daemon/src/store.rs
+164 -196
View File
@@ -1,24 +1,27 @@
//! Blob store operations on top of the iroh-blobs persistent fs store.
use std::collections::BTreeSet;
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use std::sync::{Arc, OnceLock};
use std::time::Duration;
use anyhow::Context;
use iroh_blobs::api::blobs::{AddPathOptions, BlobStatus, ImportMode};
use iroh_blobs::format::collection::Collection;
use iroh_blobs::hashseq::HashSeq;
use iroh_blobs::store::{
EntryStatus, ImportMode, Map, MapEntry, MapMut, ReadableStore, Store as _,
};
use iroh_blobs::util::local_pool::{LocalPool, LocalPoolHandle};
use iroh_blobs::util::progress::IgnoreProgressSender;
use iroh_blobs::store::fs::options::{Options, PathOptions};
use iroh_blobs::store::fs::FsStore;
use iroh_blobs::store::{GcConfig, ProtectCb, ProtectOutcome};
use iroh_blobs::{BlobFormat, Hash, HashAndFormat};
use iroh_io::{AsyncSliceReader, AsyncSliceReaderExt};
use tokio::io::AsyncWriteExt;
use tracing::{debug, info};
use tracing::{debug, info, warn};
use varde_proto::MaterializeMode;
use crate::meta::StoredFormat;
/// Snapshot of the gc roots (the pinned hashes with their formats),
/// consulted by the store's gc before every run.
pub type GcRootsFn = Arc<dyn Fn() -> Vec<HashAndFormat> + Send + Sync>;
/// Operation errors that map onto protocol error codes.
#[derive(Debug, thiserror::Error)]
pub enum OpError {
@@ -34,64 +37,102 @@ pub enum OpError {
Unimplemented(String),
}
/// The daemon's blob store: an iroh-blobs fs store rooted at
/// `<store_dir>/blobs`.
///
/// Reads from store entries yield non-`Send` futures, so those operations
/// run on a dedicated [`LocalPool`] — the same pattern iroh-blobs itself
/// uses for its provider and GC tasks.
#[derive(Debug, Clone)]
pub struct BlobStore {
store: iroh_blobs::store::fs::Store,
data_dir: PathBuf,
pool: LocalPoolHandle,
// Owns the pool threads; dropped when the last clone goes away.
_pool: std::sync::Arc<LocalPool>,
/// Wrap an iroh-blobs API error as an internal error.
fn internal(e: impl std::error::Error + Send + Sync + 'static) -> OpError {
OpError::Internal(anyhow::Error::new(e))
}
/// Reading a blob entry for a streaming copy happens in chunks this size.
const COPY_CHUNK: u64 = 1024 * 1024;
/// The daemon's blob store: an iroh-blobs fs store rooted at
/// `<store_dir>/blobs`.
#[derive(Debug, Clone)]
pub struct BlobStore {
store: FsStore,
data_dir: PathBuf,
}
impl BlobStore {
pub async fn open(store_dir: &Path) -> anyhow::Result<BlobStore> {
/// Open the store. `gc_roots` supplies the pinned hashes the built-in
/// garbage collector must keep (temp tags and stored tags are
/// protected by the store itself); their hashseq children are
/// expanded here before every run.
pub async fn open(
store_dir: &Path,
gc_interval: Duration,
gc_roots: GcRootsFn,
) -> anyhow::Result<BlobStore> {
let blobs_dir = store_dir.join("blobs");
let store = iroh_blobs::store::fs::Store::load(&blobs_dir)
// The protect callback needs the store API to expand hashseq
// roots, but it is installed before the store exists — hence the
// cell, filled right after load.
let store_cell: Arc<OnceLock<iroh_blobs::api::Store>> = Arc::new(OnceLock::new());
let protect: ProtectCb = {
let cell = store_cell.clone();
Arc::new(move |live: &mut HashSet<Hash>| {
let cell = cell.clone();
let roots = gc_roots();
// The callback future must be Sync; the store API's
// futures are not, so the expansion runs on its own task.
Box::pin(async move {
let expanded = tokio::spawn(async move {
let mut hashes: Vec<Hash> = Vec::new();
for HashAndFormat { hash, format } in roots {
hashes.push(hash);
if format.is_raw() {
continue;
}
let Some(store) = cell.get() else { continue };
match children_of(store, hash).await {
Ok(children) => hashes.extend(children),
Err(e) => {
warn!(hash = %hash.to_hex(), error = %e, "gc protect: expanding hashseq");
return None;
}
}
}
Some(hashes)
})
.await;
match expanded {
Ok(Some(hashes)) => {
live.extend(hashes);
ProtectOutcome::Continue
}
// Confused about liveness: skip this run rather
// than sweep blobs that should be protected.
Ok(None) => ProtectOutcome::Abort,
Err(e) => {
warn!(error = %e, "gc protect task failed");
ProtectOutcome::Abort
}
}
})
})
};
let options = Options {
path: PathOptions::new(&blobs_dir),
inline: Default::default(),
batch: Default::default(),
gc: Some(GcConfig {
interval: gc_interval,
add_protected: Some(protect),
}),
};
let store = FsStore::load_with_opts(blobs_dir.join("blobs.db"), options)
.await
.with_context(|| format!("opening blob store at {}", blobs_dir.display()))?;
let pool = LocalPool::default();
let _ = store_cell.set((*store).clone());
Ok(BlobStore {
store,
data_dir: blobs_dir.join("data"),
pool: pool.handle().clone(),
_pool: std::sync::Arc::new(pool),
})
}
/// Run a non-`Send` store operation on the local pool.
async fn on_pool<T, F, Fut>(&self, f: F) -> Result<T, OpError>
where
F: FnOnce(BlobStore) -> Fut + Send + 'static,
Fut: std::future::Future<Output = Result<T, OpError>> + 'static,
T: Send + 'static,
{
let this = self.clone();
self.pool
.spawn(move || f(this))
.await
.map_err(|e| OpError::Internal(anyhow::anyhow!("local pool: {e}")))?
}
/// Access to the underlying iroh-blobs store (used by the transfer
/// layer).
pub fn inner(&self) -> &iroh_blobs::store::fs::Store {
/// Access to the underlying iroh-blobs store API (used by the
/// transfer layer). `FsStore` derefs to the API handle.
pub fn api(&self) -> &iroh_blobs::api::Store {
&self.store
}
/// The local pool the transfer layer shares for non-`Send` blob work.
pub fn pool_handle(&self) -> LocalPoolHandle {
self.pool.clone()
}
/// Import a file or directory. Returns the root hash, total imported
/// bytes, and the resulting format (`Raw` for files, `HashSeq` for
/// directories).
@@ -110,7 +151,7 @@ impl BlobStore {
if meta.is_file() {
let (tag, size) = self.import_one(path.to_owned()).await?;
info!(hash = %tag.hash().to_hex(), size, "imported file");
return Ok((*tag.hash(), size, StoredFormat::Raw));
return Ok((tag.hash(), size, StoredFormat::Raw));
}
if !meta.is_dir() {
return Err(OpError::InvalidArgument(format!(
@@ -145,27 +186,40 @@ impl BlobStore {
}
let collection: Collection = children
.iter()
.map(|(name, tag)| (name.clone(), *tag.hash()))
.map(|(name, tag)| (name.clone(), tag.hash()))
.collect();
let root_tag = collection
.store(&self.store)
.await
.context("storing collection")?;
let root = *root_tag.hash();
let root = root_tag.hash();
info!(hash = %root.to_hex(), files = children.len(), total, "imported directory");
Ok((root, total, StoredFormat::HashSeq))
}
async fn import_one(&self, path: PathBuf) -> Result<(iroh_blobs::TempTag, u64), OpError> {
let (tag, size) = self
async fn import_one(
&self,
path: PathBuf,
) -> Result<(iroh_blobs::api::TempTag, u64), OpError> {
let tag = self
.store
.import_file(
.add_path_with_opts(AddPathOptions {
path,
ImportMode::Copy,
BlobFormat::Raw,
IgnoreProgressSender::default(),
)
.await?;
format: BlobFormat::Raw,
mode: ImportMode::Copy,
})
.temp_tag()
.await
.map_err(internal)?;
let size = match self.store.blobs().status(tag.hash()).await.map_err(internal)? {
BlobStatus::Complete { size } => size,
_ => {
return Err(OpError::Internal(anyhow::anyhow!(
"freshly imported blob {} is not complete",
tag.hash().to_hex()
)))
}
};
Ok((tag, size))
}
@@ -184,25 +238,11 @@ impl BlobStore {
dest.display()
)));
}
let dest = dest.to_owned();
self.on_pool(
move |this| async move { this.materialize_local(hash, format, &dest, mode).await },
)
.await
}
async fn materialize_local(
&self,
hash: Hash,
format: StoredFormat,
dest: &Path,
mode: MaterializeMode,
) -> Result<(u64, bool), OpError> {
let allow_reflink = matches!(mode, MaterializeMode::ReflinkOrCopy);
match format {
StoredFormat::Raw => self.export_blob(hash, dest, allow_reflink).await,
StoredFormat::HashSeq => {
let collection = Collection::load_db(&self.store, &hash)
let collection = Collection::load(hash, self.api())
.await
.map_err(|e| OpError::NotFound(format!("loading collection: {e}")))?;
let mut total = 0u64;
@@ -222,25 +262,30 @@ impl BlobStore {
}
/// Export one blob. Tries `FICLONE` via reflink first (zero-copy on
/// btrfs/XFS when store and dest share a filesystem), falls back to a
/// streaming copy. Never hardlinks: store files must stay immutable.
/// btrfs/XFS when store and dest share a filesystem), falls back to
/// the store's export (a copy). Never hardlinks: store files must
/// stay immutable.
async fn export_blob(
&self,
hash: Hash,
dest: &Path,
allow_reflink: bool,
) -> Result<(u64, bool), OpError> {
let entry =
self.store.get(&hash).await?.ok_or_else(|| {
OpError::NotFound(format!("{} is not in the store", hash.to_hex()))
})?;
if !entry.is_complete() {
return Err(OpError::NotFound(format!(
"{} is only partially present",
hash.to_hex()
)));
}
let size = entry.size().value();
let size = match self.store.blobs().status(hash).await.map_err(internal)? {
BlobStatus::Complete { size } => size,
BlobStatus::Partial { .. } => {
return Err(OpError::NotFound(format!(
"{} is only partially present",
hash.to_hex()
)))
}
BlobStatus::NotFound => {
return Err(OpError::NotFound(format!(
"{} is not in the store",
hash.to_hex()
)))
}
};
if let Some(parent) = dest.parent() {
tokio::fs::create_dir_all(parent).await?;
}
@@ -272,129 +317,52 @@ impl BlobStore {
}
}
let mut reader = entry.data_reader();
let mut file = tokio::fs::File::create(dest).await?;
let mut offset = 0u64;
while offset < size {
let len = (size - offset).min(COPY_CHUNK) as usize;
let chunk = reader.read_at(offset, len).await?;
if chunk.is_empty() {
return Err(OpError::Io(std::io::Error::new(
std::io::ErrorKind::UnexpectedEof,
format!("blob {} truncated at {offset}", hash.to_hex()),
)));
}
file.write_all(&chunk).await?;
offset += chunk.len() as u64;
}
file.flush().await?;
Ok((size, false))
let bytes = self
.store
.blobs()
.export(hash, dest)
.await
.map_err(internal)?;
Ok((bytes, false))
}
/// Presence of `hash`: (have_bytes, total_bytes, complete).
pub async fn presence(&self, hash: Hash) -> Result<(u64, Option<u64>, bool), OpError> {
match self.store.entry_status(&hash).await? {
EntryStatus::NotFound => Ok((0, None, false)),
EntryStatus::Partial => {
let total = self.store.get(&hash).await?.map(|e| e.size().value());
// Valid-range accounting for partials arrives with the
// transfer milestone; absence of data is the safe report.
Ok((0, total, false))
}
EntryStatus::Complete => {
let entry = self
.store
.get(&hash)
.await?
.ok_or_else(|| OpError::NotFound(hash.to_hex().to_string()))?;
let size = entry.size().value();
Ok((size, Some(size), true))
}
match self.store.blobs().status(hash).await.map_err(internal)? {
BlobStatus::NotFound => Ok((0, None, false)),
// Valid-range accounting for partials arrives with the
// transfer milestone; absence of data is the safe report.
BlobStatus::Partial { size } => Ok((0, size, false)),
BlobStatus::Complete { size } => Ok((size, Some(size), true)),
}
}
/// The child hashes of a complete HashSeq root (empty for absent or
/// partial roots).
pub async fn hashseq_children(&self, root: Hash) -> Result<Vec<Hash>, OpError> {
self.on_pool(move |this| async move {
let Some(entry) = this.store.get(&root).await? else {
return Ok(Vec::new());
};
if !entry.is_complete() {
return Ok(Vec::new());
}
let mut reader = entry.data_reader();
let bytes = reader.read_to_end().await?;
let seq = HashSeq::try_from(bytes)
.map_err(|e| OpError::Internal(anyhow::anyhow!("invalid hashseq: {e}")))?;
Ok(seq.iter().collect())
})
.await
}
/// Drop every blob not reachable from `roots` (mark and sweep).
/// In-flight imports are protected by their temp tags; tags stored in
/// the blob database are honored too.
pub async fn gc(&self, roots: Vec<HashAndFormat>) -> Result<u64, OpError> {
self.on_pool(move |this| async move { this.gc_local(roots).await })
.await
}
async fn gc_local(&self, roots: Vec<HashAndFormat>) -> Result<u64, OpError> {
let mut live: BTreeSet<Hash> = BTreeSet::new();
let mut all_roots = roots;
all_roots.extend(self.store.temp_tags());
for item in self.store.tags(None, None).await.context(TAGS_CONTEXT)? {
let (_name, haf) = item.context(TAGS_CONTEXT)?;
all_roots.push(haf);
}
for HashAndFormat { hash, format } in all_roots {
if !live.insert(hash) || format.is_raw() {
continue;
}
// HashSeq root: its children are live too. A partial root
// can't be expanded; its bytes are still protected.
let Some(entry) = self.store.get(&hash).await? else {
continue;
};
if !entry.is_complete() {
continue;
}
let mut reader = entry.data_reader();
let bytes = reader.read_to_end().await?;
let seq = HashSeq::try_from(bytes)
.map_err(|e| OpError::Internal(anyhow::anyhow!("invalid hashseq: {e}")))?;
live.extend(seq.iter());
}
let mut doomed = Vec::new();
for hash in self
.store
.blobs()
.await?
.chain(self.store.partial_blobs().await?)
{
let hash = hash?;
if !live.contains(&hash) {
doomed.push(hash);
}
}
let removed = doomed.len() as u64;
if !doomed.is_empty() {
self.store.delete(doomed).await?;
}
info!(removed, live = live.len(), "gc done");
Ok(removed)
children_of(self.api(), root).await
}
/// Flush and shut down the store actor.
pub async fn shutdown(&self) {
self.store.shutdown().await;
if let Err(e) = self.store.shutdown().await {
debug!(error = %e, "store shutdown");
}
}
}
const TAGS_CONTEXT: &str = "listing tags";
/// The child hashes of a complete HashSeq root (empty for absent or
/// partial roots).
async fn children_of(store: &iroh_blobs::api::Store, root: Hash) -> Result<Vec<Hash>, OpError> {
match store.blobs().status(root).await.map_err(internal)? {
BlobStatus::Complete { .. } => {}
_ => return Ok(Vec::new()),
}
let bytes = store.blobs().get_bytes(root).await.map_err(internal)?;
let seq = HashSeq::try_from(bytes)
.map_err(|e| OpError::Internal(anyhow::anyhow!("invalid hashseq: {e}")))?;
Ok(seq.iter().collect())
}
/// Recursively collect regular files under `dir` as (relative-name, path),
/// sorted for deterministic collection hashes. Symlinks are followed for