Render an Apple configuration profile from the server's PACC document
docs / deploy (push) Successful in 3s
Test / test (push) Successful in 2m9s
Publish release / publish (push) Successful in 1m52s

iOS is the one client none of the mail autoconfiguration reaches: Apple
Mail reads neither Mozilla autoconfig nor Autodiscover for IMAP, so
adding an account means the device probing for ports - and a probe that
meets a filtered port waits out a TCP timeout and reports the account as
unverifiable, which the person reads as a wrong password.

A profile states the hosts and the ports instead, and carries mail,
calendars and contacts in one install. It invents none of them: site.yaml
names a PACC document - the standard defined to be fetched from the mail
domain itself - and every host, port and collection path comes from
there, so the server is described in one place and the profile follows it.

The document holds no address, username or password. iOS asks for those
at install, which means one profile serves everyone with an account and
it can be fetched by anyone: it says no more than PACC already says in
public. Identifiers are derived from the site and the payload rather than
drawn at random, because iOS replaces a profile whose ids it knows and
stacks a duplicate when they differ.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015GeH3y8fwTaTudaD88P57o
This commit is contained in:
Bendik Lynghaug
2026-10-07 11:18:46 +02:00
co-authored by Claude Opus 5
parent 43a2f3a069
commit 44c7ee6870
46 changed files with 6937 additions and 1519 deletions
+4
View File
@@ -4,6 +4,10 @@ One line per change, grouped by the release that shipped it. Newest
first. A release is a Cargo.toml bump plus a `v<version>` tag; the
content repos pin the tag they run (see the end of this file).
## 0.6.22 (2026-10-07)
- `/profile.mobileconfig` renders an Apple configuration profile from the mail server's own PACC document (`profile.pacc` in `site.yaml`), so iOS can be set up in one install instead of probing for ports it cannot reach. Mail, CalDAV and CardDAV in one profile; every host, port and path from PACC, never written down twice; no address, username or password in the document, so one profile serves everyone and iOS asks at install; identifiers derived from the site, so reinstalling replaces the accounts rather than duplicating them (docs: content-reference, "A configuration profile").
## 0.6.21 (2026-10-07)
- A content repo's `.well-known/` is served at the site root (`/.well-known/security.txt`, `/.well-known/assetlinks.json`, `/.well-known/matrix/server`), and a miss there is a 404 instead of the app shell answering 200 with an HTML page. RFC 8615 reserves the prefix and a question's id cannot begin with a dot, so nothing a repo ships can shadow a page. An extensionless file under it is typed as JSON, by the convention those files follow; `.txt`, `.xml`, `.webmanifest` and `.mobileconfig` are typed too, the last so an Apple configuration profile offers to install rather than download.
Generated
+1 -1
View File
@@ -3357,7 +3357,7 @@ dependencies = [
[[package]]
name = "portal"
version = "0.6.21"
version = "0.6.22"
dependencies = [
"anyhow",
"arc-swap",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "portal"
version = "0.6.21"
version = "0.6.22"
edition = "2021"
[lib]
+44
View File
@@ -126,6 +126,50 @@ mail autoconfiguration (`/.well-known/autoconfig/`) and the CalDAV and
CardDAV redirects are usually handed to a mail server that generates
them.
## A configuration profile
iOS has no autoconfiguration for IMAP accounts - Apple Mail reads neither
Mozilla autoconfig nor Microsoft Autodiscover, which are for ActiveSync - so
adding an account by hand leaves the device probing for ports. A probe that
meets a *filtered* port, a firewall that drops rather than refuses, waits out
a TCP timeout, and the account comes back "could not be verified", which reads
to the person as a wrong password. An Apple configuration profile ends the
guessing by stating the hosts and the ports, and it carries mail, calendars
and contacts in a single install.
Portal renders one at `/profile.mobileconfig` when `site.yaml` says where to
get the values:
```yaml
profile:
pacc: https://mail.example.no/.well-known/user-agent-configuration.json
name: Example # what the accounts are called on the device
organization: Example AS # the profile's publisher, defaults to name
```
`pacc` is the mail server's **PACC** document - the standard that replaces
guessing at hostnames, defined to be fetched from the mail domain itself, and
served by Stalwart among others. Every host, port and collection path in the
profile comes from it, so nothing about the server is written down a second
place to drift from the first: move a collection on the server and the profile
moves with it. Where PACC states no port, the registered implicit-TLS default
is used (993 for IMAP, 465 for submission, 443 for DAV).
What the profile contains is only what PACC already says in public. It carries
**no address and no username** - iOS asks for those and the password at
install - so one document serves everyone with an account and may be fetched
by anyone. Payload identifiers are derived from the site and the payload
rather than drawn at random, so reinstalling replaces the accounts instead of
stacking duplicates beside them.
Two things it cannot do. A profile Portal renders is unsigned, so iOS shows it
as *Unverified* while still offering to install it; signing needs a
certificate the device already trusts. And a PACC document describing no
protocol iOS understands - JMAP alone, or an incoming server with no outgoing
one - is an error rather than a half-empty profile that looks like it worked.
No `profile:` means `/profile.mobileconfig` is a 404.
## An alternative
One choice. Never two actions behind one heading: if two things can be
+351 -68
View File
@@ -70,12 +70,22 @@ pub enum Principal {
impl Principal {
pub fn anonymous() -> Self {
Principal::User { sub: "anonymous".into(), groups: Vec::new(), phone: None, email: None }
Principal::User {
sub: "anonymous".into(),
groups: Vec::new(),
phone: None,
email: None,
}
}
pub fn from_user(user: Option<&crate::auth::User>) -> Self {
match user {
Some(u) => Principal::User { sub: u.sub.clone(), groups: u.groups.clone(), phone: u.phone.clone(), email: u.email.clone() },
Some(u) => Principal::User {
sub: u.sub.clone(),
groups: u.groups.clone(),
phone: u.phone.clone(),
email: u.email.clone(),
},
None => Principal::anonymous(),
}
}
@@ -110,13 +120,23 @@ impl Action {
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum Resource {
Page { id: String },
Bucket { name: String },
Page {
id: String,
},
Bucket {
name: String,
},
/// One resource block on one alternative of one page. `feature` is
/// the feature name, or `feature|requirement` (`|bind` appended)
/// for a requirement's own resource.
Feature { page: String, alternative: String, feature: String },
Group { name: String },
Feature {
page: String,
alternative: String,
feature: String,
},
Group {
name: String,
},
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
@@ -164,12 +184,29 @@ impl Policy {
// Each rule is parsed on its own under its own id, so a
// decision's reasons name rules, not `policy17`.
let mut sources: Vec<(String, String)> = Vec::new();
let mut emit = |id_base: String, who: String, action: &str, resource: String, when: String, policy: String, text: &mut String, rules: &mut Vec<Rule>| {
let mut emit = |id_base: String,
who: String,
action: &str,
resource: String,
when: String,
policy: String,
text: &mut String,
rules: &mut Vec<Rule>| {
let n = ids.entry(id_base.clone()).or_insert(0);
*n += 1;
let id = if *n == 1 { id_base } else { format!("{id_base}#{n}") };
let id = if *n == 1 {
id_base
} else {
format!("{id_base}#{n}")
};
text.push_str(&format!("@id({})\n{policy}\n\n", quote(&id)));
rules.push(Rule { id: id.clone(), who, action: action.to_string(), resource, when });
rules.push(Rule {
id: id.clone(),
who,
action: action.to_string(),
resource,
when,
});
sources.push((id, policy));
};
@@ -179,7 +216,10 @@ impl Policy {
let page = quote(&q.id);
let (who, principal) = match &q.qualifies {
None => ("anyone".to_string(), "principal".to_string()),
Some(g) => (format!("group {g}"), format!("principal in Group::{}", quote(g))),
Some(g) => (
format!("group {g}"),
format!("principal in Group::{}", quote(g)),
),
};
emit(
format!("open:{}", q.id), who, "open, submit", format!("page {}", q.id), String::new(),
@@ -197,16 +237,25 @@ impl Policy {
}
for req in &feature.requirements {
if let Some(spec) = &req.resource {
specs.push((format!("{}|{}|{}|{}", q.id, alt.name, feature.name, req.name), spec));
specs.push((
format!("{}|{}|{}|{}", q.id, alt.name, feature.name, req.name),
spec,
));
}
if let Some(bind) = &req.bind {
specs.push((format!("{}|{}|{}|{}|bind", q.id, alt.name, feature.name, req.name), &bind.resource));
specs.push((
format!("{}|{}|{}|{}|bind", q.id, alt.name, feature.name, req.name),
&bind.resource,
));
}
}
}
for (fid, spec) in specs {
let (who, principal) = match (&spec.requires_group, spec.public) {
(Some(g), _) => (format!("group {g}"), format!("principal in Group::{}", quote(g))),
(Some(g), _) => (
format!("group {g}"),
format!("principal in Group::{}", quote(g)),
),
(None, true) => ("anyone".to_string(), "principal".to_string()),
// Neither: unreachable by design (fail closed).
(None, false) => continue,
@@ -279,20 +328,37 @@ impl Policy {
.expect("generated policy text always parses");
policies.add(policy).expect("rule ids are unique");
}
let schema = Schema::from_cedarschema_str(SCHEMA).map(|(s, _)| s).expect("the schema is a constant");
let patterns = questions.values().filter(|q| q.is_dynamic()).map(|q| q.id.clone()).collect();
Policy { policies, schema, rules, text, patterns }
let schema = Schema::from_cedarschema_str(SCHEMA)
.map(|(s, _)| s)
.expect("the schema is a constant");
let patterns = questions
.values()
.filter(|q| q.is_dynamic())
.map(|q| q.id.clone())
.collect();
Policy {
policies,
schema,
rules,
text,
patterns,
}
}
/// Every generated rule type-checks against the schema. A rule the
/// schema cannot express is a compiler bug, and this is where it
/// shows.
pub fn validate(&self) -> Result<(), String> {
let result = Validator::new(self.schema.clone()).validate(&self.policies, ValidationMode::Strict);
let result =
Validator::new(self.schema.clone()).validate(&self.policies, ValidationMode::Strict);
if result.validation_passed() {
Ok(())
} else {
Err(result.validation_errors().map(|e| e.to_string()).collect::<Vec<_>>().join("; "))
Err(result
.validation_errors()
.map(|e| e.to_string())
.collect::<Vec<_>>()
.join("; "))
}
}
@@ -317,7 +383,10 @@ impl Policy {
context: serde_json::Value,
) -> Decision {
let uid = |ty: &str, id: &str| {
EntityUid::from_type_name_and_id(EntityTypeName::from_str(ty).expect("static type"), EntityId::new(id))
EntityUid::from_type_name_and_id(
EntityTypeName::from_str(ty).expect("static type"),
EntityId::new(id),
)
};
let mut entities: Vec<Entity> = Vec::new();
let principal_uid = match principal {
@@ -332,8 +401,14 @@ impl Policy {
}
Principal::Submitter { record, email } => {
let me = uid("Submitter", record);
let attrs = HashMap::from([("email".to_string(), RestrictedExpression::new_string(email.clone()))]);
entities.push(Entity::new(me.clone(), attrs, HashSet::new()).expect("a string attribute always evaluates"));
let attrs = HashMap::from([(
"email".to_string(),
RestrictedExpression::new_string(email.clone()),
)]);
entities.push(
Entity::new(me.clone(), attrs, HashSet::new())
.expect("a string attribute always evaluates"),
);
me
}
Principal::Service { name } => {
@@ -345,9 +420,14 @@ impl Policy {
let resource_uid = match resource {
Resource::Page { id } => uid("Page", &self.page_entity(id)),
Resource::Bucket { name } => uid("Bucket", name),
Resource::Feature { page, alternative, feature } => {
uid("Feature", &format!("{}|{alternative}|{feature}", self.page_entity(page)))
}
Resource::Feature {
page,
alternative,
feature,
} => uid(
"Feature",
&format!("{}|{alternative}|{feature}", self.page_entity(page)),
),
Resource::Group { name } => uid("Group", name),
};
entities.push(Entity::new_no_attrs(resource_uid.clone(), HashSet::new()));
@@ -357,22 +437,32 @@ impl Policy {
return Decision::default();
};
// An absent context is an empty one, not a malformed one.
let context = if context.is_object() { context } else { serde_json::json!({}) };
let context = if context.is_object() {
context
} else {
serde_json::json!({})
};
let Ok(context) = Context::from_json_value(context, None) else {
return Decision::default();
};
let Ok(request) = Request::new(principal_uid, action_uid, resource_uid, context, None) else {
let Ok(request) = Request::new(principal_uid, action_uid, resource_uid, context, None)
else {
return Decision::default();
};
let response = Authorizer::new().is_authorized(&request, &self.policies, &entities);
Decision {
allow: response.decision() == cedar_policy::Decision::Allow,
reasons: response.diagnostics().reason().map(|id| id.to_string()).collect(),
reasons: response
.diagnostics()
.reason()
.map(|id| id.to_string())
.collect(),
}
}
pub fn allows(&self, principal: &Principal, action: Action, resource: &Resource) -> bool {
self.decide(principal, action, resource, serde_json::json!({})).allow
self.decide(principal, action, resource, serde_json::json!({}))
.allow
}
}
@@ -473,21 +563,50 @@ pub async fn serve(
// A question that names a person and lists no groups is
// asking portal what they belong to: their memberships,
// matched on subject, phone number or address.
if let (Principal::User { sub, groups, phone, email }, Some(state)) = (&mut q.principal, &state) {
if let (
Principal::User {
sub,
groups,
phone,
email,
},
Some(state),
) = (&mut q.principal, &state)
{
if groups.is_empty() && sub != "anonymous" {
let who = crate::memberships::Identity {
sub: Some(sub.clone()),
phone: phone.as_deref().and_then(|p| crate::memberships::normalize_phone(p, &std::env::var("PHONE_COUNTRY").unwrap_or_else(|_| "+47".into()))),
phone: phone.as_deref().and_then(|p| {
crate::memberships::normalize_phone(
p,
&std::env::var("PHONE_COUNTRY")
.unwrap_or_else(|_| "+47".into()),
)
}),
email: email.as_deref().map(|e| e.trim().to_lowercase()),
};
*groups = crate::memberships::groups_of(&crate::memberships::all(state).await, &who);
*groups = crate::memberships::groups_of(
&crate::memberships::all(state).await,
&who,
);
}
}
decide_and_publish(&nats, &access.load(), &q.principal, q.action, &q.resource, q.context)
decide_and_publish(
&nats,
&access.load(),
&q.principal,
q.action,
&q.resource,
q.context,
)
}
Err(e) => Decision { allow: false, reasons: vec![format!("malformed question: {e}")] },
Err(e) => Decision {
allow: false,
reasons: vec![format!("malformed question: {e}")],
},
};
let bytes = serde_json::to_vec(&decision).unwrap_or_else(|_| b"{\"allow\":false,\"reasons\":[]}".to_vec());
let bytes = serde_json::to_vec(&decision)
.unwrap_or_else(|_| b"{\"allow\":false,\"reasons\":[]}".to_vec());
let _ = nats.publish(reply, bytes.into()).await;
}
}
@@ -513,14 +632,22 @@ mod tests {
}
fn user(groups: &[&str]) -> Principal {
Principal::User { sub: "u".into(), groups: groups.iter().map(|g| g.to_string()).collect(), phone: None, email: None }
Principal::User {
sub: "u".into(),
groups: groups.iter().map(|g| g.to_string()).collect(),
phone: None,
email: None,
}
}
#[test]
fn the_generated_policy_validates_against_the_schema() {
let p = site();
assert_eq!(p.validate(), Ok(()), "{}", p.text);
assert!(p.rules.iter().any(|r| r.id == "move:applicants:open>invited"));
assert!(p
.rules
.iter()
.any(|r| r.id == "move:applicants:open>invited"));
}
#[test]
@@ -530,7 +657,12 @@ mod tests {
assert!(p.allows(&Principal::anonymous(), Action::Open, &page("/")));
assert!(!p.allows(&Principal::anonymous(), Action::Open, &page("/review")));
assert!(!p.allows(&user(&["staff"]), Action::Submit, &page("/review")));
let d = p.decide(&user(&["owners"]), Action::Submit, &page("/review"), json!({}));
let d = p.decide(
&user(&["owners"]),
Action::Submit,
&page("/review"),
json!({}),
);
assert!(d.allow);
assert_eq!(d.reasons, vec!["open:/review"]);
// A concrete dynamic path resolves to its pattern's rule.
@@ -541,8 +673,16 @@ mod tests {
#[test]
fn reads_follow_requires_group_or_public() {
let p = site();
let desk = Resource::Feature { page: "/review".into(), alternative: "Applicants".into(), feature: "".into() };
let releases = Resource::Feature { page: "/review".into(), alternative: "Applicants".into(), feature: "Releases".into() };
let desk = Resource::Feature {
page: "/review".into(),
alternative: "Applicants".into(),
feature: "".into(),
};
let releases = Resource::Feature {
page: "/review".into(),
alternative: "Applicants".into(),
feature: "Releases".into(),
};
assert!(p.allows(&user(&["owners"]), Action::Read, &desk));
assert!(!p.allows(&user(&["staff"]), Action::Read, &desk));
assert!(p.allows(&Principal::anonymous(), Action::Read, &releases));
@@ -551,26 +691,88 @@ mod tests {
#[test]
fn transitions_are_exactly_the_desks_buttons() {
let p = site();
let bucket = Resource::Bucket { name: "applicants".into() };
let bucket = Resource::Bucket {
name: "applicants".into(),
};
let ctx = |from: &str, to: &str| json!({ "from": from, "to": to });
assert!(p.decide(&user(&["owners"]), Action::Transition, &bucket, ctx("open", "invited")).allow);
assert!(p.decide(&user(&["owners"]), Action::Transition, &bucket, ctx("invited", "active")).allow);
assert!(
p.decide(
&user(&["owners"]),
Action::Transition,
&bucket,
ctx("open", "invited")
)
.allow
);
assert!(
p.decide(
&user(&["owners"]),
Action::Transition,
&bucket,
ctx("invited", "active")
)
.allow
);
// Right target, wrong starting state: no rule.
assert!(!p.decide(&user(&["owners"]), Action::Transition, &bucket, ctx("open", "active")).allow);
assert!(!p.decide(&user(&["staff"]), Action::Transition, &bucket, ctx("open", "invited")).allow);
assert!(
!p.decide(
&user(&["owners"]),
Action::Transition,
&bucket,
ctx("open", "active")
)
.allow
);
assert!(
!p.decide(
&user(&["staff"]),
Action::Transition,
&bucket,
ctx("open", "invited")
)
.allow
);
}
#[test]
fn a_submitter_moves_their_own_record_only_with_the_matching_email() {
let p = site();
let bucket = Resource::Bucket { name: "subscribers".into() };
let me = Principal::Submitter { record: "r1".into(), email: "a@b.c".into() };
let ctx = |stored: &str| json!({ "from": "open", "to": "unsubscribed", "stored_email": stored });
assert!(p.decide(&me, Action::SelfTransition, &bucket, ctx("a@b.c")).allow);
assert!(!p.decide(&me, Action::SelfTransition, &bucket, ctx("x@y.z")).allow);
assert!(!p.decide(&me, Action::SelfTransition, &bucket, json!({ "from": "open", "to": "somewhere", "stored_email": "a@b.c" })).allow);
let bucket = Resource::Bucket {
name: "subscribers".into(),
};
let me = Principal::Submitter {
record: "r1".into(),
email: "a@b.c".into(),
};
let ctx =
|stored: &str| json!({ "from": "open", "to": "unsubscribed", "stored_email": stored });
assert!(
p.decide(&me, Action::SelfTransition, &bucket, ctx("a@b.c"))
.allow
);
assert!(
!p.decide(&me, Action::SelfTransition, &bucket, ctx("x@y.z"))
.allow
);
assert!(
!p.decide(
&me,
Action::SelfTransition,
&bucket,
json!({ "from": "open", "to": "somewhere", "stored_email": "a@b.c" })
)
.allow
);
// A signed-in owner has no such rule: the link is the credential.
assert!(!p.decide(&user(&["owners"]), Action::SelfTransition, &bucket, ctx("a@b.c")).allow);
assert!(
!p.decide(
&user(&["owners"]),
Action::SelfTransition,
&bucket,
ctx("a@b.c")
)
.allow
);
}
// The applicant written to for a missing drawing sends it from
@@ -590,15 +792,52 @@ mod tests {
.unwrap();
let p = Policy::from_content(&questions, &aggregates);
assert_eq!(p.validate(), Ok(()), "{}", p.text);
let bucket = Resource::Bucket { name: "cases".into() };
let me = Principal::Submitter { record: "r1".into(), email: "a@b.c".into() };
let bucket = Resource::Bucket {
name: "cases".into(),
};
let me = Principal::Submitter {
record: "r1".into(),
email: "a@b.c".into(),
};
let ctx = |from: &str, to: &str| json!({ "from": from, "to": to, "stored_email": "a@b.c" });
assert!(p.decide(&me, Action::SelfTransition, &bucket, ctx("incomplete", "complete")).allow);
assert!(p.decide(&me, Action::SelfTransition, &bucket, ctx("complete", "withdrawn")).allow);
assert!(
p.decide(
&me,
Action::SelfTransition,
&bucket,
ctx("incomplete", "complete")
)
.allow
);
assert!(
p.decide(
&me,
Action::SelfTransition,
&bucket,
ctx("complete", "withdrawn")
)
.allow
);
// And only where it was said: the paperwork cannot be sent
// after the case is already withdrawn.
assert!(!p.decide(&me, Action::SelfTransition, &bucket, ctx("withdrawn", "complete")).allow);
assert!(!p.decide(&me, Action::SelfTransition, &bucket, ctx("complete", "complete")).allow);
assert!(
!p.decide(
&me,
Action::SelfTransition,
&bucket,
ctx("withdrawn", "complete")
)
.allow
);
assert!(
!p.decide(
&me,
Action::SelfTransition,
&bucket,
ctx("complete", "complete")
)
.allow
);
}
// Omitted, `from` is the initial state alone - what every
@@ -606,8 +845,17 @@ mod tests {
#[test]
fn a_move_that_does_not_say_where_from_is_a_one_shot_on_arrival() {
let p = site();
assert!(p.rules.iter().any(|r| r.id == "self:subscribers:open>unsubscribed"));
assert_eq!(p.rules.iter().filter(|r| r.id.starts_with("self:subscribers:")).count(), 1);
assert!(p
.rules
.iter()
.any(|r| r.id == "self:subscribers:open>unsubscribed"));
assert_eq!(
p.rules
.iter()
.filter(|r| r.id.starts_with("self:subscribers:"))
.count(),
1
);
}
// "On arrival" is the bucket's own first state, not the word
@@ -633,20 +881,49 @@ mod tests {
"the rule must start where the records do: {:?}",
p.rules.iter().map(|r| &r.id).collect::<Vec<_>>()
);
let me = Principal::Submitter { record: "r".into(), email: "a@b.c".into() };
let bucket = Resource::Bucket { name: "watchers".into() };
assert!(p.decide(&me, Action::SelfTransition, &bucket, json!({ "from": "pending", "to": "open", "stored_email": "a@b.c" })).allow);
let me = Principal::Submitter {
record: "r".into(),
email: "a@b.c".into(),
};
let bucket = Resource::Bucket {
name: "watchers".into(),
};
assert!(
p.decide(
&me,
Action::SelfTransition,
&bucket,
json!({ "from": "pending", "to": "open", "stored_email": "a@b.c" })
)
.allow
);
}
#[test]
fn invites_and_automation() {
let p = site();
let owners = Resource::Group { name: "owners".into() };
let owners = Resource::Group {
name: "owners".into(),
};
assert!(p.allows(&user(&["owners"]), Action::Invite, &owners));
assert!(!p.allows(&user(&["staff"]), Action::Invite, &owners));
let bucket = Resource::Bucket { name: "applicants".into() };
assert!(p.allows(&Principal::Service { name: "automation".into() }, Action::AutomationRead, &bucket));
assert!(!p.allows(&Principal::Service { name: "someone".into() }, Action::AutomationRead, &bucket));
let bucket = Resource::Bucket {
name: "applicants".into(),
};
assert!(p.allows(
&Principal::Service {
name: "automation".into()
},
Action::AutomationRead,
&bucket
));
assert!(!p.allows(
&Principal::Service {
name: "someone".into()
},
Action::AutomationRead,
&bucket
));
assert!(!p.allows(&user(&["owners"]), Action::AutomationRead, &bucket));
}
@@ -657,12 +934,18 @@ mod tests {
r#"{"principal":{"kind":"user","sub":"n8n-as-owner","groups":["owners"]},"action":"transition","resource":{"kind":"bucket","name":"applicants"},"context":{"from":"open","to":"invited"}}"#,
)
.unwrap();
assert!(p.decide(&q.principal, q.action, &q.resource, q.context).allow);
assert!(
p.decide(&q.principal, q.action, &q.resource, q.context)
.allow
);
let q: Query = serde_json::from_str(
r#"{"principal":{"kind":"service","name":"automation"},"action":"automation_read","resource":{"kind":"bucket","name":"applicants"}}"#,
)
.unwrap();
assert!(p.decide(&q.principal, q.action, &q.resource, q.context).allow);
assert!(
p.decide(&q.principal, q.action, &q.resource, q.context)
.allow
);
assert_eq!(subject_for("uhhm"), "portal.access.uhhm.may");
}
+20 -7
View File
@@ -78,7 +78,9 @@ fn is_zero(n: &u32) -> bool {
impl LeavesSpec {
pub fn millis(&self) -> i64 {
((i64::from(self.days) * 24 + i64::from(self.hours)) * 60 + i64::from(self.minutes)) * 60 * 1000
((i64::from(self.days) * 24 + i64::from(self.hours)) * 60 + i64::from(self.minutes))
* 60
* 1000
}
}
@@ -200,8 +202,7 @@ pub fn parse_aggregates_yaml(raw: &str) -> anyhow::Result<HashMap<String, Aggreg
let mut event_for_state = HashMap::new();
let mut state_for_event = HashMap::new();
for (state, raw_state) in &raw_schema.states {
if let Some(existing) = state_for_event.insert(raw_state.event.clone(), state.clone())
{
if let Some(existing) = state_for_event.insert(raw_state.event.clone(), state.clone()) {
anyhow::bail!(
"aggregates.yaml: bucket {bucket:?}: event {:?} used by both state {:?} and {:?}",
raw_state.event, existing, state
@@ -238,7 +239,9 @@ pub fn parse_aggregates_yaml(raw: &str) -> anyhow::Result<HashMap<String, Aggreg
}
if let Some(group) = raw_state.grants {
if group.trim().is_empty() {
anyhow::bail!("aggregates.yaml: bucket {bucket:?}: state {state:?}: grants needs a group");
anyhow::bail!(
"aggregates.yaml: bucket {bucket:?}: state {state:?}: grants needs a group"
);
}
grants_for_state.insert(state.clone(), group);
}
@@ -248,7 +251,11 @@ pub fn parse_aggregates_yaml(raw: &str) -> anyhow::Result<HashMap<String, Aggreg
}
// A clock may only make a move a person could have
// made, so it is held to the same graph.
if !raw_schema.transitions.get(&state).is_some_and(|tos| tos.contains(&after.to)) {
if !raw_schema
.transitions
.get(&state)
.is_some_and(|tos| tos.contains(&after.to))
{
anyhow::bail!(
"aggregates.yaml: bucket {bucket:?}: state {state:?}: after.to {:?} is not a declared transition from it",
after.to
@@ -261,7 +268,10 @@ pub fn parse_aggregates_yaml(raw: &str) -> anyhow::Result<HashMap<String, Aggreg
anyhow::bail!("aggregates.yaml: bucket {bucket:?}: state {state:?}: mail needs a subject and a body");
}
let through_a_record = crate::mail::parent_of(&mail.to).is_some();
if !mail.to_submitter() && !through_a_record && !(mail.to.contains('@') && !mail.to.contains(char::is_whitespace)) {
if !mail.to_submitter()
&& !through_a_record
&& !(mail.to.contains('@') && !mail.to.contains(char::is_whitespace))
{
anyhow::bail!(
"aggregates.yaml: bucket {bucket:?}: state {state:?}: mail.to must be `submitter`, one address, or a field of the record this one belongs to like `{{notice.email}}`, not {:?}",
mail.to
@@ -334,7 +344,10 @@ pub fn replay(schema: &AggregateSchema, id: &str, events: &[Event]) -> Option<Ag
pub enum TransitionError {
/// No creation event found for this id - nothing to transition.
UnknownAggregate,
DisallowedTransition { from: String, to: String },
DisallowedTransition {
from: String,
to: String,
},
/// A concurrent writer advanced this aggregate past the sequence
/// this call read (see `events::store::append_event`'s CAS).
Conflict,
+26 -4
View File
@@ -37,7 +37,11 @@ pub async fn run(state: AppState) {
pub async fn sweep(state: &AppState) -> anyhow::Result<()> {
let now_ms = chrono::Utc::now().timestamp_millis();
let questions = state.questions.load();
let events: Vec<_> = questions.values().filter(|q| q.event.is_some()).cloned().collect();
let events: Vec<_> = questions
.values()
.filter(|q| q.event.is_some())
.cloned()
.collect();
if events.is_empty() {
return Ok(());
}
@@ -100,7 +104,9 @@ pub async fn sweep(state: &AppState) -> anyhow::Result<()> {
let mut refreshed = answer.clone();
refreshed.responses = responses.clone();
if let Some(store) = &store {
store.put(&id, serde_json::to_vec(&refreshed)?.into()).await?;
store
.put(&id, serde_json::to_vec(&refreshed)?.into())
.await?;
tracing::info!(question = %q.id, "event record refreshed from content");
}
}
@@ -108,13 +114,29 @@ pub async fn sweep(state: &AppState) -> anyhow::Result<()> {
if answer.state == schema.initial && now_ms >= ends {
let target = "awaiting_summary";
let payload = serde_json::json!({ "to": target, "item": id, "by": "portal" });
match crate::aggregates::transition(&state.jetstream, schema, &id, target, payload.clone(), now_ms).await {
match crate::aggregates::transition(
&state.jetstream,
schema,
&id,
target,
payload.clone(),
now_ms,
)
.await
{
Ok(_) => {}
Err(crate::aggregates::TransitionError::UnknownAggregate) => {
crate::aggregates::reseed(&state.jetstream, schema, &id, &answer.state, now_ms)
.await
.map_err(|e| anyhow::anyhow!("{e}"))?;
crate::aggregates::transition(&state.jetstream, schema, &id, target, payload.clone(), now_ms)
crate::aggregates::transition(
&state.jetstream,
schema,
&id,
target,
payload.clone(),
now_ms,
)
.await
.map_err(|e| anyhow::anyhow!("{e}"))?;
}
+126 -25
View File
@@ -113,7 +113,9 @@ pub async fn store_answer(
// submission the NATS notification event has already recorded"
// policy as the KV write above.
if let Some(schema) = aggregates.get(bucket) {
if let Err(e) = crate::aggregates::create(js, schema, &id, responses.clone(), submitted_ms).await {
if let Err(e) =
crate::aggregates::create(js, schema, &id, responses.clone(), submitted_ms).await
{
tracing::error!(bucket, id, error = %e, "failed to seed event log for submission");
}
}
@@ -127,17 +129,27 @@ pub const OWNER_FIELD: &str = "by";
/// Every record in `bucket` that belongs to `username`, newest first.
#[cfg(feature = "ssr")]
pub async fn own_records(js: &async_nats::jetstream::Context, bucket: &str, username: &str) -> Vec<Answer> {
pub async fn own_records(
js: &async_nats::jetstream::Context,
bucket: &str,
username: &str,
) -> Vec<Answer> {
use futures::TryStreamExt;
let Ok(store) = js.get_key_value(bucket).await else { return Vec::new() };
let Ok(store) = js.get_key_value(bucket).await else {
return Vec::new();
};
let keys: Vec<String> = match store.keys().await {
Ok(keys) => keys.try_collect().await.unwrap_or_default(),
Err(_) => return Vec::new(),
};
let mut own = Vec::new();
for key in keys {
let Ok(Some(bytes)) = store.get(&key).await else { continue };
let Ok(answer) = serde_json::from_slice::<Answer>(&bytes) else { continue };
let Ok(Some(bytes)) = store.get(&key).await else {
continue;
};
let Ok(answer) = serde_json::from_slice::<Answer>(&bytes) else {
continue;
};
if answer.responses.get(OWNER_FIELD).and_then(|v| v.as_str()) == Some(username) {
own.push(answer);
}
@@ -151,9 +163,16 @@ pub async fn own_records(js: &async_nats::jetstream::Context, bucket: &str, user
/// it (the event log keeps them). Run after every store, so two sends
/// racing each other - a double click - both settle on the same one.
#[cfg(feature = "ssr")]
pub async fn keep_newest_own(js: &async_nats::jetstream::Context, bucket: &str, initial: &str, username: &str) {
pub async fn keep_newest_own(
js: &async_nats::jetstream::Context,
bucket: &str,
initial: &str,
username: &str,
) {
let own = own_records(js, bucket, username).await;
let Ok(store) = js.get_key_value(bucket).await else { return };
let Ok(store) = js.get_key_value(bucket).await else {
return;
};
for older in own.iter().filter(|a| a.state == initial).skip(1) {
if let Err(e) = store.delete(&older.id).await {
tracing::error!(bucket, id = %older.id, error = %e, "could not retire an older own record");
@@ -167,16 +186,39 @@ pub async fn keep_newest_own(js: &async_nats::jetstream::Context, bucket: &str,
/// the address a list would open it from, and may be left empty to
/// keep it.
#[cfg(feature = "ssr")]
pub async fn fill_own(js: &async_nats::jetstream::Context, question: &mut crate::content::Question, user: Option<&crate::auth::User>) {
pub async fn fill_own(
js: &async_nats::jetstream::Context,
question: &mut crate::content::Question,
user: Option<&crate::auth::User>,
) {
let Some(user) = user else { return };
for alt in &mut question.alternatives {
let (true, Some(bucket)) = (alt.one_per_person, alt.record_as.clone()) else { continue };
let Some(mine) = own_records(js, &bucket, &user.username).await.into_iter().next() else { continue };
for req in alt.features.iter_mut().flat_map(|f| f.requirements.iter_mut()) {
let (true, Some(bucket)) = (alt.one_per_person, alt.record_as.clone()) else {
continue;
};
let Some(mine) = own_records(js, &bucket, &user.username)
.await
.into_iter()
.next()
else {
continue;
};
for req in alt
.features
.iter_mut()
.flat_map(|f| f.requirements.iter_mut())
{
if req.kind == "hidden" {
continue;
}
let Some(value) = mine.responses.get(&req.name).and_then(|v| v.as_str()).filter(|v| !v.trim().is_empty()) else { continue };
let Some(value) = mine
.responses
.get(&req.name)
.and_then(|v| v.as_str())
.filter(|v| !v.trim().is_empty())
else {
continue;
};
if req.kind == "file" {
req.value = Some(format!("/attachment/{bucket}/{}/{}", mine.id, req.name));
req.optional = true;
@@ -217,7 +259,12 @@ pub async fn transition_answers(
.load()
.at(storyline)
.get(&question_id)
.and_then(|q| q.alternatives.iter().find(|a| a.name == alternative).cloned())
.and_then(|q| {
q.alternatives
.iter()
.find(|a| a.name == alternative)
.cloned()
})
.ok_or_else(|| ServerFnError::new("unknown question or alternative"))?;
let session: tower_sessions::Session = leptos_axum::extract().await?;
@@ -333,7 +380,9 @@ async fn apply_transition(
&state.access.load(),
&crate::access::Principal::from_user(Some(user)),
crate::access::Action::Transition,
&crate::access::Resource::Bucket { name: bucket.to_string() },
&crate::access::Resource::Bucket {
name: bucket.to_string(),
},
serde_json::json!({ "from": answer.state, "to": item.to }),
);
if !decision.allow {
@@ -370,7 +419,15 @@ async fn apply_transition(
.put(&item.item_id, data.into())
.await
.map_err(|e| ServerFnError::new(e.to_string()))?;
crate::mail::on_state(state, bucket, &item.item_id, &item.to, &answer.responses, answer.storyline).await;
crate::mail::on_state(
state,
bucket,
&item.item_id,
&item.to,
&answer.responses,
answer.storyline,
)
.await;
crate::people::on_state(state, bucket, &item.item_id, &item.to, &answer.responses).await;
// An expired invite sent again: the move back is the asking.
if bucket == crate::people::INVITES_BUCKET && item.to == OPEN_STATE {
@@ -380,7 +437,12 @@ async fn apply_transition(
// The decision extends the DAG as a child of the answer's own
// submission hash, published like any other answer.
let parent_hashes = vec![item.item_id.clone()];
let chain_hash = hash_node(question_id, &parent_hashes, &decision_payload_for_event, decided_ms);
let chain_hash = hash_node(
question_id,
&parent_hashes,
&decision_payload_for_event,
decided_ms,
);
let event = AnswerSubmitted {
chain_hash,
parent_hashes,
@@ -447,13 +509,22 @@ pub async fn self_transition_answer(
// policy compares them and knows the one move a submitter may make.
// Refusal reads "not found" so the endpoint cannot be used to probe
// which records or emails exist.
let stored_email = answer.responses.get("email").and_then(|v| v.as_str()).unwrap_or_default();
let stored_email = answer
.responses
.get("email")
.and_then(|v| v.as_str())
.unwrap_or_default();
let decision = crate::access::decide_and_publish(
&state.nats,
&state.access.load(),
&crate::access::Principal::Submitter { record: item_id.clone(), email: email.clone() },
&crate::access::Principal::Submitter {
record: item_id.clone(),
email: email.clone(),
},
crate::access::Action::SelfTransition,
&crate::access::Resource::Bucket { name: self_transition.bucket.clone() },
&crate::access::Resource::Bucket {
name: self_transition.bucket.clone(),
},
serde_json::json!({ "from": answer.state, "to": self_transition.to, "stored_email": stored_email }),
);
if !decision.allow {
@@ -468,8 +539,15 @@ pub async fn self_transition_answer(
.get(&self_transition.bucket)
.map(|s| s.initial.clone())
.unwrap_or_else(|| OPEN_STATE.to_string());
let in_reach = self_transition.from_states(&initial).iter().any(|s| s == &answer.state);
return Err(ServerFnError::new(if in_reach { "not found" } else { "already processed" }));
let in_reach = self_transition
.from_states(&initial)
.iter()
.any(|s| s == &answer.state);
return Err(ServerFnError::new(if in_reach {
"not found"
} else {
"already processed"
}));
}
let decided_ms = chrono::Utc::now().timestamp_millis();
@@ -499,11 +577,31 @@ pub async fn self_transition_answer(
.put(&item_id, data.into())
.await
.map_err(|e| ServerFnError::new(e.to_string()))?;
crate::mail::on_state(&state, &self_transition.bucket, &item_id, &self_transition.to, &answer.responses, answer.storyline).await;
crate::people::on_state(&state, &self_transition.bucket, &item_id, &self_transition.to, &answer.responses).await;
crate::mail::on_state(
&state,
&self_transition.bucket,
&item_id,
&self_transition.to,
&answer.responses,
answer.storyline,
)
.await;
crate::people::on_state(
&state,
&self_transition.bucket,
&item_id,
&self_transition.to,
&answer.responses,
)
.await;
let parent_hashes = vec![item_id.clone()];
let chain_hash = hash_node(&question_id, &parent_hashes, &decision_payload_for_event, decided_ms);
let chain_hash = hash_node(
&question_id,
&parent_hashes,
&decision_payload_for_event,
decided_ms,
);
let event = AnswerSubmitted {
chain_hash,
parent_hashes,
@@ -560,7 +658,10 @@ mod tests {
},
];
let value = serde_json::Value::Array(
answers.iter().map(|a| serde_json::to_value(a).unwrap()).collect(),
answers
.iter()
.map(|a| serde_json::to_value(a).unwrap())
.collect(),
);
let round_tripped: Vec<Answer> =
serde_json::from_value(value).expect("Answer list round-trips");
+486 -136
View File
File diff suppressed because it is too large Load Diff
+3 -1
View File
@@ -53,7 +53,9 @@ pub async fn record_node(
alternative: Some(alternative.to_string()),
timestamp_ms,
};
store.put(chain_hash, serde_json::to_vec(&node)?.into()).await?;
store
.put(chain_hash, serde_json::to_vec(&node)?.into())
.await?;
Ok(())
}
+38 -12
View File
@@ -9,7 +9,10 @@ use super::*;
/// config; a file that exists but doesn't parse or validate is a real
/// error, surfaced at boot rather than papered over.
#[cfg(feature = "ssr")]
pub async fn load_site_from_gitea(repo_url: &str, branch: &str) -> anyhow::Result<Storylines<SiteConfig>> {
pub async fn load_site_from_gitea(
repo_url: &str,
branch: &str,
) -> anyhow::Result<Storylines<SiteConfig>> {
let (owner, repo) = parse_owner_repo(repo_url)?;
let api_base = gitea_api_base(repo_url)?;
let raw = match fetch_gitea_file(&api_base, &owner, &repo, branch, "site.yaml").await {
@@ -24,8 +27,8 @@ pub async fn load_site_from_gitea(repo_url: &str, branch: &str) -> anyhow::Resul
let arity = super::variants::arity(&files)?;
let mut each = Vec::with_capacity(arity);
for n in 0..arity {
let site: SiteConfig =
serde_yaml::from_str(&super::variants::storyline(&files, n)[0].1).map_err(|e| anyhow::anyhow!("parsing site.yaml: {e}"))?;
let site: SiteConfig = serde_yaml::from_str(&super::variants::storyline(&files, n)[0].1)
.map_err(|e| anyhow::anyhow!("parsing site.yaml: {e}"))?;
site.validate()?;
each.push(site);
}
@@ -40,7 +43,10 @@ pub async fn load_site_from_gitea(repo_url: &str, branch: &str) -> anyhow::Resul
#[cfg(feature = "ssr")]
pub fn gitea_raw_base(repo_url: &str) -> anyhow::Result<String> {
let (owner, repo) = parse_owner_repo(repo_url)?;
Ok(format!("{}/api/v1/repos/{owner}/{repo}/raw", gitea_api_base(repo_url)?))
Ok(format!(
"{}/api/v1/repos/{owner}/{repo}/raw",
gitea_api_base(repo_url)?
))
}
#[cfg(feature = "ssr")]
@@ -114,7 +120,8 @@ pub(crate) async fn fetch_gitea_file(
pub async fn load_aggregates_from_gitea(
repo_url: &str,
branch: &str,
) -> anyhow::Result<Storylines<std::collections::HashMap<String, crate::aggregates::AggregateSchema>>> {
) -> anyhow::Result<Storylines<std::collections::HashMap<String, crate::aggregates::AggregateSchema>>>
{
let (owner, repo) = parse_owner_repo(repo_url)?;
let api_base = gitea_api_base(repo_url)?;
let raw = fetch_gitea_file(&api_base, &owner, &repo, branch, "aggregates.yaml").await?;
@@ -123,7 +130,10 @@ pub async fn load_aggregates_from_gitea(
let files = vec![("aggregates.yaml".to_string(), raw)];
let arity = super::variants::arity(&files)?;
let each = (0..arity)
.map(|n| crate::aggregates::parse_aggregates_yaml(&super::variants::storyline(&files, n)[0].1).map(with_builtin_aggregates))
.map(|n| {
crate::aggregates::parse_aggregates_yaml(&super::variants::storyline(&files, n)[0].1)
.map(with_builtin_aggregates)
})
.collect::<anyhow::Result<Vec<_>>>()?;
Ok(Storylines { each })
}
@@ -183,7 +193,11 @@ pub async fn load_questions_from_gitea(
#[cfg(feature = "ssr")]
pub fn same_arity(questions: usize, aggregates: usize, site: usize) -> anyhow::Result<usize> {
let arity = questions.max(aggregates).max(site);
for (what, n) in [("questions", questions), ("aggregates.yaml", aggregates), ("site.yaml", site)] {
for (what, n) in [
("questions", questions),
("aggregates.yaml", aggregates),
("site.yaml", site),
] {
if n != 1 && n != arity {
anyhow::bail!("{what} tells {n} storyline(s), but the site tells {arity}; every list of texts has the same length");
}
@@ -219,9 +233,13 @@ pub async fn watch_for_reload(
repo_url: String,
branch: String,
subdir: String,
questions: std::sync::Arc<arc_swap::ArcSwap<Storylines<std::collections::HashMap<String, Question>>>>,
questions: std::sync::Arc<
arc_swap::ArcSwap<Storylines<std::collections::HashMap<String, Question>>>,
>,
aggregates: std::sync::Arc<
arc_swap::ArcSwap<Storylines<std::collections::HashMap<String, crate::aggregates::AggregateSchema>>>,
arc_swap::ArcSwap<
Storylines<std::collections::HashMap<String, crate::aggregates::AggregateSchema>>,
>,
>,
site: std::sync::Arc<arc_swap::ArcSwap<Storylines<SiteConfig>>>,
access: std::sync::Arc<arc_swap::ArcSwap<crate::access::Policy>>,
@@ -255,14 +273,23 @@ pub async fn watch_for_reload(
};
match load_questions_from_gitea(&repo_url, &branch, &subdir).await {
Ok(loaded) => {
if let Err(e) = validate_questions(&loaded, &loaded_aggregates).and_then(|_| same_arity(loaded.arity(), loaded_aggregates.arity(), loaded_site.arity())) {
if let Err(e) = validate_questions(&loaded, &loaded_aggregates).and_then(|_| {
same_arity(
loaded.arity(),
loaded_aggregates.arity(),
loaded_site.arity(),
)
}) {
tracing::error!(error = %e, "content reload failed validation, keeping last-good content");
continue;
}
let count = loaded.len();
// The policy is compiled from the content it gates, so
// it swaps in the same breath.
access.store(std::sync::Arc::new(crate::access::Policy::from_content(&loaded, &loaded_aggregates)));
access.store(std::sync::Arc::new(crate::access::Policy::from_content(
&loaded,
&loaded_aggregates,
)));
questions.store(std::sync::Arc::new(loaded));
aggregates.store(std::sync::Arc::new(loaded_aggregates));
site.store(std::sync::Arc::new(loaded_site));
@@ -275,4 +302,3 @@ pub async fn watch_for_reload(
}
}
}
+20 -7
View File
@@ -118,7 +118,6 @@ pub(crate) fn rewrite_img_tag(tag: &str) -> String {
cleaned.replacen("<img", &format!("<img{attrs}"), 1)
}
/// Markdown that a person wrote into a record - a bio on a card - as
/// opposed to the site's own content. Paragraphs, line breaks as they
/// were typed, lists, emphasis and links; no images (a picture has its
@@ -147,7 +146,9 @@ pub fn render_written_markdown(text: &str) -> String {
// What someone typed on a new line is on a new line.
Event::SoftBreak => Some(Event::HardBreak),
Event::Start(Tag::Link { dest_url, .. })
if !(dest_url.starts_with("https://") || dest_url.starts_with("mailto:") || dest_url.starts_with("tel:")) =>
if !(dest_url.starts_with("https://")
|| dest_url.starts_with("mailto:")
|| dest_url.starts_with("tel:")) =>
{
in_dropped_link += 1;
None
@@ -160,7 +161,8 @@ pub fn render_written_markdown(text: &str) -> String {
});
let mut out = String::new();
html::push_html(&mut out, filtered);
out.trim().replace("<a href=", "<a rel=\"nofollow ugc\" href=")
out.trim()
.replace("<a href=", "<a rel=\"nofollow ugc\" href=")
}
#[cfg(test)]
@@ -171,14 +173,25 @@ mod written_tests {
fn a_bio_keeps_what_a_person_would_write_and_nothing_that_reaches_out() {
assert_eq!(md("Line one\nline two"), "<p>Line one<br />\nline two</p>");
assert!(md("- NS 3424\n- NS 3423").contains("<ul>\n<li>NS 3424</li>"));
assert!(md("**Sivilingeniør**, *NTNU*").contains("<strong>Sivilingeniør</strong>, <em>NTNU</em>"));
assert_eq!(md("[Westra](https://westra.no)"), "<p><a rel=\"nofollow ugc\" href=\"https://westra.no\">Westra</a></p>");
assert!(md("**Sivilingeniør**, *NTNU*")
.contains("<strong>Sivilingeniør</strong>, <em>NTNU</em>"));
assert_eq!(
md("[Westra](https://westra.no)"),
"<p><a rel=\"nofollow ugc\" href=\"https://westra.no\">Westra</a></p>"
);
// Everything that would put something else in the page is gone.
assert_eq!(md("<script>alert(1)</script>"), "");
assert_eq!(md("hi <img src=x onerror=alert(1)>"), "<p>hi </p>");
assert_eq!(md("[click](javascript:alert(1))"), "<p>click</p>");
assert_eq!(md("[home](/kontor)"), "<p>home</p>", "a site path is the site's to link, not a bio's");
assert_eq!(md("before ![pixel](https://tracker.example/p.gif) after"), "<p>before after</p>");
assert_eq!(
md("[home](/kontor)"),
"<p>home</p>",
"a site path is the site's to link, not a bio's"
);
assert_eq!(
md("before ![pixel](https://tracker.example/p.gif) after"),
"<p>before after</p>"
);
assert_eq!(md("# Shouting"), "<p>Shouting</p>");
}
}
+325 -80
View File
@@ -13,19 +13,19 @@ use serde::{Deserialize, Serialize};
// Loading is server-only; the rest is shared with the browser build.
#[cfg(feature = "ssr")]
mod gitea;
#[cfg(feature = "ssr")]
mod variants;
mod handlers;
mod markdown;
mod validate;
#[cfg(feature = "ssr")]
mod variants;
#[cfg(feature = "ssr")]
pub use gitea::*;
#[cfg(feature = "ssr")]
pub use variants::*;
pub use handlers::*;
pub use markdown::*;
pub use validate::*;
#[cfg(feature = "ssr")]
pub use variants::*;
/// One page: a prompt plus the paths on from it. The `questions/`
/// directory tree IS the URL tree (`index.yaml` names its directory,
@@ -129,7 +129,10 @@ impl Place {
pub fn validate(&self) -> Result<(), String> {
if !(-90.0..=90.0).contains(&self.lat) || !(-180.0..=180.0).contains(&self.lon) {
return Err(format!("lat {} / lon {} is not a point on Earth", self.lat, self.lon));
return Err(format!(
"lat {} / lon {} is not a point on Earth",
self.lat, self.lon
));
}
if !(0.0..=22.0).contains(&self.zoom) {
return Err(format!("zoom {} is outside 0 to 22", self.zoom));
@@ -158,13 +161,23 @@ pub fn parse_duration_ms(text: &str) -> Result<i64, String> {
let mut total: i64 = 0;
let mut seen = false;
for token in text.split_whitespace() {
let (num, unit) = token.split_at(token.trim_end_matches(|c: char| c.is_ascii_alphabetic()).len());
let n: i64 = num.parse().map_err(|_| format!("bad duration {text:?}: {token:?}"))?;
let (num, unit) = token.split_at(
token
.trim_end_matches(|c: char| c.is_ascii_alphabetic())
.len(),
);
let n: i64 = num
.parse()
.map_err(|_| format!("bad duration {text:?}: {token:?}"))?;
let per = match unit {
"m" => 60_000,
"h" => 3_600_000,
"d" => 86_400_000,
_ => return Err(format!("bad duration {text:?}: unit {unit:?} (use m, h, d)")),
_ => {
return Err(format!(
"bad duration {text:?}: unit {unit:?} (use m, h, d)"
))
}
};
total += n * per;
seen = true;
@@ -228,7 +241,9 @@ pub fn fill_from_user(question: &mut Question, user: Option<&crate::auth::User>)
for alt in &mut question.alternatives {
for feature in &mut alt.features {
for req in &mut feature.requirements {
let Some(value) = req.value.as_deref() else { continue };
let Some(value) = req.value.as_deref() else {
continue;
};
let filled = match value {
"{user.email}" => user.and_then(|u| u.email.clone()),
"{user.name}" => user.map(|u| u.display_name.clone()),
@@ -274,7 +289,13 @@ pub fn template_action(action: &str, responses: &serde_json::Value) -> Option<St
// A URL segment: keep it to what a segment can carry.
let safe: String = scalar
.chars()
.map(|c| if c.is_ascii_alphanumeric() || "-_.".contains(c) { c } else { '-' })
.map(|c| {
if c.is_ascii_alphanumeric() || "-_.".contains(c) {
c
} else {
'-'
}
})
.collect();
out.push_str(&safe);
rest = &rest[end + 1..];
@@ -539,18 +560,27 @@ pub struct ResourceSpec {
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum ResourceSource {
Kv { bucket: String },
GiteaStarred { username: String },
Kv {
bucket: String,
},
GiteaStarred {
username: String,
},
/// A repo's releases - fetched with `GITEA_API_TOKEN`, so it works
/// for private repos too (shape the link with jq: a private
/// release's html_url 404s for anonymous visitors, so map it to
/// null unless the repo is public).
GiteaReleases { owner: String, repo: String },
GiteaReleases {
owner: String,
repo: String,
},
/// Any other HTTPS JSON endpoint. Scheme-restricted and checked
/// against loopback/private/link-local addresses at fetch time
/// (`resource::fetch_url_resource`) - a server-side fetch of a
/// content-supplied URL is SSRF surface, so it fails closed.
Url { url: String },
Url {
url: String,
},
}
impl ResourceSpec {
@@ -826,6 +856,28 @@ pub struct SiteConfig {
/// skipped, never half-sent (see `mail.rs`).
#[serde(default)]
pub mail: Option<MailConfig>,
/// Where `/profile.mobileconfig` gets its values. Absent means the
/// site offers no configuration profile and that route is a 404.
#[serde(default)]
pub profile: Option<ProfileConfig>,
}
/// An Apple configuration profile rendered from the mail server's own PACC
/// document - see `profile.rs`. Nothing here describes a server: `pacc` says
/// where the description already lives, so hosts and ports are not written
/// down a second time to drift from the first.
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)]
pub struct ProfileConfig {
/// The PACC document, normally
/// `https://<mail host>/.well-known/user-agent-configuration.json`.
pub pacc: String,
/// What the accounts are called on the device. Defaults to the site's
/// own host.
#[serde(default)]
pub name: Option<String>,
/// Shown as the profile's publisher. Defaults to `name`.
#[serde(default)]
pub organization: Option<String>,
}
/// The sender identity, and the one built-in mail's wording.
@@ -867,7 +919,10 @@ pub struct HeroConfig {
impl Default for HeroConfig {
fn default() -> Self {
Self { kind: default_hero_kind(), module: None }
Self {
kind: default_hero_kind(),
module: None,
}
}
}
@@ -907,6 +962,14 @@ impl SiteConfig {
);
}
}
if let Some(profile) = &self.profile {
if !profile.pacc.starts_with("https://") {
anyhow::bail!(
"site.yaml: profile.pacc {:?} must be an https:// URL (the PACC document the profile is built from)",
profile.pacc
);
}
}
if let Some(icon) = &self.favicon {
let ok = [".svg", ".png", ".ico"].iter().any(|e| icon.ends_with(e));
if !is_safe_site_path(icon) || !ok {
@@ -1084,7 +1147,12 @@ pub fn earned_followups(
queue.push_back(q);
}
while let Some(q) = queue.pop_front() {
for action in q.alternatives.iter().filter(|a| asks_nothing(a)).filter_map(|a| a.action.as_deref()) {
for action in q
.alternatives
.iter()
.filter(|a| asks_nothing(a))
.filter_map(|a| a.action.as_deref())
{
let Some(id) = action_target_id(questions, action) else {
continue;
};
@@ -1177,9 +1245,13 @@ pub fn build_questions(
/// Every storyline of the content, each a full map of questions.
#[cfg(feature = "ssr")]
pub fn build_storylines(files: &[(String, String)]) -> anyhow::Result<variants::Storylines<std::collections::HashMap<String, Question>>> {
pub fn build_storylines(
files: &[(String, String)],
) -> anyhow::Result<variants::Storylines<std::collections::HashMap<String, Question>>> {
let arity = variants::arity(files)?;
let each = (0..arity).map(|n| build_one_storyline(&variants::storyline(files, n))).collect::<anyhow::Result<Vec<_>>>()?;
let each = (0..arity)
.map(|n| build_one_storyline(&variants::storyline(files, n)))
.collect::<anyhow::Result<Vec<_>>>()?;
Ok(variants::Storylines { each })
}
@@ -1195,8 +1267,8 @@ fn build_one_storyline(
if name != "_section.yaml" {
continue;
}
let mut section: SectionConfig = serde_yaml::from_str(raw)
.map_err(|e| anyhow::anyhow!("parsing {rel}: {e}"))?;
let mut section: SectionConfig =
serde_yaml::from_str(raw).map_err(|e| anyhow::anyhow!("parsing {rel}: {e}"))?;
let dir = dir_from_path(rel);
if let Some(rc) = &section.requires_chain {
section.requires_chain = Some(resolve_ref(&dir, rc));
@@ -1301,7 +1373,9 @@ aggregates:
.unwrap()
}
fn question_with_transitions(transitions_yaml: &str) -> std::collections::HashMap<String, Question> {
fn question_with_transitions(
transitions_yaml: &str,
) -> std::collections::HashMap<String, Question> {
let question: Question = serde_yaml::from_str(&format!(
r#"
id: /t
@@ -1321,7 +1395,6 @@ alternatives:
std::collections::HashMap::from([(question.id.clone(), question)])
}
fn question_with_record(field_yaml: &str) -> std::collections::HashMap<String, Question> {
let question: Question = serde_yaml::from_str(&format!(
r#"
@@ -1356,9 +1429,13 @@ alternatives:
let good = question_with_record(" - { name: case, type: record, of: things }");
validate_questions(&good, &schema_fixture()).unwrap();
let nowhere = question_with_record(" - { name: case, type: record, of: nothing_declared }");
let nowhere =
question_with_record(" - { name: case, type: record, of: nothing_declared }");
let err = validate_questions(&nowhere, &schema_fixture()).unwrap_err();
assert!(err.to_string().contains("has no aggregate declared"), "{err}");
assert!(
err.to_string().contains("has no aggregate declared"),
"{err}"
);
let unsaid = question_with_record(" - { name: case, type: record }");
let err = validate_questions(&unsaid, &schema_fixture()).unwrap_err();
@@ -1367,12 +1444,19 @@ alternatives:
// `of` means one thing, so it belongs to one type.
let stray = question_with_record(" - { name: case, of: things }");
let err = validate_questions(&stray, &schema_fixture()).unwrap_err();
assert!(err.to_string().contains("only a type: record field has"), "{err}");
assert!(
err.to_string().contains("only a type: record field has"),
"{err}"
);
// The id arrives on the link; it is not looked up.
let bound = question_with_record(" - name: case\n type: record\n of: things\n bind:\n field: what\n resource:\n source: { kind: url, url: \"https://x.example/{p}\" }");
let err = validate_questions(&bound, &schema_fixture()).unwrap_err();
assert!(err.to_string().contains("the link it came in on carries the id"), "{err}");
assert!(
err.to_string()
.contains("the link it came in on carries the id"),
"{err}"
);
}
// A state that grants a group makes an account for someone, so the
@@ -1384,7 +1468,9 @@ alternatives:
)
.unwrap();
let asks = question_with_record(" - { name: name }\n - { name: email, type: email }");
let asks = question_with_record(
" - { name: name }\n - { name: email, type: email }",
);
validate_questions(&asks, &granting).unwrap();
let no_name = question_with_record(" - { name: email, type: email }");
@@ -1407,7 +1493,10 @@ alternatives:
let undeclared = crate::aggregates::parse_aggregates_yaml(
"aggregates:\n - bucket: hearings\n initial: open\n states:\n open: { event: opened, after: { days: 42, to: no_answer } }\n no_answer: { event: gave_up }\n transitions: {}\n",
);
assert!(undeclared.unwrap_err().to_string().contains("not a declared transition"));
assert!(undeclared
.unwrap_err()
.to_string()
.contains("not a declared transition"));
}
#[test]
@@ -1458,7 +1547,10 @@ alternatives:
" bind:\n field: other\n param: p\n resource:\n source: { kind: url, url: \"https://x.example/{p}\" }",
);
let err = validate_questions(&bound, &schema_fixture()).unwrap_err();
assert!(err.to_string().contains("the visitor puts the pin"), "{err}");
assert!(
err.to_string().contains("the visitor puts the pin"),
"{err}"
);
}
#[test]
@@ -1468,7 +1560,13 @@ alternatives:
let no_zoom = question_with_location(" value: \"59.8938,11.0332\"");
validate_questions(&no_zoom, &schema_fixture()).unwrap();
for bad in ["\"59.8938\"", "\"91,11\"", "\"59,200\"", "\"here\"", "\"59,11,15,3\""] {
for bad in [
"\"59.8938\"",
"\"91,11\"",
"\"59,200\"",
"\"here\"",
"\"59,11,15,3\"",
] {
let questions = question_with_location(&format!(" value: {bad}"));
let err = validate_questions(&questions, &schema_fixture()).unwrap_err();
assert!(err.to_string().contains("opens the map"), "{bad}: {err}");
@@ -1477,7 +1575,12 @@ alternatives:
#[test]
fn a_page_about_a_place_says_where_and_how_close() {
let place = Place { lat: 59.8938, lon: 11.0332, zoom: 15.0, name: Some("Tomter".into()) };
let place = Place {
lat: 59.8938,
lon: 11.0332,
zoom: 15.0,
name: Some("Tomter".into()),
};
place.validate().unwrap();
assert_eq!(place.as_value(), "59.8938,11.0332,15");
// The default is a street and its neighbours, not the globe.
@@ -1486,10 +1589,30 @@ alternatives:
assert!(bare.name.is_none());
for bad in [
Place { lat: 91.0, lon: 11.0, zoom: 15.0, name: None },
Place { lat: 59.0, lon: 181.0, zoom: 15.0, name: None },
Place { lat: 59.0, lon: 11.0, zoom: 30.0, name: None },
Place { lat: 59.0, lon: 11.0, zoom: 15.0, name: Some(" ".into()) },
Place {
lat: 91.0,
lon: 11.0,
zoom: 15.0,
name: None,
},
Place {
lat: 59.0,
lon: 181.0,
zoom: 15.0,
name: None,
},
Place {
lat: 59.0,
lon: 11.0,
zoom: 30.0,
name: None,
},
Place {
lat: 59.0,
lon: 11.0,
zoom: 15.0,
name: Some(" ".into()),
},
] {
assert!(bad.validate().is_err(), "{bad:?}");
}
@@ -1497,7 +1620,8 @@ alternatives:
#[test]
fn undeclared_from_state_is_rejected() {
let questions = question_with_transitions(" - { from: bogus, to: middle, label: X }");
let questions =
question_with_transitions(" - { from: bogus, to: middle, label: X }");
let err = validate_questions(&questions, &schema_fixture()).unwrap_err();
assert!(err.to_string().contains("transition.from"));
}
@@ -1506,19 +1630,23 @@ alternatives:
fn undeclared_edge_is_rejected() {
// Both states exist, but open -> done skips a step the graph
// never declared.
let questions = question_with_transitions(" - { from: open, to: done, label: Skip }");
let questions =
question_with_transitions(" - { from: open, to: done, label: Skip }");
let err = validate_questions(&questions, &schema_fixture()).unwrap_err();
assert!(err.to_string().contains("not a declared edge"));
}
#[test]
fn dangling_action_is_rejected() {
let question: Question =
serde_yaml::from_str("id: /a\nname: A\nalternatives:\n - name: Go\n action: /nowhere\n")
let question: Question = serde_yaml::from_str(
"id: /a\nname: A\nalternatives:\n - name: Go\n action: /nowhere\n",
)
.unwrap();
let questions = std::collections::HashMap::from([(question.id.clone(), question)]);
let err = validate_questions(&questions, &Default::default()).unwrap_err();
assert!(err.to_string().contains("does not match any declared question id"));
assert!(err
.to_string()
.contains("does not match any declared question id"));
}
#[test]
@@ -1606,7 +1734,9 @@ alternatives:
" - { name: email, type: email, relay: \"wss://relay.redoal.com\" }",
);
let err = validate_questions(&questions, &Default::default()).unwrap_err();
assert!(err.to_string().contains("relay only makes sense on type: gesture"));
assert!(err
.to_string()
.contains("relay only makes sense on type: gesture"));
}
#[test]
@@ -1645,7 +1775,8 @@ alternatives:
let absolute: SiteConfig =
serde_yaml::from_str("hero: { kind: module, module: https://x/y.js }").unwrap();
assert!(absolute.validate().is_err());
let ok: SiteConfig = serde_yaml::from_str("hero: { kind: module, module: hero.js }").unwrap();
let ok: SiteConfig =
serde_yaml::from_str("hero: { kind: module, module: hero.js }").unwrap();
assert!(ok.validate().is_ok());
}
@@ -1665,7 +1796,10 @@ alternatives:
assert_eq!(route_from_path("index.yaml"), "/");
assert_eq!(route_from_path("applied.yaml"), "/applied");
assert_eq!(route_from_path("develop/index.yaml"), "/develop");
assert_eq!(route_from_path("develop/proposal.yaml"), "/develop/proposal");
assert_eq!(
route_from_path("develop/proposal.yaml"),
"/develop/proposal"
);
assert_eq!(route_from_path("review/[record].yaml"), "/review/[record]");
}
@@ -1680,8 +1814,10 @@ alternatives:
#[test]
fn dynamic_patterns_capture_segments() {
assert_eq!(path_matches("/review/[record]", "/review/abc"),
Some(vec![("record".into(), "abc".into())]));
assert_eq!(
path_matches("/review/[record]", "/review/abc"),
Some(vec![("record".into(), "abc".into())])
);
assert_eq!(path_matches("/review/[record]", "/review"), None);
assert_eq!(path_matches("/review/[record]", "/other/abc"), None);
assert_eq!(path_matches("/review", "/review"), Some(vec![]));
@@ -1690,25 +1826,55 @@ alternatives:
#[test]
fn tree_becomes_router_with_inference_and_sections() {
let files = vec![
("index.yaml".to_string(), "name: Home\nalternatives:\n - name: go\n action: applied\n".to_string()),
("applied.yaml".to_string(), "name: Applied\nfollowup: true\n".to_string()),
("develop/index.yaml".to_string(), "name: Develop\nalternatives:\n - name: propose\n action: proposal\n".to_string()),
("develop/proposal.yaml".to_string(), "name: Proposal\nalternatives:\n - name: send\n action: proposed\n".to_string()),
("develop/proposed.yaml".to_string(), "name: Proposed\n".to_string()),
("review/_section.yaml".to_string(), "qualifies: portal_owners\n".to_string()),
("review/index.yaml".to_string(), "name: Review\n".to_string()),
("review/_draft.yaml".to_string(), "not even valid yaml: [\n".to_string()),
(
"index.yaml".to_string(),
"name: Home\nalternatives:\n - name: go\n action: applied\n".to_string(),
),
(
"applied.yaml".to_string(),
"name: Applied\nfollowup: true\n".to_string(),
),
(
"develop/index.yaml".to_string(),
"name: Develop\nalternatives:\n - name: propose\n action: proposal\n"
.to_string(),
),
(
"develop/proposal.yaml".to_string(),
"name: Proposal\nalternatives:\n - name: send\n action: proposed\n".to_string(),
),
(
"develop/proposed.yaml".to_string(),
"name: Proposed\n".to_string(),
),
(
"review/_section.yaml".to_string(),
"qualifies: portal_owners\n".to_string(),
),
(
"review/index.yaml".to_string(),
"name: Review\n".to_string(),
),
(
"review/_draft.yaml".to_string(),
"not even valid yaml: [\n".to_string(),
),
];
let questions = build_questions(&files).unwrap();
// Derived ids and resolved relative actions.
assert_eq!(questions["/"].alternatives[0].action.as_deref(), Some("/applied"));
assert_eq!(
questions["/"].alternatives[0].action.as_deref(),
Some("/applied")
);
assert_eq!(
questions["/develop"].alternatives[0].action.as_deref(),
Some("/develop/proposal")
);
assert_eq!(
questions["/develop/proposal"].alternatives[0].action.as_deref(),
questions["/develop/proposal"].alternatives[0]
.action
.as_deref(),
Some("/develop/proposed")
);
@@ -1720,7 +1886,10 @@ alternatives:
assert!(questions["/develop/proposal"].is_followup());
// Section inheritance gates the directory; drafts are skipped.
assert_eq!(questions["/review"].qualifies.as_deref(), Some("portal_owners"));
assert_eq!(
questions["/review"].qualifies.as_deref(),
Some("portal_owners")
);
assert_eq!(questions["/"].qualifies, None);
assert!(!questions.contains_key("/review/_draft"));
}
@@ -1751,7 +1920,10 @@ alternatives:
};
// Templated actions resolve to the dynamic page's pattern.
assert_eq!(action_target_id(&questions, "/shape/{curve.key}").as_deref(), Some("/shape/[key]"));
assert_eq!(
action_target_id(&questions, "/shape/{curve.key}").as_deref(),
Some("/shape/[key]")
);
assert_eq!(action_target_id(&questions, "/nowhere"), None);
// No chain, no followups: standing on the front door earns none
@@ -1762,9 +1934,18 @@ alternatives:
assert_eq!(earned("/applied", &[]), set(&[]));
// A verified answer earns exactly where it sent the visitor.
assert_eq!(unlocked_by(&questions, "/", Some("apply")), vec!["/applied"]);
assert_eq!(unlocked_by(&questions, "/", Some("draw")), vec!["/shape/[key]"]);
assert_eq!(unlocked_by(&questions, "/kontakt", Some("send")), vec!["/takk"]);
assert_eq!(
unlocked_by(&questions, "/", Some("apply")),
vec!["/applied"]
);
assert_eq!(
unlocked_by(&questions, "/", Some("draw")),
vec!["/shape/[key]"]
);
assert_eq!(
unlocked_by(&questions, "/kontakt", Some("send")),
vec!["/takk"]
);
assert_eq!(earned("/applied", &["/applied"]), set(&["/applied"]));
// An old node without the alternative: the answered question's
// recording alternatives, and nothing beyond that question.
@@ -1776,7 +1957,10 @@ alternatives:
// "watch" records an answer, so /subscribed is not earned by
// having drawn a shape; "back" leads to the front door, which
// is never a followup and ends the walk.
assert_eq!(earned("/shape/abc", &["/shape/[key]"]), set(&["/shape/[key]"]));
assert_eq!(
earned("/shape/abc", &["/shape/[key]"]),
set(&["/shape/[key]"])
);
assert_eq!(earned("/takk", &["/takk"]), set(&["/takk"]));
// The email-link page is earned by nothing on the site.
assert!(!earned("/", &["/applied"]).contains("/watching"));
@@ -1817,7 +2001,12 @@ alternatives:
let page = resolve_question(&questions, "/review/abc123").unwrap();
assert_eq!(page.id, "/review/abc123");
assert_eq!(
page.alternatives[0].features[0].resource.as_ref().unwrap().key.as_deref(),
page.alternatives[0].features[0]
.resource
.as_ref()
.unwrap()
.key
.as_deref(),
Some("abc123")
);
assert!(resolve_question(&questions, "/review").is_none());
@@ -1835,7 +2024,10 @@ alternatives:
),
];
let questions = build_questions(&files).unwrap();
assert_eq!(questions["/shaped"].requires_chain.as_deref(), Some("/shape"));
assert_eq!(
questions["/shaped"].requires_chain.as_deref(),
Some("/shape")
);
assert!(validate_questions(&questions, &Default::default()).is_ok());
let dangling = vec![(
@@ -1850,7 +2042,8 @@ alternatives:
fn recorded_buckets_must_be_attended() {
let write_only = vec![(
"index.yaml".to_string(),
"name: Home\nalternatives:\n - name: send\n action: /\n record_as: black_hole\n".to_string(),
"name: Home\nalternatives:\n - name: send\n action: /\n record_as: black_hole\n"
.to_string(),
)];
let questions = build_questions(&write_only).unwrap();
let err = validate_questions(&questions, &Default::default()).unwrap_err();
@@ -1885,9 +2078,13 @@ alternatives:
let files = vec![
(
"index.yaml".to_string(),
"name: Home\nalternatives:\n - name: go\n action: /review/[record]\n".to_string(),
"name: Home\nalternatives:\n - name: go\n action: /review/[record]\n"
.to_string(),
),
(
"review/[record].yaml".to_string(),
"name: Record\n".to_string(),
),
("review/[record].yaml".to_string(), "name: Record\n".to_string()),
];
let questions = build_questions(&files).unwrap();
assert!(validate_questions(&questions, &Default::default()).is_err());
@@ -1930,7 +2127,9 @@ alternatives:
.unwrap();
let mut qs = std::collections::HashMap::new();
qs.insert(bad.id.clone(), bad);
let err = validate_questions(&qs, &Default::default()).unwrap_err().to_string();
let err = validate_questions(&qs, &Default::default())
.unwrap_err()
.to_string();
assert!(err.contains("bad starts"), "{err}");
let dynamic: Question = serde_yaml::from_str(
@@ -1939,7 +2138,9 @@ alternatives:
.unwrap();
let mut qs = std::collections::HashMap::new();
qs.insert(dynamic.id.clone(), dynamic);
let err = validate_questions(&qs, &Default::default()).unwrap_err().to_string();
let err = validate_questions(&qs, &Default::default())
.unwrap_err()
.to_string();
assert!(err.contains("dynamic page can't carry an event"), "{err}");
}
@@ -1948,14 +2149,20 @@ alternatives:
let merged = with_builtin_aggregates(Default::default());
let schema = &merged[EVENTS_BUCKET];
assert_eq!(schema.initial, "announced");
assert_eq!(schema.allowed("announced"), ["awaiting_summary", "summarized"]);
assert_eq!(
schema.allowed("announced"),
["awaiting_summary", "summarized"]
);
assert_eq!(schema.allowed("awaiting_summary"), ["summarized"]);
let content = crate::aggregates::parse_aggregates_yaml(
"aggregates:\n - bucket: portal_events\n initial: mine\n states:\n mine: { event: mine }\n",
)
.unwrap();
assert_eq!(with_builtin_aggregates(content)[EVENTS_BUCKET].initial, "mine");
assert_eq!(
with_builtin_aggregates(content)[EVENTS_BUCKET].initial,
"mine"
);
}
#[test]
@@ -1981,20 +2188,39 @@ alternatives:
fn a_signed_in_person_is_not_asked_who_they_are() {
let yaml = "id: /oppslag/[key]\nname: P\nalternatives:\n - name: A\n features:\n - name: F\n requirements:\n - name: email\n type: email\n value: \"{user.email}\"\n - name: navn\n value: \"{user.name}\"\n - name: hva\n type: textarea\n value: \"{user.email} says\"\n";
let q: Question = serde_yaml::from_str(yaml).unwrap();
let user = crate::auth::User { sub: "s".into(), username: "kari".into(), display_name: "Kari N".into(), groups: vec![], phone: None, email: Some("kari@vel.no".into()) };
let user = crate::auth::User {
sub: "s".into(),
username: "kari".into(),
display_name: "Kari N".into(),
groups: vec![],
phone: None,
email: Some("kari@vel.no".into()),
};
let mut signed_in = q.clone();
fill_from_user(&mut signed_in, Some(&user));
let reqs = &signed_in.alternatives[0].features[0].requirements;
assert_eq!((reqs[0].kind.as_str(), reqs[0].value.as_deref()), ("hidden", Some("kari@vel.no")));
assert_eq!((reqs[1].kind.as_str(), reqs[1].value.as_deref()), ("hidden", Some("Kari N")));
assert_eq!(
(reqs[0].kind.as_str(), reqs[0].value.as_deref()),
("hidden", Some("kari@vel.no"))
);
assert_eq!(
(reqs[1].kind.as_str(), reqs[1].value.as_deref()),
("hidden", Some("Kari N"))
);
// Mixed text is not a user field, and is left as written.
assert_eq!(reqs[2].value.as_deref(), Some("{user.email} says"));
// A visitor is asked: the placeholder goes, the field stays.
let mut visitor = q.clone();
fill_from_user(&mut visitor, None);
let reqs = &visitor.alternatives[0].features[0].requirements;
assert_eq!((reqs[0].kind.as_str(), reqs[0].value.as_deref()), ("email", None));
assert_eq!((reqs[1].kind.as_str(), reqs[1].value.as_deref()), ("text", None));
assert_eq!(
(reqs[0].kind.as_str(), reqs[0].value.as_deref()),
("email", None)
);
assert_eq!(
(reqs[1].kind.as_str(), reqs[1].value.as_deref()),
("text", None)
);
}
#[test]
@@ -2040,8 +2266,14 @@ alternatives:
render_inline_markdown("![J](https://x.no/j.jpg \"wat\")"),
"<img src=\"https://x.no/j.jpg\" alt=\"J\" />"
);
assert_eq!(render_inline_markdown("[ok](/shape)"), "<a href=\"/shape\">ok</a>");
assert_eq!(render_inline_markdown("[mail](mailto:bl@uhhm.no)"), "<a href=\"mailto:bl@uhhm.no\">mail</a>");
assert_eq!(
render_inline_markdown("[ok](/shape)"),
"<a href=\"/shape\">ok</a>"
);
assert_eq!(
render_inline_markdown("[mail](mailto:bl@uhhm.no)"),
"<a href=\"mailto:bl@uhhm.no\">mail</a>"
);
}
#[test]
@@ -2063,7 +2295,10 @@ alternatives:
#[test]
fn action_templates_from_responses() {
let r = serde_json::json!({"curve": {"key": "20aa98", "points": [[0,0]]}, "n": 3});
assert_eq!(template_action("/shape/{curve.key}", &r).as_deref(), Some("/shape/20aa98"));
assert_eq!(
template_action("/shape/{curve.key}", &r).as_deref(),
Some("/shape/20aa98")
);
assert_eq!(template_action("/x/{n}/y", &r).as_deref(), Some("/x/3/y"));
assert_eq!(template_action("/shape", &r).as_deref(), Some("/shape"));
assert!(template_action("/shape/{curve.missing}", &r).is_none());
@@ -2081,7 +2316,12 @@ alternatives:
let mut qs = std::collections::HashMap::new();
qs.insert(q.id.clone(), q);
let page = resolve_question(&qs, "/shape/20aa98").unwrap();
assert_eq!(page.alternatives[0].features[0].requirements[0].value.as_deref(), Some("20aa98"));
assert_eq!(
page.alternatives[0].features[0].requirements[0]
.value
.as_deref(),
Some("20aa98")
);
let q: Question = serde_yaml::from_str(
"id: /shape/[key]\nname: P\nalternatives:\n - name: A\n features:\n - name: F\n resource:\n source: { kind: url, url: \"https://relay.example/place/{key}\" }\n public: true\n",
@@ -2090,7 +2330,12 @@ alternatives:
let mut qs = std::collections::HashMap::new();
qs.insert(q.id.clone(), q);
let page = resolve_question(&qs, "/shape/20aa98").unwrap();
match &page.alternatives[0].features[0].resource.as_ref().unwrap().source {
match &page.alternatives[0].features[0]
.resource
.as_ref()
.unwrap()
.source
{
ResourceSource::Url { url } => assert_eq!(url, "https://relay.example/place/20aa98"),
_ => panic!(),
}
+74 -20
View File
@@ -107,12 +107,18 @@ pub fn with_builtin_aggregates(
/// runtime's own `portal_events` (a warning, not a failure: a repo
/// with no event pages has nothing to attend).
#[cfg(feature = "ssr")]
pub fn bucket_is_read(questions: &std::collections::HashMap<String, Question>, bucket: &str) -> bool {
pub fn bucket_is_read(
questions: &std::collections::HashMap<String, Question>,
bucket: &str,
) -> bool {
questions.values().any(|q| {
q.alternatives.iter().any(|a| {
a.features
.iter()
.any(|f| f.resource.as_ref().and_then(|r| r.bucket()).is_some_and(|b| b == bucket))
a.features.iter().any(|f| {
f.resource
.as_ref()
.and_then(|r| r.bucket())
.is_some_and(|b| b == bucket)
})
})
})
}
@@ -125,12 +131,21 @@ pub fn validate_questions(
// A mail to the submitter needs an address to go to: every form
// that records into that bucket has to collect a required email.
for (bucket, schema) in aggregates {
let submitter_states: Vec<&String> = schema.mail_for_state.iter().filter(|(_, m)| m.to_submitter()).map(|(s, _)| s).collect();
let submitter_states: Vec<&String> = schema
.mail_for_state
.iter()
.filter(|(_, m)| m.to_submitter())
.map(|(s, _)| s)
.collect();
if submitter_states.is_empty() {
continue;
}
for question in questions.values() {
for alternative in question.alternatives.iter().filter(|a| a.record_as.as_deref() == Some(bucket.as_str())) {
for alternative in question
.alternatives
.iter()
.filter(|a| a.record_as.as_deref() == Some(bucket.as_str()))
{
let collects_email = alternative
.features
.iter()
@@ -154,10 +169,20 @@ pub fn validate_questions(
anyhow::bail!("question {:?}: slideshow has no slides", question.id);
}
if !(20..=100).contains(&show.max_height) {
anyhow::bail!("question {:?}: slideshow max_height is {} svh; between 20 and 100", question.id, show.max_height);
anyhow::bail!(
"question {:?}: slideshow max_height is {} svh; between 20 and 100",
question.id,
show.max_height
);
}
for slide in &show.slides {
let ok = slide.image.starts_with("https://") || crate::content::is_safe_site_path(slide.image.trim_start_matches("/site/").trim_start_matches('/'));
let ok = slide.image.starts_with("https://")
|| crate::content::is_safe_site_path(
slide
.image
.trim_start_matches("/site/")
.trim_start_matches('/'),
);
if !ok {
anyhow::bail!("question {:?}: slideshow image {:?} is neither an https address nor a path in the content repo", question.id, slide.image);
}
@@ -171,10 +196,20 @@ pub fn validate_questions(
// has nowhere to go.
for (bucket, schema) in aggregates {
for (state, spec) in &schema.mail_for_state {
let Some(parent) = crate::mail::parent_of(&spec.to) else { continue };
let Some(parent) = crate::mail::parent_of(&spec.to) else {
continue;
};
for question in questions.values() {
for alternative in question.alternatives.iter().filter(|a| a.record_as.as_deref() == Some(bucket.as_str())) {
let has = alternative.features.iter().flat_map(|f| &f.requirements).any(|r| r.name == parent && r.kind == "record" && r.of.is_some());
for alternative in question
.alternatives
.iter()
.filter(|a| a.record_as.as_deref() == Some(bucket.as_str()))
{
let has = alternative
.features
.iter()
.flat_map(|f| &f.requirements)
.any(|r| r.name == parent && r.kind == "record" && r.of.is_some());
if !has {
anyhow::bail!(
"question {:?} alternative {:?}: records into {bucket:?}, whose state {state:?} mails {:?}, so it must carry a `type: record` field called {parent:?} with `of`",
@@ -199,7 +234,11 @@ pub fn validate_questions(
}
granting.sort();
for question in questions.values() {
for alternative in question.alternatives.iter().filter(|a| a.record_as.as_deref() == Some(bucket.as_str())) {
for alternative in question
.alternatives
.iter()
.filter(|a| a.record_as.as_deref() == Some(bucket.as_str()))
{
let required = |name: &str, kind: &str| {
alternative
.features
@@ -220,7 +259,10 @@ pub fn validate_questions(
for question in questions.values() {
if let Some(event) = &question.event {
if question.is_dynamic() {
anyhow::bail!("question {:?}: a dynamic page can't carry an event", question.id);
anyhow::bail!(
"question {:?}: a dynamic page can't carry an event",
question.id
);
}
if let Err(e) = event.window_ms() {
anyhow::bail!("question {:?}: event: {e}", question.id);
@@ -361,10 +403,12 @@ pub fn validate_questions(
question.id, alternative.name, feature.name, requirement.name, requirement.kind
);
}
validate_relay_url(relay).map_err(|e| anyhow::anyhow!(
validate_relay_url(relay).map_err(|e| {
anyhow::anyhow!(
"question {:?} alternative {:?} feature {:?}: requirement {:?}: {e}",
question.id, alternative.name, feature.name, requirement.name
))?;
)
})?;
}
if requirement.kind == "gesture"
&& (requirement.resource.is_some() || requirement.bind.is_some())
@@ -388,8 +432,12 @@ pub fn validate_questions(
let parts: Vec<&str> = value.split(',').map(str::trim).collect();
let ok = (2..=3).contains(&parts.len())
&& parts.iter().all(|p| p.parse::<f64>().is_ok())
&& parts[0].parse::<f64>().is_ok_and(|lat| (-90.0..=90.0).contains(&lat))
&& parts[1].parse::<f64>().is_ok_and(|lon| (-180.0..=180.0).contains(&lon));
&& parts[0]
.parse::<f64>()
.is_ok_and(|lat| (-90.0..=90.0).contains(&lat))
&& parts[1]
.parse::<f64>()
.is_ok_and(|lon| (-180.0..=180.0).contains(&lon));
if !ok {
anyhow::bail!(
"question {:?} alternative {:?} feature {:?}: requirement {:?} is type: location - its `value` opens the map and must be `lat,lon` or `lat,lon,zoom`, not {value:?}",
@@ -468,7 +516,11 @@ pub fn validate_questions(
question.id, alternative.name, feature.name, transition.to, bucket
);
}
if !schema.allowed(&transition.from).iter().any(|s| s == &transition.to) {
if !schema
.allowed(&transition.from)
.iter()
.any(|s| s == &transition.to)
{
anyhow::bail!(
"question {:?} alternative {:?} feature {:?}: transition {:?} -> {:?} is not a declared edge for bucket {:?}",
question.id, alternative.name, feature.name, transition.from, transition.to, bucket
@@ -515,7 +567,10 @@ pub fn validate_questions(
// A form is sent with its button, and the button is only
// drawn for an alternative that says where to go afterwards:
// without `action` the person fills it in and cannot send it.
let asks = alternative.features.iter().any(|f| f.requirements.iter().any(|r| r.kind != "hidden"));
let asks = alternative
.features
.iter()
.any(|f| f.requirements.iter().any(|r| r.kind != "hidden"));
if (asks || alternative.invite.is_some() || alternative.record_as.is_some())
&& alternative.action.is_none()
&& alternative.self_transition.is_none()
@@ -549,4 +604,3 @@ pub fn validate_questions(
}
Ok(())
}
+83 -17
View File
@@ -31,7 +31,16 @@
use serde_yaml::Value;
/// Keys whose value is one string to a person.
const TEXT: &[&str] = &["name", "description", "label", "placeholder", "subject", "body", "empty", "title"];
const TEXT: &[&str] = &[
"name",
"description",
"label",
"placeholder",
"subject",
"body",
"empty",
"title",
];
/// Keys whose value is a list of strings to a person; their storylines
/// are a list of lists.
const TEXT_LIST: &[&str] = &["consequence", "encouragements"];
@@ -41,7 +50,12 @@ const TEXT_LIST: &[&str] = &["consequence", "encouragements"];
fn is_identifier(path: &[String], key: &str) -> bool {
// `path` ends with the key itself, then the item's index, then the
// list it is in.
key == "name" && path.iter().rev().nth(2).is_some_and(|p| p == "requirements")
key == "name"
&& path
.iter()
.rev()
.nth(2)
.is_some_and(|p| p == "requirements")
}
fn is_strings(seq: &[Value]) -> bool {
@@ -49,7 +63,11 @@ fn is_strings(seq: &[Value]) -> bool {
}
fn is_lists_of_strings(seq: &[Value]) -> bool {
!seq.is_empty() && seq.iter().all(|v| v.as_sequence().is_some_and(|s| s.iter().all(|x| x.is_string())))
!seq.is_empty()
&& seq.iter().all(|v| {
v.as_sequence()
.is_some_and(|s| s.iter().all(|x| x.is_string()))
})
}
/// The storyline counts found in a document, with where each was.
@@ -60,8 +78,18 @@ fn counts(value: &Value, path: &mut Vec<String>, out: &mut Vec<(String, usize)>)
let key = k.as_str().unwrap_or_default().to_string();
path.push(key.clone());
let variants = match v {
Value::Sequence(seq) if TEXT.contains(&key.as_str()) && !is_identifier(path, &key) && is_strings(seq) => Some(seq.len()),
Value::Sequence(seq) if TEXT_LIST.contains(&key.as_str()) && is_lists_of_strings(seq) => Some(seq.len()),
Value::Sequence(seq)
if TEXT.contains(&key.as_str())
&& !is_identifier(path, &key)
&& is_strings(seq) =>
{
Some(seq.len())
}
Value::Sequence(seq)
if TEXT_LIST.contains(&key.as_str()) && is_lists_of_strings(seq) =>
{
Some(seq.len())
}
_ => None,
};
match variants {
@@ -90,8 +118,18 @@ fn pick(value: &mut Value, path: &mut Vec<String>, n: usize) {
let key = k.as_str().unwrap_or_default().to_string();
path.push(key.clone());
let chosen = match v {
Value::Sequence(seq) if TEXT.contains(&key.as_str()) && !is_identifier(path, &key) && is_strings(seq) => Some(seq[n.min(seq.len() - 1)].clone()),
Value::Sequence(seq) if TEXT_LIST.contains(&key.as_str()) && is_lists_of_strings(seq) => Some(seq[n.min(seq.len() - 1)].clone()),
Value::Sequence(seq)
if TEXT.contains(&key.as_str())
&& !is_identifier(path, &key)
&& is_strings(seq) =>
{
Some(seq[n.min(seq.len() - 1)].clone())
}
Value::Sequence(seq)
if TEXT_LIST.contains(&key.as_str()) && is_lists_of_strings(seq) =>
{
Some(seq[n.min(seq.len() - 1)].clone())
}
_ => None,
};
match chosen {
@@ -118,7 +156,9 @@ fn pick(value: &mut Value, path: &mut Vec<String>, n: usize) {
pub fn arity(files: &[(String, String)]) -> anyhow::Result<usize> {
let mut found: Vec<(String, String, usize)> = Vec::new();
for (rel, raw) in files {
let Ok(value) = serde_yaml::from_str::<Value>(raw) else { continue };
let Ok(value) = serde_yaml::from_str::<Value>(raw) else {
continue;
};
let mut out = Vec::new();
counts(&value, &mut Vec::new(), &mut out);
found.extend(out.into_iter().map(|(p, n)| (rel.clone(), p, n)));
@@ -135,14 +175,19 @@ pub fn storyline(files: &[(String, String)], n: usize) -> Vec<(String, String)>
files
.iter()
.map(|(rel, raw)| {
let Ok(mut value) = serde_yaml::from_str::<Value>(raw) else { return (rel.clone(), raw.clone()) };
let Ok(mut value) = serde_yaml::from_str::<Value>(raw) else {
return (rel.clone(), raw.clone());
};
let mut out = Vec::new();
counts(&value, &mut Vec::new(), &mut out);
if out.is_empty() {
return (rel.clone(), raw.clone());
}
pick(&mut value, &mut Vec::new(), n);
(rel.clone(), serde_yaml::to_string(&value).unwrap_or_else(|_| raw.clone()))
(
rel.clone(),
serde_yaml::to_string(&value).unwrap_or_else(|_| raw.clone()),
)
})
.collect()
}
@@ -192,17 +237,32 @@ mod tests {
assert_eq!(v["name"], "Trenger naboene deg?");
assert_eq!(v["description"], "Alle trenger flere hender.");
assert_eq!(v["alternatives"][0]["name"], "Meld deg");
assert_eq!(v["alternatives"][0]["consequence"], serde_yaml::to_value(vec!["Meld meg"]).unwrap());
assert_eq!(
v["alternatives"][0]["consequence"],
serde_yaml::to_value(vec!["Meld meg"]).unwrap()
);
// A requirement's name is the stored field: left as written.
assert_eq!(v["alternatives"][0]["features"][0]["requirements"][0]["name"], serde_yaml::to_value(vec!["navn"]).unwrap());
assert_eq!(v["alternatives"][0]["features"][0]["requirements"][0]["label"], "Hva heter du?");
assert_eq!(
v["alternatives"][0]["features"][0]["requirements"][0]["name"],
serde_yaml::to_value(vec!["navn"]).unwrap()
);
assert_eq!(
v["alternatives"][0]["features"][0]["requirements"][0]["label"],
"Hva heter du?"
);
// Options are values people choose from and are stored: untouched.
assert_eq!(v["alternatives"][0]["features"][0]["requirements"][1]["options"], serde_yaml::to_value(vec!["Dugnad", "Komité"]).unwrap());
assert_eq!(
v["alternatives"][0]["features"][0]["requirements"][1]["options"],
serde_yaml::to_value(vec!["Dugnad", "Komité"]).unwrap()
);
}
#[test]
fn a_site_with_no_lists_tells_one_storyline_and_a_short_list_is_refused() {
let plain = vec![("index.yaml".to_string(), "name: Hva?\nalternatives:\n - name: A\n consequence: [Send]\n".to_string())];
let plain = vec![(
"index.yaml".to_string(),
"name: Hva?\nalternatives:\n - name: A\n consequence: [Send]\n".to_string(),
)];
assert_eq!(arity(&plain).unwrap(), 1);
assert_eq!(storyline(&plain, 0)[0].1, plain[0].1);
let uneven = vec![
@@ -210,12 +270,18 @@ mod tests {
("b.yaml".to_string(), "name: [En, To, Tre]\n".to_string()),
];
let err = arity(&uneven).unwrap_err().to_string();
assert!(err.contains("b.yaml: name tells 3") && err.contains("tells 2") || err.contains("tells 3"), "{err}");
assert!(
err.contains("b.yaml: name tells 3") && err.contains("tells 2")
|| err.contains("tells 3"),
"{err}"
);
}
#[test]
fn the_first_storyline_is_what_the_machinery_reads() {
let s = Storylines { each: vec!["a", "b"] };
let s = Storylines {
each: vec!["a", "b"],
};
assert_eq!(*s, "a");
assert_eq!(*s.at(1), "b");
assert_eq!(*s.at(7), "b");
+86 -23
View File
@@ -63,8 +63,13 @@ pub fn has_left(schema: &crate::aggregates::AggregateSchema, answer: &Answer, no
pub async fn sweep_leaving(state: &AppState) -> anyhow::Result<()> {
let now_ms = chrono::Utc::now().timestamp_millis();
let aggregates = state.aggregates.load();
for schema in aggregates.values().filter(|s| !s.leaves_for_state.is_empty()) {
let Ok(store) = state.jetstream.get_key_value(&schema.bucket).await else { continue };
for schema in aggregates
.values()
.filter(|s| !s.leaves_for_state.is_empty())
{
let Ok(store) = state.jetstream.get_key_value(&schema.bucket).await else {
continue;
};
let keys: Vec<String> = {
use futures::TryStreamExt;
match store.keys().await {
@@ -73,12 +78,20 @@ pub async fn sweep_leaving(state: &AppState) -> anyhow::Result<()> {
}
};
for id in keys {
let Ok(Some(bytes)) = store.get(&id).await else { continue };
let Ok(answer) = serde_json::from_slice::<Answer>(&bytes) else { continue };
let Ok(Some(bytes)) = store.get(&id).await else {
continue;
};
let Ok(answer) = serde_json::from_slice::<Answer>(&bytes) else {
continue;
};
if has_left(schema, &answer, now_ms) {
match store.delete(&id).await {
Ok(()) => tracing::info!(bucket = %schema.bucket, id, state = %answer.state, "a record left its list"),
Err(e) => tracing::warn!(bucket = %schema.bucket, id, "a record could not leave its list: {e}"),
Ok(()) => {
tracing::info!(bucket = %schema.bucket, id, state = %answer.state, "a record left its list")
}
Err(e) => {
tracing::warn!(bucket = %schema.bucket, id, "a record could not leave its list: {e}")
}
}
}
}
@@ -93,7 +106,11 @@ fn waiting_since(answer: &Answer) -> i64 {
}
/// Whether a record in `state` has waited out its deadline by `now_ms`.
pub fn due(schema: &crate::aggregates::AggregateSchema, answer: &Answer, now_ms: i64) -> Option<String> {
pub fn due(
schema: &crate::aggregates::AggregateSchema,
answer: &Answer,
now_ms: i64,
) -> Option<String> {
let after = schema.after_for(&answer.state)?;
let deadline = waiting_since(answer) + i64::from(after.days) * DAY_MS;
(now_ms >= deadline).then(|| after.to.clone())
@@ -120,13 +137,24 @@ pub async fn sweep(state: &AppState) -> anyhow::Result<()> {
}
};
for id in keys {
let Ok(Some(bytes)) = store.get(&id).await else { continue };
let Ok(mut answer) = serde_json::from_slice::<Answer>(&bytes) else { continue };
let Some(target) = due(schema, &answer, now_ms) else { continue };
let Ok(Some(bytes)) = store.get(&id).await else {
continue;
};
let Ok(mut answer) = serde_json::from_slice::<Answer>(&bytes) else {
continue;
};
let Some(target) = due(schema, &answer, now_ms) else {
continue;
};
let payload = serde_json::json!({ "to": target, "item": id, "by": BY });
if let Err(e) = crate::aggregates::transition(
&state.jetstream, schema, &id, &target, payload.clone(), now_ms,
&state.jetstream,
schema,
&id,
&target,
payload.clone(),
now_ms,
)
.await
{
@@ -143,11 +171,20 @@ pub async fn sweep(state: &AppState) -> anyhow::Result<()> {
if let Ok(data) = serde_json::to_vec(&answer) {
let _ = store.put(&id, data.into()).await;
}
crate::mail::on_state(state, &schema.bucket, &id, &target, &answer.responses, answer.storyline).await;
crate::mail::on_state(
state,
&schema.bucket,
&id,
&target,
&answer.responses,
answer.storyline,
)
.await;
crate::people::on_state(state, &schema.bucket, &id, &target, &answer.responses).await;
let parent_hashes = vec![id.clone()];
let chain_hash = crate::chain::hash_node(&answer.question_id, &parent_hashes, &payload, now_ms);
let chain_hash =
crate::chain::hash_node(&answer.question_id, &parent_hashes, &payload, now_ms);
if let Err(e) = emit_answer_submitted(
&state.nats,
&AnswerSubmitted {
@@ -175,7 +212,11 @@ mod tests {
use super::*;
fn schema(yaml: &str) -> crate::aggregates::AggregateSchema {
crate::aggregates::parse_aggregates_yaml(yaml).unwrap().into_values().next().unwrap()
crate::aggregates::parse_aggregates_yaml(yaml)
.unwrap()
.into_values()
.next()
.unwrap()
}
const HEARINGS: &str = r#"
@@ -192,9 +233,16 @@ aggregates:
fn waiting(state: &str, since_ms: i64) -> Answer {
Answer {
id: "r".into(), question_id: "/q".into(), alternative: "a".into(),
responses: serde_json::json!({}), state: state.into(),
submitted_ms: since_ms, decided_ms: None, decided_by: None, notes: vec![], finished: false,
id: "r".into(),
question_id: "/q".into(),
alternative: "a".into(),
responses: serde_json::json!({}),
state: state.into(),
submitted_ms: since_ms,
decided_ms: None,
decided_by: None,
notes: vec![],
finished: false,
storyline: 0,
}
}
@@ -207,7 +255,10 @@ aggregates:
let t = 1_000_000_000_000;
let mut answer = waiting("gone", t - 60 * 60 * 1000);
answer.decided_ms = Some(t);
assert!(!has_left(&s, &answer, t + 9 * 60 * 1000), "measured from entering the state, not from sending");
assert!(
!has_left(&s, &answer, t + 9 * 60 * 1000),
"measured from entering the state, not from sending"
);
assert!(has_left(&s, &answer, t + 10 * 60 * 1000));
// A state without it keeps its records.
assert!(!has_left(&s, &waiting("open", 0), t));
@@ -221,7 +272,10 @@ aggregates:
let answer = waiting("open", sent);
assert_eq!(due(&s, &answer, sent), None);
assert_eq!(due(&s, &answer, sent + 41 * DAY_MS), None);
assert_eq!(due(&s, &answer, sent + 42 * DAY_MS), Some("no_answer".into()));
assert_eq!(
due(&s, &answer, sent + 42 * DAY_MS),
Some("no_answer".into())
);
}
// The six weeks run from the last move, not from the day the case
@@ -233,7 +287,10 @@ aggregates:
let mut answer = waiting("open", sent);
answer.decided_ms = Some(sent + 40 * DAY_MS);
assert_eq!(due(&s, &answer, sent + 42 * DAY_MS), None);
assert_eq!(due(&s, &answer, sent + 82 * DAY_MS), Some("no_answer".into()));
assert_eq!(
due(&s, &answer, sent + 82 * DAY_MS),
Some("no_answer".into())
);
}
#[test]
@@ -248,15 +305,21 @@ aggregates:
// not when the hour comes round.
#[test]
fn a_deadline_onto_an_undeclared_edge_is_refused_on_load() {
let bad = HEARINGS.replace("to: no_answer }", "to: answered }").replace("open: [answered, no_answer]", "open: [no_answer]");
let err = crate::aggregates::parse_aggregates_yaml(&bad).unwrap_err().to_string();
let bad = HEARINGS
.replace("to: no_answer }", "to: answered }")
.replace("open: [answered, no_answer]", "open: [no_answer]");
let err = crate::aggregates::parse_aggregates_yaml(&bad)
.unwrap_err()
.to_string();
assert!(err.contains("not a declared transition"), "{err}");
}
#[test]
fn a_deadline_of_no_days_is_refused() {
let bad = HEARINGS.replace("days: 42", "days: 0");
let err = crate::aggregates::parse_aggregates_yaml(&bad).unwrap_err().to_string();
let err = crate::aggregates::parse_aggregates_yaml(&bad)
.unwrap_err()
.to_string();
assert!(err.contains("at least one"), "{err}");
}
}
+7 -2
View File
@@ -109,11 +109,16 @@ pub async fn load_events(
.await
.map_err(|e| anyhow::anyhow!("fetching messages: {e}"))?
{
let info = message.info().map_err(|e| anyhow::anyhow!("reading message info: {e}"))?;
let info = message
.info()
.map_err(|e| anyhow::anyhow!("reading message info: {e}"))?;
let mut event: Event = serde_json::from_slice(&message.payload)?;
event.sequence = info.stream_sequence;
events.push(event);
message.ack().await.map_err(|e| anyhow::anyhow!("acking: {e}"))?;
message
.ack()
.await
.map_err(|e| anyhow::anyhow!("acking: {e}"))?;
}
events.sort_by_key(|e| e.sequence);
+9 -6
View File
@@ -31,15 +31,18 @@ pub mod people;
#[cfg(feature = "ssr")]
pub mod mail;
#[cfg(feature = "ssr")]
pub mod seed;
#[cfg(feature = "ssr")]
pub mod memberships;
#[cfg(feature = "ssr")]
pub mod report;
/// The lint: `portal lint`, what a content repo's CI runs (once uhhm/iris).
#[cfg(feature = "ssr")]
pub mod lint;
#[cfg(feature = "ssr")]
pub mod memberships;
/// The Apple configuration profile at `/profile.mobileconfig`.
#[cfg(feature = "ssr")]
pub mod profile;
#[cfg(feature = "ssr")]
pub mod report;
#[cfg(feature = "ssr")]
pub mod seed;
/// The trainer: `portal plan`, `portal build`, `portal report` and the
/// rest (once uhhm/portal-trainer).
#[cfg(feature = "ssr")]
+64 -15
View File
@@ -8,7 +8,9 @@
//! than compiled there - keeps that repo's CI coupling to "run a
//! static binary," not "build a Rust workspace."
use crate::lint::local::{load_aggregates_from_dir, load_from_dir, load_needs_from_dir, load_site_from_dir};
use crate::lint::local::{
load_aggregates_from_dir, load_from_dir, load_needs_from_dir, load_site_from_dir,
};
use crate::lint::needs;
use crate::{aggregates, content};
@@ -39,7 +41,10 @@ pub async fn run(argv: Vec<String>) -> anyhow::Result<()> {
"--findings" => findings_out = args.next(),
"--needs-score" => answers_in = args.next(),
"--min-accuracy" => {
min_accuracy = args.next().and_then(|v| v.parse().ok()).unwrap_or(min_accuracy)
min_accuracy = args
.next()
.and_then(|v| v.parse().ok())
.unwrap_or(min_accuracy)
}
other => {
eprintln!("unknown argument: {other}");
@@ -58,13 +63,30 @@ pub async fn run(argv: Vec<String>) -> anyhow::Result<()> {
} else if let Some(repo_url) = repo {
// The first storyline of each, as the checks read it; the
// loaders have held every list of texts to the site's arity.
let questions = content::load_questions_from_gitea(&repo_url, &branch, &subdir).await?.at(0).clone();
let aggregates_map = content::load_aggregates_from_gitea(&repo_url, &branch).await?.at(0).clone();
let questions = content::load_questions_from_gitea(&repo_url, &branch, &subdir)
.await?
.at(0)
.clone();
let aggregates_map = content::load_aggregates_from_gitea(&repo_url, &branch)
.await?
.at(0)
.clone();
// load_site_from_gitea already validates; a missing file is
// the default config, same as at portal boot.
let site = content::load_site_from_gitea(&repo_url, &branch).await?.at(0).clone();
let site = content::load_site_from_gitea(&repo_url, &branch)
.await?
.at(0)
.clone();
// Optional: a fetch failure means the repo declares no needs.
let needs_raw = crate::http::get_text(&format!("{}/needs.yaml?ref={branch}", content::gitea_raw_base(&repo_url)?), crate::http::GITEA_CAP).await.ok();
let needs_raw = crate::http::get_text(
&format!(
"{}/needs.yaml?ref={branch}",
content::gitea_raw_base(&repo_url)?
),
crate::http::GITEA_CAP,
)
.await
.ok();
(questions, aggregates_map, site, needs_raw)
} else {
eprintln!(
@@ -106,13 +128,24 @@ pub async fn run(argv: Vec<String>) -> anyhow::Result<()> {
eprintln!("FAIL: access policy does not validate: {e}");
std::process::exit(1);
}
println!("OK: access policy, {} rule(s), validates", policy.rules.len());
println!(
"OK: access policy, {} rule(s), validates",
policy.rules.len()
);
if show_access {
println!("\n{:<16} {:<44} {:<40} {}", "ACTION", "RESOURCE", "WHO", "WHEN");
println!(
"\n{:<16} {:<44} {:<40} {}",
"ACTION", "RESOURCE", "WHO", "WHEN"
);
let mut rows = policy.rules.clone();
rows.sort_by(|a, b| (&a.resource, &a.action, &a.who).cmp(&(&b.resource, &b.action, &b.who)));
rows.sort_by(|a, b| {
(&a.resource, &a.action, &a.who).cmp(&(&b.resource, &b.action, &b.who))
});
for r in rows {
println!("{:<16} {:<44} {:<40} {}", r.action, r.resource, r.who, r.when);
println!(
"{:<16} {:<44} {:<40} {}",
r.action, r.resource, r.who, r.when
);
}
println!();
}
@@ -164,7 +197,10 @@ fn schema_finding(message: &str) -> needs::Finding {
/// "clean" from "not run".
fn write_findings(out: Option<&str>, findings: &[needs::Finding]) {
let Some(out) = out else { return };
let lines: String = findings.iter().map(|f| format!("{}\n", f.to_json())).collect();
let lines: String = findings
.iter()
.map(|f| format!("{}\n", f.to_json()))
.collect();
if let Err(e) = std::fs::write(out, lines) {
eprintln!("WARN: could not write the findings to {out}: {e}");
}
@@ -181,7 +217,10 @@ fn print_findings(findings: &[needs::Finding]) -> usize {
}
needs::Level::Warn => "WARN",
};
eprintln!("{level}: {} {:?}: {}", finding.scope, finding.need, finding.message);
eprintln!(
"{level}: {} {:?}: {}",
finding.scope, finding.need, finding.message
);
}
failed
}
@@ -225,7 +264,12 @@ fn check_needs(
.iter()
.map(|s| format!("{} [{}]", s.page, s.alternative))
.collect();
println!("need {:?}: {} -> {:?}", need.id, steps.join(" -> "), need.lands_in);
println!(
"need {:?}: {} -> {:?}",
need.id,
steps.join(" -> "),
need.lands_in
);
}
}
if failed == 0 {
@@ -259,9 +303,14 @@ fn check_needs(
println!("wrote {} wording task(s) to {out}", tasks.len());
}
if let Some(path) = answers_in {
let raw = std::fs::read_to_string(path).map_err(|e| anyhow::anyhow!("reading {path}: {e}"))?;
let raw =
std::fs::read_to_string(path).map_err(|e| anyhow::anyhow!("reading {path}: {e}"))?;
let mut answers = Vec::new();
for (n, line) in raw.lines().enumerate().filter(|(_, l)| !l.trim().is_empty()) {
for (n, line) in raw
.lines()
.enumerate()
.filter(|(_, l)| !l.trim().is_empty())
{
answers.push(
serde_json::from_str::<needs::Answer>(line)
.map_err(|e| anyhow::anyhow!("{path}:{}: {e}", n + 1))?,
+2 -2
View File
@@ -22,7 +22,8 @@ pub fn load_site_from_dir(dir: &str) -> anyhow::Result<content::SiteConfig> {
// The first storyline, held to the arity (content::variants).
let files = vec![("site.yaml".to_string(), raw)];
content::arity(&files)?;
serde_yaml::from_str(&content::storyline(&files, 0)[0].1).map_err(|e| anyhow::anyhow!("parsing {}: {e}", site_path.display()))
serde_yaml::from_str(&content::storyline(&files, 0)[0].1)
.map_err(|e| anyhow::anyhow!("parsing {}: {e}", site_path.display()))
}
/// The offline counterpart to `content::load_aggregates_from_gitea` -
@@ -102,4 +103,3 @@ pub fn load_needs_from_dir(dir: &str) -> anyhow::Result<Option<String>> {
.map(Some)
.map_err(|e| anyhow::anyhow!("reading {}: {e}", needs_path.display()))
}
-1
View File
@@ -50,4 +50,3 @@ pub async fn run(mut argv: Vec<String>) -> anyhow::Result<()> {
}
}
}
+230 -51
View File
@@ -238,7 +238,11 @@ fn viable(
if remaining == 0 {
return out;
}
for alt in page.alternatives.iter().filter(|a| include_disabled || !a.disabled) {
for alt in page
.alternatives
.iter()
.filter(|a| include_disabled || !a.disabled)
{
if entry && need.via.as_deref().is_some_and(|via| via != alt.name) {
continue;
}
@@ -268,7 +272,15 @@ fn viable(
continue;
}
trail.push(target.id.clone());
let onward = viable(questions, need, target, remaining - 1, false, include_disabled, trail);
let onward = viable(
questions,
need,
target,
remaining - 1,
false,
include_disabled,
trail,
);
trail.pop();
if let Some(best) = onward.into_iter().min_by_key(|v| (v.path.len(), v.fields)) {
let mut path = vec![step];
@@ -287,7 +299,15 @@ fn viable(
fn entry_viable(questions: &Questions, need: &Need, include_disabled: bool) -> Option<Vec<Viable>> {
let page = questions.get(&need.enters)?;
let mut trail = vec![page.id.clone()];
Some(viable(questions, need, page, need.max_steps, true, include_disabled, &mut trail))
Some(viable(
questions,
need,
page,
need.max_steps,
true,
include_disabled,
&mut trail,
))
}
/// The cheapest path that meets the need, if any.
@@ -321,7 +341,10 @@ fn desks_over(questions: &Questions, bucket: &str) -> Vec<Desk> {
{
desks.push(Desk {
page: q.id.clone(),
group: resource.requires_group.clone().or_else(|| q.qualifies.clone()),
group: resource
.requires_group
.clone()
.or_else(|| q.qualifies.clone()),
page_group: q.qualifies.clone(),
moves: resource
.transitions
@@ -344,15 +367,22 @@ fn self_moves(questions: &Questions, bucket: &str, initial: &str) -> Vec<(String
.flat_map(|q| &q.alternatives)
.filter_map(|a| a.self_transition.as_ref())
.filter(|s| s.bucket == bucket)
.flat_map(|s| s.from_states(initial).into_iter().map(|from| (from, s.to.clone())))
.flat_map(|s| {
s.from_states(initial)
.into_iter()
.map(|from| (from, s.to.clone()))
})
.collect()
}
/// The moves a clock makes in `schema` (`after:` on a state), as
/// (from, to). Nobody presses them, and they still move records.
fn clock_moves(schema: &AggregateSchema) -> Vec<(String, String)> {
let mut moves: Vec<(String, String)> =
schema.after_for_state.iter().map(|(from, after)| (from.clone(), after.to.clone())).collect();
let mut moves: Vec<(String, String)> = schema
.after_for_state
.iter()
.map(|(from, after)| (from.clone(), after.to.clone()))
.collect();
moves.sort();
moves
}
@@ -402,7 +432,11 @@ pub fn check(
scope: "need",
code,
subject,
level: if need.planned || need.paused { Level::Warn } else { level },
level: if need.planned || need.paused {
Level::Warn
} else {
level
},
message: match (need.paused, need.planned, level == Level::Fail) {
(true, _, true) => format!("paused, not open now: {message}"),
(false, true, true) => format!("planned, not met yet: {message}"),
@@ -411,13 +445,23 @@ pub fn check(
})
};
if !seen.insert(need.id.as_str()) {
report(Level::Fail, "need-duplicate", String::new(), "declared more than once".to_string());
report(
Level::Fail,
"need-duplicate",
String::new(),
"declared more than once".to_string(),
);
continue;
}
// The visitor's side: can they get their answer recorded?
match questions.get(&need.enters) {
None => report(Level::Fail, "entry-missing", need.enters.clone(), format!("entry page {:?} does not exist", need.enters)),
None => report(
Level::Fail,
"entry-missing",
need.enters.clone(),
format!("entry page {:?} does not exist", need.enters),
),
Some(page) if !visible(page, need.enters_as.as_deref()) => report(
Level::Fail,
"entry-gated",
@@ -433,7 +477,10 @@ pub fn check(
Level::Warn,
"entry-no-responsible",
need.enters.clone(),
format!("entry page {:?} names no one responsible to turn to", need.enters),
format!(
"entry page {:?} names no one responsible to turn to",
need.enters
),
);
}
let best = entry_viable(questions, need, false)
@@ -515,7 +562,10 @@ pub fn check(
Level::Fail,
"machine-missing",
need.lands_in.clone(),
format!("aggregates.yaml declares no state machine for {:?}", need.lands_in),
format!(
"aggregates.yaml declares no state machine for {:?}",
need.lands_in
),
);
continue;
};
@@ -523,10 +573,15 @@ pub fn check(
let own: Vec<&Desk> = desks
.iter()
.filter(|d| {
d.group.as_deref().is_some_and(|g| g == need.handled_by || need.also_moved_by.iter().any(|m| m == g))
d.group.as_deref().is_some_and(|g| {
g == need.handled_by || need.also_moved_by.iter().any(|m| m == g)
})
})
.collect();
if !desks.iter().any(|d| d.group.as_deref() == Some(need.handled_by.as_str())) {
if !desks
.iter()
.any(|d| d.group.as_deref() == Some(need.handled_by.as_str()))
{
let others: Vec<String> = desks
.iter()
.map(|d| format!("{} ({})", d.page, d.group.as_deref().unwrap_or("ungated")))
@@ -560,7 +615,11 @@ pub fn check(
moves.extend(self_moves(questions, &need.lands_in, &schema.initial));
moves.extend(clock_moves(schema));
let exits = |state: &str| -> Vec<String> {
moves.iter().filter(|(from, _)| from == state).map(|(_, to)| to.clone()).collect()
moves
.iter()
.filter(|(from, _)| from == state)
.map(|(_, to)| to.clone())
.collect()
};
let movers = std::iter::once(need.handled_by.as_str())
.chain(need.also_moved_by.iter().map(String::as_str))
@@ -573,7 +632,9 @@ pub fn check(
let elsewhere = offered_elsewhere(&desks, from, to);
let desk = own.first().map(|d| d.page.as_str()).unwrap_or("their desk");
if elsewhere.is_empty() {
format!("add `{{ from: {from}, to: {to}, label: ... }}` to the transitions on {desk}")
format!(
"add `{{ from: {from}, to: {to}, label: ... }}` to the transitions on {desk}"
)
} else {
format!(
"{} already offers it - add that group to also_moved_by, or add `{{ from: {from}, to: {to}, label: ... }}` on {desk}",
@@ -627,7 +688,10 @@ pub fn check(
Level::Fail,
"done-unknown",
state.clone(),
format!("done_when names {state:?}, which {:?} has no such state", need.lands_in),
format!(
"done_when names {state:?}, which {:?} has no such state",
need.lands_in
),
);
} else if !reached.contains(state) {
unreached_reported.insert((need.lands_in.clone(), state.clone()));
@@ -678,7 +742,10 @@ pub fn check(
if done.contains(before.as_str()) || done.contains(after.as_str()) {
continue;
}
if reached.contains(before) && reached.contains(after) && !reachable(before, &moves).contains(after) {
if reached.contains(before)
&& reached.contains(after)
&& !reachable(before, &moves).contains(after)
{
report(
Level::Fail,
"stage-order",
@@ -724,7 +791,13 @@ pub fn check(
}
}
findings.extend(operations(file, questions, aggregates, &strands_reported, &unreached_reported));
findings.extend(operations(
file,
questions,
aggregates,
&strands_reported,
&unreached_reported,
));
let need_ids: HashSet<&str> = file.needs.iter().map(|n| n.id.as_str()).collect();
for persona in &file.personas {
@@ -735,7 +808,10 @@ pub fn check(
code: "persona-unknown-need",
subject: persona.id.clone(),
level: Level::Fail,
message: format!("persona {:?} points at a need that is not declared", persona.id),
message: format!(
"persona {:?} points at a need that is not declared",
persona.id
),
});
}
}
@@ -761,7 +837,9 @@ fn operations(
buckets.sort();
buckets.dedup();
for bucket in buckets {
let Some(schema) = aggregates.get(bucket) else { continue };
let Some(schema) = aggregates.get(bucket) else {
continue;
};
let needs: Vec<&Need> = file.needs.iter().filter(|n| n.lands_in == bucket).collect();
// Only open needs make a gap fail; a bucket that only planned
// or paused needs use reports it as a warning.
@@ -791,18 +869,29 @@ fn operations(
// for the bucket's first need.
let home = needs
.iter()
.find_map(|n| desks.iter().find(|d| d.group.as_deref() == Some(n.handled_by.as_str())))
.find_map(|n| {
desks
.iter()
.find(|d| d.group.as_deref() == Some(n.handled_by.as_str()))
})
.map(|d| d.page.clone())
.unwrap_or_else(|| "the handling group's desk".to_string());
let button = |from: &str, to: &str| format!("add `{{ from: {from}, to: {to}, label: ... }}` to the transitions on {home}");
let button = |from: &str, to: &str| {
format!("add `{{ from: {from}, to: {to}, label: ... }}` to the transitions on {home}")
};
let mut states: Vec<&String> = schema.event_for_state.keys().collect();
states.sort();
for state in &states {
if reached.contains(*state) || unreached_reported.contains(&(bucket.to_string(), (*state).clone())) {
if reached.contains(*state)
|| unreached_reported.contains(&(bucket.to_string(), (*state).clone()))
{
continue;
}
let mut ways: Vec<&String> = reached.iter().filter(|from| schema.allowed(from).contains(state)).collect();
let mut ways: Vec<&String> = reached
.iter()
.filter(|from| schema.allowed(from).contains(state))
.collect();
ways.sort();
report(
Level::Fail,
@@ -875,17 +964,26 @@ fn operations(
.flat_map(|f| &f.requirements)
.map(|r| (r.name.as_str(), r.kind.as_str()))
.collect();
let mut mail_states: Vec<(&String, &crate::aggregates::MailSpec)> = schema.mail_for_state.iter().collect();
let mut mail_states: Vec<(&String, &crate::aggregates::MailSpec)> =
schema.mail_for_state.iter().collect();
mail_states.sort_by(|a, b| a.0.cmp(b.0));
for (state, mail) in mail_states {
if !reached.contains(state) {
report(Level::Warn, "mail-unreachable", state.clone(), format!("the mail on entering {state:?} never goes out: no record reaches {state:?} - fix: make {state:?} reachable, or drop its mail"));
}
let mut unknown: Vec<String> = Vec::new();
for key in placeholders(&mail.subject).into_iter().chain(placeholders(&mail.body)) {
for key in placeholders(&mail.subject)
.into_iter()
.chain(placeholders(&mail.body))
{
let known = match key.split_once('.') {
Some((parent, _)) => fields.iter().any(|(name, kind)| *name == parent && *kind == "record"),
None => MAIL_BUILTINS.contains(&key.as_str()) || fields.iter().any(|(name, _)| *name == key),
Some((parent, _)) => fields
.iter()
.any(|(name, kind)| *name == parent && *kind == "record"),
None => {
MAIL_BUILTINS.contains(&key.as_str())
|| fields.iter().any(|(name, _)| *name == key)
}
};
if !known && !unknown.contains(&key) {
unknown.push(key);
@@ -913,7 +1011,17 @@ fn operations(
/// What the mailer fills in on every state mail, whatever the record
/// holds (`mail::vars_for`, `mail::site_vars`, `mail::record_links`).
const MAIL_BUILTINS: &[&str] = &["site", "site_name", "chain", "email", "bucket", "state", "link", "self_link", "desk_link"];
const MAIL_BUILTINS: &[&str] = &[
"site",
"site_name",
"chain",
"email",
"bucket",
"state",
"link",
"self_link",
"desk_link",
];
/// The `{key}`s in a mail template, read the way `mail::render` reads
/// them: letters, digits, `_`, `-` and `.`, closed by `}`.
@@ -1011,7 +1119,10 @@ pub fn sim_model(
// The simulator's shape: the states a visitor's own link
// moves to (it does not model where from).
let mut self_move_targets: Vec<String> =
self_moves(questions, &need.lands_in, &schema.initial).into_iter().map(|(_, to)| to).collect();
self_moves(questions, &need.lands_in, &schema.initial)
.into_iter()
.map(|(_, to)| to)
.collect();
self_move_targets.dedup();
buckets.entry(need.lands_in.clone()).or_insert_with(|| {
json!({
@@ -1116,13 +1227,25 @@ pub fn tasks(file: &NeedsFile, questions: &Questions, skim: bool) -> Vec<Task> {
};
for (i, step) in path.iter().enumerate() {
let page = &questions[&step.page];
let open: Vec<&Alternative> = page.alternatives.iter().filter(|a| !a.disabled && leads_somewhere(a)).collect();
let open: Vec<&Alternative> = page
.alternatives
.iter()
.filter(|a| !a.disabled && leads_somewhere(a))
.collect();
if open.len() < 2 {
continue;
}
let context = page_context(page);
let mut trail: Vec<String> = path[..=i].iter().map(|s| s.page.clone()).collect();
let expect: Vec<String> = viable(questions, need, page, need.max_steps - i, i == 0, false, &mut trail)
let expect: Vec<String> = viable(
questions,
need,
page,
need.max_steps - i,
i == 0,
false,
&mut trail,
)
.into_iter()
.map(|v| v.alternative)
.collect();
@@ -1132,7 +1255,11 @@ pub fn tasks(file: &NeedsFile, questions: &Questions, skim: bool) -> Vec<Task> {
let description = plain(&a.description);
(
a.name.clone(),
if skim || description.is_empty() { a.name.clone() } else { description },
if skim || description.is_empty() {
a.name.clone()
} else {
description
},
)
})
.collect();
@@ -1273,7 +1400,10 @@ mod tests {
fn effort_budget_counts_required_fields_only() {
let (questions, aggregates) = site(FULL_DESK);
let findings = check(&needs(" max_fields: 1\n"), &questions, &aggregates);
assert_eq!(fails(&findings), vec!["the path asks for 2 required field(s), over the budget of 1"]);
assert_eq!(
fails(&findings),
vec!["the path asks for 2 required field(s), over the budget of 1"]
);
// A field that carries rather than asks is not effort: the
// record an answer is about arrives on the link.
let (mut questions, aggregates) = site(FULL_DESK);
@@ -1292,7 +1422,9 @@ mod tests {
#[test]
fn a_desk_missing_a_move_strands_records_and_cannot_finish() {
// The desk can quote, but never close a quoted job.
let (questions, aggregates) = site(" - { to: quoted, label: Quote }\n - { to: lost, label: Lose }\n");
let (questions, aggregates) = site(
" - { to: quoted, label: Quote }\n - { to: lost, label: Lose }\n",
);
let findings = check(&needs(""), &questions, &aggregates);
assert_eq!(
fails(&findings),
@@ -1344,8 +1476,12 @@ mod tests {
assert_eq!(fields.len(), 3);
assert_eq!(fields[2]["optional"], true);
// The desk's buttons, labelled, per state.
let moves = model["buckets"]["projects"]["desks"][0]["moves"].as_array().unwrap();
assert!(moves.iter().any(|m| m["from"] == "quoted" && m["to"] == "won" && m["label"] == "Win"));
let moves = model["buckets"]["projects"]["desks"][0]["moves"]
.as_array()
.unwrap();
assert!(moves
.iter()
.any(|m| m["from"] == "quoted" && m["to"] == "won" && m["label"] == "Win"));
assert_eq!(model["buckets"]["projects"]["initial"], "open");
assert_eq!(model["personas"][0]["id"], "founder");
@@ -1369,7 +1505,9 @@ mod tests {
fn a_need_can_be_carried_by_more_than_one_groups_desk() {
// owners may only quote or lose; closing a quoted job is the
// site managers' button, on a desk of their own.
let (mut questions, aggregates) = site(" - { to: quoted, label: Quote }\n - { to: lost, label: Lose }\n");
let (mut questions, aggregates) = site(
" - { to: quoted, label: Quote }\n - { to: lost, label: Lose }\n",
);
let more = build_questions(&[(
"site/index.yaml".to_string(),
"name: Site desk\nqualifies: site_managers\nalternatives:\n - name: Projects\n features:\n - name: \"\"\n resource:\n source: { kind: kv, bucket: projects }\n requires_group: site_managers\n transitions:\n - { from: quoted, to: won, label: Win }\n - { from: quoted, to: lost, label: Lose }\n".to_string(),
@@ -1385,7 +1523,8 @@ mod tests {
assert!(check(&file, &questions, &aggregates).is_empty());
// But someone has to be answerable: handled_by needs its own desk.
file.needs[0].handled_by = "nobody".to_string();
assert!(fails(&check(&file, &questions, &aggregates))[0].starts_with("no desk gated on \"nobody\""));
assert!(fails(&check(&file, &questions, &aggregates))[0]
.starts_with("no desk gated on \"nobody\""));
}
#[test]
@@ -1418,7 +1557,10 @@ mod tests {
let file = needs("");
let tasks = tasks(&file, &questions, false);
let skimmed = super::tasks(&file, &questions, true);
assert_eq!(skimmed[0].questions["pick"].criteria["Follow along"], "Follow along");
assert_eq!(
skimmed[0].questions["pick"].criteria["Follow along"],
"Follow along"
);
assert_eq!(tasks.len(), 1);
let task = &tasks[0];
assert_eq!(task.id, "founder@/");
@@ -1428,11 +1570,19 @@ mod tests {
assert_eq!(q.criteria["Follow along"], "Notes by mail.");
assert!(q.instructions.contains("What are you here for?"));
let right = vec![Answer { id: "founder@/".into(), choice: "Hire us".into(), confidence: Some(0.55) }];
let right = vec![Answer {
id: "founder@/".into(),
choice: "Hire us".into(),
confidence: Some(0.55),
}];
let s = score(&tasks, &right, 0.6);
assert_eq!((s.correct, s.misroutes.len(), s.hesitant.len()), (1, 0, 1));
let wrong = vec![Answer { id: "founder@/".into(), choice: "Follow along".into(), confidence: Some(0.9) }];
let wrong = vec![Answer {
id: "founder@/".into(),
choice: "Follow along".into(),
confidence: Some(0.9),
}];
let s = score(&tasks, &wrong, 0.6);
assert_eq!((s.correct, s.misroutes.len()), (0, 1));
assert_eq!(s.accuracy(), 0.0);
@@ -1442,7 +1592,10 @@ mod tests {
/// The fixture site with its aggregates replaced, for the checks
/// that are about the state machine rather than the pages.
fn site_with(desk_moves: &str, projects: &str) -> (Questions, HashMap<String, AggregateSchema>) {
fn site_with(
desk_moves: &str,
projects: &str,
) -> (Questions, HashMap<String, AggregateSchema>) {
let (questions, _) = site(desk_moves);
let aggregates = parse_aggregates_yaml(&format!(
"aggregates:\n - bucket: projects\n{projects} - bucket: subscribers\n attended_by: a mailer\n initial: open\n states:\n open: {{ event: subscribed }}\n transitions: {{}}\n"
@@ -1452,7 +1605,11 @@ mod tests {
}
fn warns(findings: &[Finding]) -> Vec<&str> {
findings.iter().filter(|f| f.level == Level::Warn).map(|f| f.message.as_str()).collect()
findings
.iter()
.filter(|f| f.level == Level::Warn)
.map(|f| f.message.as_str())
.collect()
}
#[test]
@@ -1507,7 +1664,11 @@ mod tests {
" - { to: quoted, label: Quote }\n - { to: surveyed, label: Survey }\n - { from: surveyed, to: lost, label: Lose }\n - { from: quoted, to: won, label: Win }\n - { from: quoted, to: lost, label: Lose }\n",
" initial: open\n states:\n open: { event: submitted }\n surveyed: { event: surveyed }\n quoted: { event: quoted }\n won: { event: won }\n lost: { event: lost }\n transitions:\n open: [surveyed, quoted]\n surveyed: [lost]\n quoted: [won, lost]\n",
);
let findings = check(&needs(" stages: [open, surveyed, quoted, won, lost]\n"), &questions, &aggregates);
let findings = check(
&needs(" stages: [open, surveyed, quoted, won, lost]\n"),
&questions,
&aggregates,
);
assert_eq!(
fails(&findings),
vec!["stage \"quoted\" comes after \"surveyed\" for the business, but no move leads on from \"surveyed\" to it - fix: add an edge from \"surveyed\" toward \"quoted\" in aggregates.yaml under \"projects\""]
@@ -1540,14 +1701,25 @@ mod tests {
#[test]
fn a_finding_keeps_its_id_when_reworded_and_says_its_fix() {
let (questions, aggregates) = site(" - { to: quoted, label: Quote }\n - { to: lost, label: Lose }\n");
let (questions, aggregates) = site(
" - { to: quoted, label: Quote }\n - { to: lost, label: Lose }\n",
);
let findings = check(&needs(""), &questions, &aggregates);
let strand = findings.iter().find(|f| f.code == "strand").unwrap();
assert_eq!(strand.subject, "quoted");
assert_eq!(strand.fix(), Some("add `{ from: quoted, to: won, label: ... }` to the transitions on /desk"));
let reworded = Finding { message: "something else entirely".into(), ..clone_of(strand) };
assert_eq!(
strand.fix(),
Some("add `{ from: quoted, to: won, label: ... }` to the transitions on /desk")
);
let reworded = Finding {
message: "something else entirely".into(),
..clone_of(strand)
};
assert_eq!(reworded.id(), strand.id());
let elsewhere = Finding { subject: "open".into(), ..clone_of(strand) };
let elsewhere = Finding {
subject: "open".into(),
..clone_of(strand)
};
assert_ne!(elsewhere.id(), strand.id());
let json = strand.to_json();
assert_eq!(json["code"], "strand");
@@ -1556,6 +1728,13 @@ mod tests {
}
fn clone_of(f: &Finding) -> Finding {
Finding { need: f.need.clone(), level: f.level, message: f.message.clone(), scope: f.scope, code: f.code, subject: f.subject.clone() }
Finding {
need: f.need.clone(),
level: f.level,
message: f.message.clone(),
scope: f.scope,
code: f.code,
subject: f.subject.clone(),
}
}
}
+104 -20
View File
@@ -14,14 +14,19 @@
use std::collections::{BTreeMap, HashMap};
use futures::StreamExt;
use crate::events::store::{ensure_stream, load_events};
use crate::lint::local::{load_aggregates_from_dir, load_from_dir, load_needs_from_dir};
use crate::lint::needs;
use crate::{aggregates, answers, content};
use futures::StreamExt;
pub async fn run(argv: Vec<String>) -> anyhow::Result<()> {
let value = |flag: &str| argv.iter().position(|a| a == flag).and_then(|i| argv.get(i + 1)).cloned();
let value = |flag: &str| {
argv.iter()
.position(|a| a == flag)
.and_then(|i| argv.get(i + 1))
.cloned()
};
let usage = "usage: iris replay --path <questions dir> --nats <url> (--cases <cases.jsonl> | --report-only)";
let dir = value("--path").ok_or_else(|| anyhow::anyhow!(usage))?;
let nats_url = value("--nats").ok_or_else(|| anyhow::anyhow!(usage))?;
@@ -32,31 +37,71 @@ pub async fn run(argv: Vec<String>) -> anyhow::Result<()> {
let questions = load_from_dir(&dir)?;
let schemas = content::with_builtin_aggregates(load_aggregates_from_dir(&dir)?);
let file = needs::parse(&load_needs_from_dir(&dir)?.ok_or_else(|| anyhow::anyhow!("the repo has no needs.yaml"))?)?;
let file = needs::parse(
&load_needs_from_dir(&dir)?.ok_or_else(|| anyhow::anyhow!("the repo has no needs.yaml"))?,
)?;
let js = async_nats::jetstream::new(async_nats::connect(&nats_url).await?);
// The same call the site makes at boot.
ensure_stream(&js).await?;
if let Some(path) = cases_path {
let raw = std::fs::read_to_string(&path).map_err(|e| anyhow::anyhow!("reading {path}: {e}"))?;
let raw =
std::fs::read_to_string(&path).map_err(|e| anyhow::anyhow!("reading {path}: {e}"))?;
let (mut created, mut moved, mut refused) = (0, 0, 0);
for (n, line) in raw.lines().filter(|l| !l.trim().is_empty()).enumerate() {
let case: serde_json::Value = serde_json::from_str(line)?;
let Some(need) = file.needs.iter().find(|need| case["need"] == need.id.as_str()) else { continue };
let Some(schema) = schemas.get(&need.lands_in) else { continue };
let Some(need) = file
.needs
.iter()
.find(|need| case["need"] == need.id.as_str())
else {
continue;
};
let Some(schema) = schemas.get(&need.lands_in) else {
continue;
};
// Only a case that got as far as submitting leaves a record.
let states: Vec<&str> = case["states"].as_array().into_iter().flatten().filter_map(|s| s.as_str()).collect();
let states: Vec<&str> = case["states"]
.as_array()
.into_iter()
.flatten()
.filter_map(|s| s.as_str())
.collect();
if states.is_empty() {
continue;
}
let step = needs::path_for(&questions, need).and_then(|p| p.last().cloned());
let (page, alternative) = step.map(|s| (s.page, s.alternative)).unwrap_or_default();
let id = format!("sim-{}-{}-{n}", case["persona"].as_str().unwrap_or("x"), case["run"]);
let id = format!(
"sim-{}-{}-{n}",
case["persona"].as_str().unwrap_or("x"),
case["run"]
);
let now = chrono::Utc::now().timestamp_millis();
answers::store_answer(&js, &schemas, &need.lands_in, id.clone(), &page, &alternative, &case["record"], now, 0).await?;
answers::store_answer(
&js,
&schemas,
&need.lands_in,
id.clone(),
&page,
&alternative,
&case["record"],
now,
0,
)
.await?;
created += 1;
for target in &states[1..] {
match aggregates::transition(&js, schema, &id, target, serde_json::json!({"by": "simulation"}), now).await {
match aggregates::transition(
&js,
schema,
&id,
target,
serde_json::json!({"by": "simulation"}),
now,
)
.await
{
Ok(_) => moved += 1,
Err(e) => {
refused += 1;
@@ -72,33 +117,72 @@ pub async fn run(argv: Vec<String>) -> anyhow::Result<()> {
// Read every bucket back the way the engine knows it: by replaying
// each record's own event log.
let mut by_bucket: HashMap<String, BTreeMap<String, usize>> = HashMap::new();
for bucket in file.needs.iter().map(|n| n.lands_in.clone()).collect::<std::collections::BTreeSet<_>>() {
let (Some(schema), Ok(store)) = (schemas.get(&bucket), js.get_key_value(&bucket).await) else { continue };
for bucket in file
.needs
.iter()
.map(|n| n.lands_in.clone())
.collect::<std::collections::BTreeSet<_>>()
{
let (Some(schema), Ok(store)) = (schemas.get(&bucket), js.get_key_value(&bucket).await)
else {
continue;
};
let mut keys = store.keys().await?;
while let Some(Ok(id)) = keys.next().await {
let events = load_events(&js, &bucket, &id).await?;
if let Some(record) = aggregates::replay(schema, &id, &events) {
*by_bucket.entry(bucket.clone()).or_default().entry(record.state).or_default() += 1;
*by_bucket
.entry(bucket.clone())
.or_default()
.entry(record.state)
.or_default() += 1;
}
}
}
println!("{:<18} {:>8} {:>9} {:>9} {}", "bucket", "records", "finished", "success", "by state");
println!(
"{:<18} {:>8} {:>9} {:>9} {}",
"bucket", "records", "finished", "success", "by state"
);
let mut buckets: Vec<_> = by_bucket.iter().collect();
buckets.sort();
for (bucket, states) in buckets {
let needs_here: Vec<&needs::Need> = file.needs.iter().filter(|n| &n.lands_in == bucket).collect();
let is = |pick: fn(&needs::Need) -> &Vec<String>, state: &String| needs_here.iter().any(|n| pick(n).contains(state));
let needs_here: Vec<&needs::Need> = file
.needs
.iter()
.filter(|n| &n.lands_in == bucket)
.collect();
let is = |pick: fn(&needs::Need) -> &Vec<String>, state: &String| {
needs_here.iter().any(|n| pick(n).contains(state))
};
let total: usize = states.values().sum();
let finished: usize = states.iter().filter(|(s, _)| is(|n| &n.done_when, s)).map(|(_, c)| c).sum();
let success: usize = states.iter().filter(|(s, _)| is(|n| &n.success, s)).map(|(_, c)| c).sum();
let pct = |n: usize| if total == 0 { "-".to_string() } else { format!("{:.0}%", n as f64 * 100.0 / total as f64) };
let finished: usize = states
.iter()
.filter(|(s, _)| is(|n| &n.done_when, s))
.map(|(_, c)| c)
.sum();
let success: usize = states
.iter()
.filter(|(s, _)| is(|n| &n.success, s))
.map(|(_, c)| c)
.sum();
let pct = |n: usize| {
if total == 0 {
"-".to_string()
} else {
format!("{:.0}%", n as f64 * 100.0 / total as f64)
}
};
println!(
"{:<18} {:>8} {:>9} {:>9} {}",
bucket,
total,
pct(finished),
pct(success),
states.iter().map(|(s, c)| format!("{s} {c}")).collect::<Vec<_>>().join(", ")
states
.iter()
.map(|(s, c)| format!("{s} {c}"))
.collect::<Vec<_>>()
.join(", ")
);
}
Ok(())
+388 -79
View File
@@ -74,7 +74,10 @@ pub struct Incoming {
/// address can be forged. The sender must therefore also match the
/// address the record already holds, which is the one portal wrote to
/// in the first place.
pub fn note_for(incoming: &Incoming, answer: &crate::answers::Answer) -> Result<crate::answers::Note, String> {
pub fn note_for(
incoming: &Incoming,
answer: &crate::answers::Answer,
) -> Result<crate::answers::Note, String> {
if incoming.v != CONTRACT {
return Err(format!("reply contract v{} is not v{CONTRACT}", incoming.v));
}
@@ -90,7 +93,10 @@ pub fn note_for(incoming: &Incoming, answer: &crate::answers::Answer) -> Result<
.filter(|e| !e.is_empty())
.ok_or_else(|| "the record holds no address to compare the sender with".to_string())?;
if address_of(&incoming.from) != held {
return Err(format!("{} is not the address this record was written to", incoming.from));
return Err(format!(
"{} is not the address this record was written to",
incoming.from
));
}
if answer.notes.iter().any(|n| n.id == incoming.id) {
return Err("this reply is already on the record".to_string());
@@ -128,7 +134,10 @@ pub async fn receive(state: crate::server::AppState) {
return;
}
};
tracing::info!(subject = RECEIVED_SUBJECT, "listening for replies to case mail");
tracing::info!(
subject = RECEIVED_SUBJECT,
"listening for replies to case mail"
);
while let Some(msg) = sub.next().await {
let incoming: Incoming = match serde_json::from_slice(&msg.payload) {
Ok(i) => i,
@@ -138,8 +147,12 @@ pub async fn receive(state: crate::server::AppState) {
}
};
match note_onto(&state, &incoming).await {
Ok(()) => tracing::info!(bucket = %incoming.bucket, record = %incoming.record, "a reply was added to the record"),
Err(e) => tracing::warn!(bucket = %incoming.bucket, record = %incoming.record, "reply refused: {e}"),
Ok(()) => {
tracing::info!(bucket = %incoming.bucket, record = %incoming.record, "a reply was added to the record")
}
Err(e) => {
tracing::warn!(bucket = %incoming.bucket, record = %incoming.record, "reply refused: {e}")
}
}
}
}
@@ -147,7 +160,9 @@ pub async fn receive(state: crate::server::AppState) {
/// The record's own bucket is the only one a reply may touch, and the
/// note is appended under the same read-modify-write the desks use.
async fn note_onto(state: &crate::server::AppState, incoming: &Incoming) -> anyhow::Result<()> {
if !state.aggregates.load().contains_key(&incoming.bucket) && !crate::content::bucket_is_read(&state.questions.load(), &incoming.bucket) {
if !state.aggregates.load().contains_key(&incoming.bucket)
&& !crate::content::bucket_is_read(&state.questions.load(), &incoming.bucket)
{
anyhow::bail!("{} is not a bucket of this site", incoming.bucket);
}
let store = state.jetstream.get_key_value(&incoming.bucket).await?;
@@ -158,14 +173,18 @@ async fn note_onto(state: &crate::server::AppState, incoming: &Incoming) -> anyh
let mut answer: crate::answers::Answer = serde_json::from_slice(&bytes)?;
let note = note_for(incoming, &answer).map_err(|e| anyhow::anyhow!(e))?;
answer.notes.push(note);
store.put(&incoming.record, serde_json::to_vec(&answer)?.into()).await?;
store
.put(&incoming.record, serde_json::to_vec(&answer)?.into())
.await?;
Ok(())
}
/// Idempotent, called by portal at boot and by gdo. A week of
/// retention: a mail nobody delivered in a week is stale, not owed.
/// The duplicate window has to fit inside it.
pub async fn ensure_stream(js: &async_nats::jetstream::Context) -> anyhow::Result<async_nats::jetstream::stream::Stream> {
pub async fn ensure_stream(
js: &async_nats::jetstream::Context,
) -> anyhow::Result<async_nats::jetstream::stream::Stream> {
Ok(js
.get_or_create_stream(async_nats::jetstream::stream::Config {
name: STREAM.to_string(),
@@ -213,7 +232,8 @@ pub fn render(template: &str, vars: &serde_json::Map<String, serde_json::Value>)
/// in the body is dropped, the same rule as page descriptions.
pub fn to_html(markdown: &str) -> String {
use pulldown_cmark::{html, Event, Options, Parser};
let parser = Parser::new_ext(markdown, Options::empty()).filter(|e| !matches!(e, Event::Html(_) | Event::InlineHtml(_)));
let parser = Parser::new_ext(markdown, Options::empty())
.filter(|e| !matches!(e, Event::Html(_) | Event::InlineHtml(_)));
let mut body = String::new();
html::push_html(&mut body, parser);
format!(
@@ -221,28 +241,46 @@ pub fn to_html(markdown: &str) -> String {
)
}
fn public_url() -> String {
std::env::var("PUBLIC_URL").unwrap_or_default().trim_end_matches('/').to_string()
pub(crate) fn public_url() -> String {
std::env::var("PUBLIC_URL")
.unwrap_or_default()
.trim_end_matches('/')
.to_string()
}
fn site_name(site: &crate::content::SiteConfig) -> String {
site.title.clone().unwrap_or_else(|| option_env!("SITE_NAME").unwrap_or("portal").to_string())
site.title
.clone()
.unwrap_or_else(|| option_env!("SITE_NAME").unwrap_or("portal").to_string())
}
/// The variables a state mail may use: every field of the record,
/// then the runtime's own.
pub fn vars_for(bucket: &str, id: &str, state: &str, responses: &serde_json::Value) -> serde_json::Map<String, serde_json::Value> {
pub fn vars_for(
bucket: &str,
id: &str,
state: &str,
responses: &serde_json::Value,
) -> serde_json::Map<String, serde_json::Value> {
let mut vars = serde_json::Map::new();
if let Some(fields) = responses.as_object() {
for (k, v) in fields {
vars.insert(k.clone(), match v {
vars.insert(
k.clone(),
match v {
serde_json::Value::String(s) => serde_json::Value::String(s.clone()),
serde_json::Value::Null => serde_json::Value::String(String::new()),
other => serde_json::Value::String(other.to_string()),
});
},
);
}
}
let email = responses.get("email").and_then(|v| v.as_str()).unwrap_or_default().trim().to_string();
let email = responses
.get("email")
.and_then(|v| v.as_str())
.unwrap_or_default()
.trim()
.to_string();
vars.insert("email".into(), email.into());
vars.insert("site".into(), public_url().into());
vars.insert("chain".into(), id.into());
@@ -303,7 +341,10 @@ pub fn record_links(
.filter_map(|f| f.resource.as_ref())
.any(|r| r.bucket() == Some(bucket) && (!moving || !r.transitions.is_empty()))
};
let desk_page = pages.iter().find(|q| reads(q, true)).or_else(|| pages.iter().find(|q| reads(q, false)));
let desk_page = pages
.iter()
.find(|q| reads(q, true))
.or_else(|| pages.iter().find(|q| reads(q, false)));
let site = public_url();
let self_link = match self_page {
Some(q) if email.contains('@') => {
@@ -312,7 +353,9 @@ pub fn record_links(
}
_ => String::new(),
};
let desk_link = desk_page.map(|q| format!("{site}{}", q.id)).unwrap_or_default();
let desk_link = desk_page
.map(|q| format!("{site}{}", q.id))
.unwrap_or_default();
let link = match (to_submitter, self_link.is_empty(), desk_link.is_empty()) {
(true, false, _) => self_link.clone(),
(false, _, false) => desk_link.clone(),
@@ -338,7 +381,12 @@ pub fn parent_of(to: &str) -> Option<&str> {
/// `bucket`, the record it names, read from its own bucket. A
/// reference that does not resolve adds nothing, and a template that
/// needed it renders empty there.
async fn parent_vars(state: &crate::server::AppState, bucket: &str, responses: &serde_json::Value, vars: &mut serde_json::Map<String, serde_json::Value>) {
async fn parent_vars(
state: &crate::server::AppState,
bucket: &str,
responses: &serde_json::Value,
vars: &mut serde_json::Map<String, serde_json::Value>,
) {
let questions = state.questions.load();
let refs: Vec<(String, String)> = questions
.values()
@@ -349,10 +397,22 @@ async fn parent_vars(state: &crate::server::AppState, bucket: &str, responses: &
.filter_map(|r| Some((r.name.clone(), r.of.clone()?)))
.collect();
for (field, of) in refs {
let Some(id) = responses.get(&field).and_then(|v| v.as_str()).filter(|s| !s.is_empty()) else { continue };
let Ok(store) = state.jetstream.get_key_value(&of).await else { continue };
let Ok(Some(bytes)) = store.get(id).await else { continue };
let Ok(parent) = serde_json::from_slice::<crate::answers::Answer>(&bytes) else { continue };
let Some(id) = responses
.get(&field)
.and_then(|v| v.as_str())
.filter(|s| !s.is_empty())
else {
continue;
};
let Ok(store) = state.jetstream.get_key_value(&of).await else {
continue;
};
let Ok(Some(bytes)) = store.get(id).await else {
continue;
};
let Ok(parent) = serde_json::from_slice::<crate::answers::Answer>(&bytes) else {
continue;
};
if let Some(fields) = parent.responses.as_object() {
for (k, v) in fields {
let text = match v {
@@ -372,7 +432,9 @@ async fn publish(js: &async_nats::jetstream::Context, mail: &Outgoing) -> anyhow
let mut headers = async_nats::HeaderMap::new();
headers.insert("Nats-Msg-Id", mail.id.as_str());
let bytes = serde_json::to_vec(mail)?;
js.publish_with_headers(SUBJECT, headers, bytes.into()).await?.await?;
js.publish_with_headers(SUBJECT, headers, bytes.into())
.await?
.await?;
Ok(())
}
@@ -380,10 +442,24 @@ async fn publish(js: &async_nats::jetstream::Context, mail: &Outgoing) -> anyhow
/// if anything. Best-effort by design - the transition is already
/// committed, and a mail that cannot be rendered is logged, never a
/// failed request.
pub async fn on_state(state: &crate::server::AppState, bucket: &str, id: &str, to_state: &str, responses: &serde_json::Value, storyline: u32) {
pub async fn on_state(
state: &crate::server::AppState,
bucket: &str,
id: &str,
to_state: &str,
responses: &serde_json::Value,
storyline: u32,
) {
// The mail tells the storyline the person read.
let storyline = storyline as usize;
let Some(spec) = state.aggregates.load().at(storyline).get(bucket).and_then(|s| s.mail_for(to_state)).cloned() else {
let Some(spec) = state
.aggregates
.load()
.at(storyline)
.get(bucket)
.and_then(|s| s.mail_for(to_state))
.cloned()
else {
// A suggestion has its own words for this unless the content
// gave the state a mail, which then is the one sent.
if bucket == crate::content::EDIT_SUGGESTIONS_BUCKET {
@@ -394,23 +470,54 @@ pub async fn on_state(state: &crate::server::AppState, bucket: &str, id: &str, t
let site_storylines = state.site.load();
let site = site_storylines.at(storyline);
let Some(cfg) = &site.mail else {
tracing::warn!(bucket, to_state, "a mail is declared for this state, but site.yaml has no `mail.from` - not sent");
tracing::warn!(
bucket,
to_state,
"a mail is declared for this state, but site.yaml has no `mail.from` - not sent"
);
return;
};
let mut vars = vars_for(bucket, id, to_state, responses);
parent_vars(state, bucket, responses, &mut vars).await;
vars.extend(site_vars(site));
{
let email = vars.get("email").and_then(|v| v.as_str()).unwrap_or_default().to_string();
let email = vars
.get("email")
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string();
let questions = state.questions.load();
let initial = state.aggregates.load().at(storyline).get(bucket).map(|s| s.initial.clone()).unwrap_or_default();
vars.extend(record_links(questions.at(storyline), &initial, bucket, id, to_state, &email, spec.to_submitter()));
let initial = state
.aggregates
.load()
.at(storyline)
.get(bucket)
.map(|s| s.initial.clone())
.unwrap_or_default();
vars.extend(record_links(
questions.at(storyline),
&initial,
bucket,
id,
to_state,
&email,
spec.to_submitter(),
));
}
let to = if spec.to_submitter() {
match vars.get("email").and_then(|v| v.as_str()).filter(|e| e.contains('@')) {
match vars
.get("email")
.and_then(|v| v.as_str())
.filter(|e| e.contains('@'))
{
Some(e) => e.to_string(),
None => {
tracing::warn!(bucket, id, to_state, "mail to the submitter, but the record carries no email - not sent");
tracing::warn!(
bucket,
id,
to_state,
"mail to the submitter, but the record carries no email - not sent"
);
return;
}
}
@@ -444,14 +551,24 @@ pub async fn on_state(state: &crate::server::AppState, bucket: &str, id: &str, t
/// quotes, and past the first three only how many more - someone who
/// answers for a whole site is not read its table of contents.
pub fn questions_listed(questions: &[String], norwegian: bool) -> String {
let quote = |q: &String| if norwegian { format!("«{q}»") } else { format!("\u{201c}{q}\u{201d}") };
let quote = |q: &String| {
if norwegian {
format!("«{q}»")
} else {
format!("\u{201c}{q}\u{201d}")
}
};
let and = if norwegian { "og" } else { "and" };
match questions {
[] => String::new(),
[one] => quote(one),
few if few.len() <= 3 => {
let quoted: Vec<String> = few.iter().map(quote).collect();
format!("{} {and} {}", quoted[..quoted.len() - 1].join(", "), quoted[quoted.len() - 1])
format!(
"{} {and} {}",
quoted[..quoted.len() - 1].join(", "),
quoted[quoted.len() - 1]
)
}
many => {
let quoted: Vec<String> = many[..3].iter().map(quote).collect();
@@ -471,13 +588,20 @@ pub fn questions_listed(questions: &[String], norwegian: bool) -> String {
/// a question and they had no account: who they are to the site (the
/// person listed as asking that question), and the link that lets them
/// sign in and suggest changes to it. Built in, in the site's language.
pub async fn on_responsible(state: &crate::server::AppState, invited: &crate::people::Invited, questions: &[String]) {
pub async fn on_responsible(
state: &crate::server::AppState,
invited: &crate::people::Invited,
questions: &[String],
) {
let site = state.site.load();
let Some(cfg) = &site.mail else {
tracing::warn!(user = %invited.username, "an account was made for a responsible person, but site.yaml has no `mail.from` - they were not told");
return;
};
let norwegian = site.lang.as_deref().is_some_and(|l| l.starts_with("no") || l.starts_with("nb") || l.starts_with("nn"));
let norwegian = site
.lang
.as_deref()
.is_some_and(|l| l.starts_with("no") || l.starts_with("nb") || l.starts_with("nn"));
let list = questions_listed(questions, norwegian);
let (subject, body) = if norwegian {
(
@@ -500,7 +624,9 @@ pub async fn on_responsible(state: &crate::server::AppState, invited: &crate::pe
// The form's own fields first, so the invite's fixed placeholders
// win a clash; a date and time reads as one.
for (key, value) in &invited.fields {
let text = value.as_str().map(|v| spoken_datetime(v, norwegian).unwrap_or_else(|| v.to_string()));
let text = value
.as_str()
.map(|v| spoken_datetime(v, norwegian).unwrap_or_else(|| v.to_string()));
if let Some(text) = text {
vars.insert(key.clone(), text.into());
}
@@ -509,7 +635,10 @@ pub async fn on_responsible(state: &crate::server::AppState, invited: &crate::pe
vars.insert("username".into(), invited.username.clone().into());
vars.insert("questions".into(), list.into());
vars.extend(site_vars(&site));
vars.insert("link".into(), invited.reset_link.clone().unwrap_or_default().into());
vars.insert(
"link".into(),
invited.reset_link.clone().unwrap_or_default().into(),
);
vars.insert("expires".into(), expires.into());
let text = render(&body, &vars);
let mail = Outgoing {
@@ -564,32 +693,65 @@ pub fn suggestion_words(to_state: &str, norwegian: bool) -> Option<(&'static str
/// A suggestion moved: tell whoever made it. Replies go to the site's
/// inbox, which is who decided.
async fn on_suggestion_state(state: &crate::server::AppState, id: &str, to_state: &str, responses: &serde_json::Value) {
async fn on_suggestion_state(
state: &crate::server::AppState,
id: &str,
to_state: &str,
responses: &serde_json::Value,
) {
let site = state.site.load();
let norwegian = site.lang.as_deref().is_some_and(|l| l.starts_with("no") || l.starts_with("nb") || l.starts_with("nn"));
let norwegian = site
.lang
.as_deref()
.is_some_and(|l| l.starts_with("no") || l.starts_with("nb") || l.starts_with("nn"));
let Some((subject, body)) = suggestion_words(to_state, norwegian) else {
return;
};
let Some(cfg) = &site.mail else {
tracing::warn!(id, to_state, "a suggestion moved, but site.yaml has no `mail.from` - whoever made it was not told");
tracing::warn!(
id,
to_state,
"a suggestion moved, but site.yaml has no `mail.from` - whoever made it was not told"
);
return;
};
let field = |name: &str| responses.get(name).and_then(|v| v.as_str()).unwrap_or_default().trim().to_string();
let field = |name: &str| {
responses
.get(name)
.and_then(|v| v.as_str())
.unwrap_or_default()
.trim()
.to_string()
};
let to = field("email");
if !to.contains('@') {
tracing::warn!(id, to_state, "a suggestion moved, but the record carries no email - not sent");
tracing::warn!(
id,
to_state,
"a suggestion moved, but the record carries no email - not sent"
);
return;
}
let quoted: String = field("text").lines().map(|l| format!("> {l}")).collect::<Vec<_>>().join("\n");
let quoted: String = field("text")
.lines()
.map(|l| format!("> {l}"))
.collect::<Vec<_>>()
.join("\n");
let mut vars = site_vars(&site);
vars.insert("name".into(), field("name").into());
vars.insert("question".into(), field("question").into());
vars.insert("page".into(), format!("{}{}", public_url(), field("page")).into());
vars.insert(
"page".into(),
format!("{}{}", public_url(), field("page")).into(),
);
vars.insert("quoted".into(), quoted.into());
let text = render(body, &vars);
let mail = Outgoing {
v: CONTRACT,
id: format!("{}:{id}:{to_state}", crate::content::EDIT_SUGGESTIONS_BUCKET),
id: format!(
"{}:{id}:{to_state}",
crate::content::EDIT_SUGGESTIONS_BUCKET
),
site: site_name(&site),
from: cfg.from.clone(),
reply_to: cfg.reply_to.clone(),
@@ -606,14 +768,24 @@ async fn on_suggestion_state(state: &crate::server::AppState, id: &str, to_state
/// A change suggested by the person responsible for a page, to the
/// site's own inbox (`mail.reply_to`, else `mail.from`), with the
/// suggester as reply-to so answering the mail answers them.
pub async fn on_edit_suggestion(state: &crate::server::AppState, question_id: &str, question_name: &str, by_name: &str, by_email: &str, text: &str) {
pub async fn on_edit_suggestion(
state: &crate::server::AppState,
question_id: &str,
question_name: &str,
by_name: &str,
by_email: &str,
text: &str,
) {
let site = state.site.load();
let Some(cfg) = &site.mail else {
tracing::warn!(question_id, "an edit was suggested, but site.yaml has no `mail.from` - it is in portal_edit_suggestions only");
return;
};
let to = cfg.reply_to.clone().unwrap_or_else(|| cfg.from.clone());
let norwegian = site.lang.as_deref().is_some_and(|l| l.starts_with("no") || l.starts_with("nb") || l.starts_with("nn"));
let norwegian = site
.lang
.as_deref()
.is_some_and(|l| l.starts_with("no") || l.starts_with("nb") || l.starts_with("nn"));
let (subject, lead) = if norwegian {
(format!("Forslag til endring: «{question_name}»"), format!("{by_name} ({by_email}), som står som ansvarlig for spørsmålet, foreslår en endring på {}{question_id}:", public_url()))
} else {
@@ -650,13 +822,63 @@ pub fn spoken_datetime(value: &str, norwegian: bool) -> Option<String> {
let day = t.weekday().num_days_from_monday() as usize;
let month = t.month0() as usize;
Some(if norwegian {
const DAYS: [&str; 7] = ["mandag", "tirsdag", "onsdag", "torsdag", "fredag", "lørdag", "søndag"];
const MONTHS: [&str; 12] = ["januar", "februar", "mars", "april", "mai", "juni", "juli", "august", "september", "oktober", "november", "desember"];
format!("{} {}. {} kl. {:02}:{:02}", DAYS[day], t.day(), MONTHS[month], t.hour(), t.minute())
const DAYS: [&str; 7] = [
"mandag", "tirsdag", "onsdag", "torsdag", "fredag", "lørdag", "søndag",
];
const MONTHS: [&str; 12] = [
"januar",
"februar",
"mars",
"april",
"mai",
"juni",
"juli",
"august",
"september",
"oktober",
"november",
"desember",
];
format!(
"{} {}. {} kl. {:02}:{:02}",
DAYS[day],
t.day(),
MONTHS[month],
t.hour(),
t.minute()
)
} else {
const DAYS: [&str; 7] = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"];
const MONTHS: [&str; 12] = ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"];
format!("{} {} {} at {:02}:{:02}", DAYS[day], t.day(), MONTHS[month], t.hour(), t.minute())
const DAYS: [&str; 7] = [
"Monday",
"Tuesday",
"Wednesday",
"Thursday",
"Friday",
"Saturday",
"Sunday",
];
const MONTHS: [&str; 12] = [
"January",
"February",
"March",
"April",
"May",
"June",
"July",
"August",
"September",
"October",
"November",
"December",
];
format!(
"{} {} {} at {:02}:{:02}",
DAYS[day],
t.day(),
MONTHS[month],
t.hour(),
t.minute()
)
})
}
@@ -666,7 +888,10 @@ pub async fn on_invite(state: &crate::server::AppState, invited: &crate::people:
tracing::warn!(user = %invited.username, "invite made, but site.yaml has no `mail.from` - the link is on the desk only");
return;
};
let norwegian = site.lang.as_deref().is_some_and(|l| l.starts_with("no") || l.starts_with("nb") || l.starts_with("nn"));
let norwegian = site
.lang
.as_deref()
.is_some_and(|l| l.starts_with("no") || l.starts_with("nb") || l.starts_with("nn"));
let (subject, body) = match (invited.mail.as_ref().or(cfg.invite.as_ref()), invited.reset_link.is_some(), norwegian) {
(Some(t), _, _) => (t.subject.clone(), t.body.clone()),
(None, true, true) => (
@@ -698,7 +923,10 @@ pub async fn on_invite(state: &crate::server::AppState, invited: &crate::people:
vars.insert("email".into(), invited.email.clone().into());
vars.insert("group".into(), invited.group.clone().into());
vars.extend(site_vars(&site));
vars.insert("link".into(), invited.reset_link.clone().unwrap_or_default().into());
vars.insert(
"link".into(),
invited.reset_link.clone().unwrap_or_default().into(),
);
vars.insert("expires".into(), expires.into());
let text = render(&body, &vars);
let mail = Outgoing {
@@ -730,15 +958,42 @@ mod tests {
];
let questions = crate::content::build_questions(&files).unwrap();
let site = public_url();
let to_tenant = record_links(&questions, "open", "feil", "abc", "befaring", "kari@x.no", true);
assert_eq!(to_tenant["self_link"], format!("{site}/trekk?chain=abc&email=kari%40x.no"));
let to_tenant = record_links(
&questions,
"open",
"feil",
"abc",
"befaring",
"kari@x.no",
true,
);
assert_eq!(
to_tenant["self_link"],
format!("{site}/trekk?chain=abc&email=kari%40x.no")
);
assert_eq!(to_tenant["desk_link"], format!("{site}/styret"));
assert_eq!(to_tenant["link"], to_tenant["self_link"]);
let to_board = record_links(&questions, "open", "feil", "abc", "befaring", "kari@x.no", false);
let to_board = record_links(
&questions,
"open",
"feil",
"abc",
"befaring",
"kari@x.no",
false,
);
assert_eq!(to_board["link"], to_board["desk_link"]);
// Past where the self-service move can be made: the person has
// no page of their own, so `{link}` is the site.
let after = record_links(&questions, "open", "feil", "abc", "utbedret", "kari@x.no", true);
let after = record_links(
&questions,
"open",
"feil",
"abc",
"utbedret",
"kari@x.no",
true,
);
assert_eq!(after["self_link"], "");
assert_eq!(after["link"], site);
}
@@ -751,10 +1006,22 @@ mod tests {
assert_eq!(questions_listed(&q(0), true), "");
assert_eq!(questions_listed(&q(1), true), "«Q1?»");
assert_eq!(questions_listed(&q(3), true), "«Q1?», «Q2?» og «Q3?»");
assert_eq!(questions_listed(&q(4), true), "«Q1?», «Q2?», «Q3?» og ett spørsmål til");
assert_eq!(questions_listed(&q(17), true), "«Q1?», «Q2?», «Q3?» og 14 spørsmål til");
assert_eq!(questions_listed(&q(2), false), "\u{201c}Q1?\u{201d} and \u{201c}Q2?\u{201d}");
assert_eq!(questions_listed(&q(5), false), "\u{201c}Q1?\u{201d}, \u{201c}Q2?\u{201d}, \u{201c}Q3?\u{201d} and 2 more questions");
assert_eq!(
questions_listed(&q(4), true),
"«Q1?», «Q2?», «Q3?» og ett spørsmål til"
);
assert_eq!(
questions_listed(&q(17), true),
"«Q1?», «Q2?», «Q3?» og 14 spørsmål til"
);
assert_eq!(
questions_listed(&q(2), false),
"\u{201c}Q1?\u{201d} and \u{201c}Q2?\u{201d}"
);
assert_eq!(
questions_listed(&q(5), false),
"\u{201c}Q1?\u{201d}, \u{201c}Q2?\u{201d}, \u{201c}Q3?\u{201d} and 2 more questions"
);
}
#[test]
@@ -762,8 +1029,14 @@ mod tests {
for norwegian in [true, false] {
for state in ["discussed", "done", "declined"] {
let (subject, body) = suggestion_words(state, norwegian).expect(state);
assert!(subject.contains("{question}"), "{state}: the subject names the question");
assert!(body.contains("{page}") && body.contains("{quoted}"), "{state}: the body says where and what");
assert!(
subject.contains("{question}"),
"{state}: the subject names the question"
);
assert!(
body.contains("{page}") && body.contains("{quoted}"),
"{state}: the body says where and what"
);
}
// Arriving is told to the inbox, not back to who sent it.
assert!(suggestion_words("open", norwegian).is_none());
@@ -784,18 +1057,34 @@ mod tests {
#[test]
fn templates_take_fields_and_leave_stray_braces_alone() {
let responses = serde_json::json!({ "name": "Kari", "email": " kari@x.no ", "count": 3, "hva": null });
let responses =
serde_json::json!({ "name": "Kari", "email": " kari@x.no ", "count": 3, "hva": null });
let vars = vars_for("reports", "abc", "fixed", &responses);
assert_eq!(render("Hi {name}, {count} at {site}/decide?chain={chain}&email={email}", &vars), "Hi Kari, 3 at /decide?chain=abc&email=kari@x.no");
assert_eq!(render("{unknown} {hva} {not a key} {", &vars), " {not a key} {");
assert_eq!(
render(
"Hi {name}, {count} at {site}/decide?chain={chain}&email={email}",
&vars
),
"Hi Kari, 3 at /decide?chain=abc&email=kari@x.no"
);
assert_eq!(
render("{unknown} {hva} {not a key} {", &vars),
" {not a key} {"
);
assert_eq!(vars["bucket"], "reports");
assert_eq!(vars["state"], "fixed");
}
#[test]
fn a_date_and_time_from_a_form_reads_as_a_person_says_it() {
assert_eq!(spoken_datetime("2026-10-06T15:00", true).as_deref(), Some("tirsdag 6. oktober kl. 15:00"));
assert_eq!(spoken_datetime("2026-10-06T15:00", false).as_deref(), Some("Tuesday 6 October at 15:00"));
assert_eq!(
spoken_datetime("2026-10-06T15:00", true).as_deref(),
Some("tirsdag 6. oktober kl. 15:00")
);
assert_eq!(
spoken_datetime("2026-10-06T15:00", false).as_deref(),
Some("Tuesday 6 October at 15:00")
);
assert_eq!(spoken_datetime("Anders Kluge", true), None);
}
@@ -831,7 +1120,9 @@ mod tests {
)
.unwrap();
assert_eq!(older.subject, "s");
assert!(serde_json::to_string(&older).unwrap().contains("\"html\":\"h\""));
assert!(serde_json::to_string(&older)
.unwrap()
.contains("\"html\":\"h\""));
}
fn record(email: &str) -> crate::answers::Answer {
@@ -865,7 +1156,11 @@ mod tests {
#[test]
fn a_reply_from_the_address_on_the_record_becomes_a_note() {
let note = note_for(&reply("Kari <KARI@x.no>", " It is fixed now. "), &record("kari@x.no")).unwrap();
let note = note_for(
&reply("Kari <KARI@x.no>", " It is fixed now. "),
&record("kari@x.no"),
)
.unwrap();
assert_eq!(note.from, "kari@x.no");
assert_eq!(note.text, "It is fixed now.");
assert_eq!(note.id, "mail-1");
@@ -876,21 +1171,33 @@ mod tests {
fn a_reply_from_anyone_else_is_refused() {
// The address a reply arrives on is unguessable, but a From is
// a claim, and a mailbox is a door anyone may knock on.
let err = note_for(&reply("someone@else.no", "Let me in"), &record("kari@x.no")).unwrap_err();
assert!(err.contains("not the address this record was written to"), "{err}");
let err =
note_for(&reply("someone@else.no", "Let me in"), &record("kari@x.no")).unwrap_err();
assert!(
err.contains("not the address this record was written to"),
"{err}"
);
// Nothing to compare against is also a refusal, not a pass.
let mut anonymous = record("kari@x.no");
anonymous.responses = serde_json::json!({ "where": "The bend" });
assert!(note_for(&reply("kari@x.no", "hi"), &anonymous).unwrap_err().contains("no address"));
assert!(note_for(&reply("kari@x.no", "hi"), &anonymous)
.unwrap_err()
.contains("no address"));
}
#[test]
fn an_empty_reply_and_a_repeated_one_add_nothing() {
assert!(note_for(&reply("kari@x.no", " \n "), &record("kari@x.no")).unwrap_err().contains("no text"));
assert!(
note_for(&reply("kari@x.no", " \n "), &record("kari@x.no"))
.unwrap_err()
.contains("no text")
);
let mut already = record("kari@x.no");
already.notes.push(note_for(&reply("kari@x.no", "It is fixed."), &already).unwrap());
already
.notes
.push(note_for(&reply("kari@x.no", "It is fixed."), &already).unwrap());
let err = note_for(&reply("kari@x.no", "It is fixed."), &already).unwrap_err();
assert!(err.contains("already on the record"), "{err}");
}
@@ -899,7 +1206,9 @@ mod tests {
fn a_reply_from_a_contract_this_portal_does_not_know_is_refused() {
let mut future = reply("kari@x.no", "hello");
future.v = CONTRACT + 1;
assert!(note_for(&future, &record("kari@x.no")).unwrap_err().contains("contract"));
assert!(note_for(&future, &record("kari@x.no"))
.unwrap_err()
.contains("contract"));
}
#[test]
+1
View File
@@ -316,6 +316,7 @@ async fn serve() -> anyhow::Result<()> {
// Content-shipped assets (hero module etc.), same-origin.
.route("/site/{*path}", get(content::site_asset_handler))
.route("/.well-known/{*path}", get(content::well_known_handler))
.route("/profile.mobileconfig", get(portal::profile::handler))
.route(
"/automation/kv/{bucket}",
get(content::automation_kv_handler),
+76 -15
View File
@@ -42,7 +42,11 @@ fn style_id() -> String {
let (Some(owner), Some(id), None) = (parts.next(), parts.next(), parts.next()) else {
return false;
};
[owner, id].iter().all(|p| !p.is_empty() && p.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'))
[owner, id].iter().all(|p| {
!p.is_empty()
&& p.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_')
})
})
.unwrap_or_else(|| "mapbox/streets-v12".to_string())
}
@@ -50,7 +54,9 @@ fn style_id() -> String {
/// The instance's Mapbox token, or `None` when maps are off.
#[cfg(feature = "ssr")]
pub fn token() -> Option<String> {
std::env::var("MAPBOX_TOKEN").ok().filter(|t| !t.trim().is_empty())
std::env::var("MAPBOX_TOKEN")
.ok()
.filter(|t| !t.trim().is_empty())
}
#[cfg(feature = "ssr")]
@@ -64,7 +70,11 @@ pub fn enabled() -> bool {
#[cfg(feature = "ssr")]
pub async fn style() -> Response {
if !enabled() {
return (StatusCode::NOT_FOUND, "maps are not configured on this instance").into_response();
return (
StatusCode::NOT_FOUND,
"maps are not configured on this instance",
)
.into_response();
}
let style = serde_json::json!({
"version": 8,
@@ -79,7 +89,10 @@ pub async fn style() -> Response {
"layers": [{ "id": "base", "type": "raster", "source": "base" }],
});
(
[(header::CONTENT_TYPE, "application/json"), (header::CACHE_CONTROL, "public, max-age=3600")],
[
(header::CONTENT_TYPE, "application/json"),
(header::CACHE_CONTROL, "public, max-age=3600"),
],
style.to_string(),
)
.into_response()
@@ -90,7 +103,11 @@ pub async fn style() -> Response {
#[cfg(feature = "ssr")]
pub async fn tile(Path((z, x, y)): Path<(u32, u32, u32)>) -> Response {
let Some(token) = token() else {
return (StatusCode::NOT_FOUND, "maps are not configured on this instance").into_response();
return (
StatusCode::NOT_FOUND,
"maps are not configured on this instance",
)
.into_response();
};
if z > 22 || x >= 1 << z.min(22) || y >= 1 << z.min(22) {
return (StatusCode::BAD_REQUEST, "no such tile").into_response();
@@ -108,13 +125,21 @@ pub async fn tile(Path((z, x, y)): Path<(u32, u32, u32)>) -> Response {
#[cfg(feature = "ssr")]
pub async fn still(Path((place, size)): Path<(String, String)>) -> Response {
let Some(token) = token() else {
return (StatusCode::NOT_FOUND, "maps are not configured on this instance").into_response();
return (
StatusCode::NOT_FOUND,
"maps are not configured on this instance",
)
.into_response();
};
let Some((lon, lat, zoom)) = parse_place(&place) else {
return (StatusCode::BAD_REQUEST, "expected <lon>,<lat>,<zoom>").into_response();
};
let Some((w, h)) = parse_size(&size) else {
return (StatusCode::BAD_REQUEST, "expected <width>x<height>, at most 1280").into_response();
return (
StatusCode::BAD_REQUEST,
"expected <width>x<height>, at most 1280",
)
.into_response();
};
// The pin is drawn by Mapbox so the image is self-contained; the
// marker colour is portal's ink, not a brand colour.
@@ -127,7 +152,12 @@ pub async fn still(Path((place, size)): Path<(String, String)>) -> Response {
#[cfg(feature = "ssr")]
async fn fetch(url: &str, mime: &'static str, cache: &'static str) -> Response {
let resp = match crate::http::client().get(url).send().await.and_then(|r| r.error_for_status()) {
let resp = match crate::http::client()
.get(url)
.send()
.await
.and_then(|r| r.error_for_status())
{
Ok(r) => r,
Err(e) => {
tracing::warn!("map upstream: {e}");
@@ -143,7 +173,10 @@ async fn fetch(url: &str, mime: &'static str, cache: &'static str) -> Response {
match crate::http::bytes(resp, crate::http::ASSET_CAP).await {
Ok(bytes) => (
[
(header::CONTENT_TYPE, upstream.unwrap_or_else(|| mime.to_string())),
(
header::CONTENT_TYPE,
upstream.unwrap_or_else(|| mime.to_string()),
),
(header::CACHE_CONTROL, cache.to_string()),
],
axum::body::Bytes::from(bytes),
@@ -159,16 +192,25 @@ async fn fetch(url: &str, mime: &'static str, cache: &'static str) -> Response {
#[cfg(feature = "ssr")]
fn parse_place(place: &str) -> Option<(String, String, String)> {
let mut parts = place.split(',');
let (Some(lon), Some(lat), Some(zoom), None) = (parts.next(), parts.next(), parts.next(), parts.next()) else {
let (Some(lon), Some(lat), Some(zoom), None) =
(parts.next(), parts.next(), parts.next(), parts.next())
else {
return None;
};
let lon: f64 = lon.parse().ok()?;
let lat: f64 = lat.parse().ok()?;
let zoom: f64 = zoom.parse().ok()?;
if !(-180.0..=180.0).contains(&lon) || !(-85.0511..=85.0511).contains(&lat) || !(0.0..=22.0).contains(&zoom) {
if !(-180.0..=180.0).contains(&lon)
|| !(-85.0511..=85.0511).contains(&lat)
|| !(0.0..=22.0).contains(&zoom)
{
return None;
}
Some((format!("{lon:.5}"), format!("{lat:.5}"), format!("{zoom:.2}")))
Some((
format!("{lon:.5}"),
format!("{lat:.5}"),
format!("{zoom:.2}"),
))
}
#[cfg(feature = "ssr")]
@@ -203,7 +245,15 @@ mod tests {
parse_place("11.123456,59.987654,13.5"),
Some(("11.12346".into(), "59.98765".into(), "13.50".into()))
);
for bad in ["11,59", "11,59,13,0", "200,59,13", "11,95,13", "11,59,30", "11,59,x", "11,59,13?a=b"] {
for bad in [
"11,59",
"11,59,13,0",
"200,59,13",
"11,95,13",
"11,59,30",
"11,59,x",
"11,59,13?a=b",
] {
assert_eq!(parse_place(bad), None, "{bad}");
}
}
@@ -212,7 +262,15 @@ mod tests {
#[cfg(feature = "ssr")]
fn a_size_is_two_numbers_within_what_the_service_draws() {
assert_eq!(parse_size("600x300"), Some((600, 300)));
for bad in ["600", "0x300", "600x0", "2000x300", "600x2000", "600x300x2", "sixx300"] {
for bad in [
"600",
"0x300",
"600x0",
"2000x300",
"600x2000",
"600x300x2",
"sixx300",
] {
assert_eq!(parse_size(bad), None, "{bad}");
}
}
@@ -221,7 +279,10 @@ mod tests {
fn a_location_is_a_lat_and_a_lon_and_a_zoom_it_may_leave_out() {
let v = serde_json::json!({ "lat": 59.5, "lon": 11.0, "zoom": 16 });
assert_eq!(read(&v), Some((59.5, 11.0, 16.0)));
assert_eq!(read(&serde_json::json!({ "lat": 59.5, "lon": 11.0 })), Some((59.5, 11.0, 13.0)));
assert_eq!(
read(&serde_json::json!({ "lat": 59.5, "lon": 11.0 })),
Some((59.5, 11.0, 13.0))
);
for not in [
serde_json::json!({ "lat": 91.0, "lon": 11.0 }),
serde_json::json!({ "lat": 59.5 }),
+216 -43
View File
@@ -32,7 +32,10 @@ pub const BY: &str = "portal";
/// The country a phone number without one is in, as a calling code.
fn default_country() -> String {
std::env::var("PHONE_COUNTRY").ok().filter(|c| !c.trim().is_empty()).unwrap_or_else(|| "+47".to_string())
std::env::var("PHONE_COUNTRY")
.ok()
.filter(|c| !c.trim().is_empty())
.unwrap_or_else(|| "+47".to_string())
}
/// Who a person is, as far as a record or a session knows.
@@ -69,7 +72,11 @@ pub fn normalize_phone(raw: &str, country: &str) -> Option<String> {
}
fn normalize_email(raw: &str) -> Option<String> {
let e = raw.trim().trim_start_matches("mailto:").trim().to_lowercase();
let e = raw
.trim()
.trim_start_matches("mailto:")
.trim()
.to_lowercase();
(e.contains('@') && !e.contains(char::is_whitespace)).then_some(e)
}
@@ -88,7 +95,8 @@ impl Identity {
let country = default_country();
Identity {
sub: None,
phone: field(&["phone", "telefon", "tel", "mobil", "mobile", "phone_number"]).and_then(|p| normalize_phone(&p, &country)),
phone: field(&["phone", "telefon", "tel", "mobil", "mobile", "phone_number"])
.and_then(|p| normalize_phone(&p, &country)),
email: field(&["email", "epost", "e-post", "mail"]).and_then(|e| normalize_email(&e)),
}
}
@@ -97,7 +105,10 @@ impl Identity {
let country = default_country();
Identity {
sub: Some(user.sub.clone()),
phone: user.phone.as_deref().and_then(|p| normalize_phone(p, &country)),
phone: user
.phone
.as_deref()
.and_then(|p| normalize_phone(p, &country)),
email: user.email.as_deref().and_then(normalize_email),
}
}
@@ -107,8 +118,16 @@ impl Identity {
pub fn from_contact(contact: &str) -> Self {
let country = default_country();
match normalize_email(contact) {
Some(email) => Identity { sub: None, phone: None, email: Some(email) },
None => Identity { sub: None, phone: normalize_phone(contact, &country), email: None },
Some(email) => Identity {
sub: None,
phone: None,
email: Some(email),
},
None => Identity {
sub: None,
phone: normalize_phone(contact, &country),
email: None,
},
}
}
@@ -120,14 +139,26 @@ impl Identity {
.as_deref()
.or(self.email.as_deref())
.or(self.sub.as_deref())
.map(|k| k.chars().map(|c| if c.is_ascii_alphanumeric() || c == '-' || c == '_' { c } else { '_' }).collect())
.map(|k| {
k.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '-' || c == '_' {
c
} else {
'_'
}
})
.collect()
})
}
/// Whether a record about `other` is about this person: the same
/// subject, phone number or address, any one of them.
pub fn matches(&self, other: &Identity) -> bool {
let same = |a: &Option<String>, b: &Option<String>| a.is_some() && a == b;
same(&self.sub, &other.sub) || same(&self.phone, &other.phone) || same(&self.email, &other.email)
same(&self.sub, &other.sub)
|| same(&self.phone, &other.phone)
|| same(&self.email, &other.email)
}
/// What is worth writing on a record from a login: everything the
@@ -149,14 +180,34 @@ pub fn record_id(group: &str, key: &str) -> String {
/// The identity a record holds.
pub fn identity_of(answer: &Answer) -> Identity {
Identity {
sub: answer.responses.get("sub").and_then(|v| v.as_str()).filter(|s| !s.is_empty()).map(String::from),
phone: answer.responses.get("phone").and_then(|v| v.as_str()).filter(|s| !s.is_empty()).map(String::from),
email: answer.responses.get("email").and_then(|v| v.as_str()).filter(|s| !s.is_empty()).map(String::from),
sub: answer
.responses
.get("sub")
.and_then(|v| v.as_str())
.filter(|s| !s.is_empty())
.map(String::from),
phone: answer
.responses
.get("phone")
.and_then(|v| v.as_str())
.filter(|s| !s.is_empty())
.map(String::from),
email: answer
.responses
.get("email")
.and_then(|v| v.as_str())
.filter(|s| !s.is_empty())
.map(String::from),
}
}
pub fn group_of(answer: &Answer) -> String {
answer.responses.get("group").and_then(|v| v.as_str()).unwrap_or_default().to_string()
answer
.responses
.get("group")
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string()
}
/// The groups `who` is an active member of, from every record.
@@ -172,19 +223,26 @@ pub fn groups_of(records: &[Answer], who: &Identity) -> Vec<String> {
groups
}
async fn store(state: &crate::server::AppState) -> anyhow::Result<async_nats::jetstream::kv::Store> {
async fn store(
state: &crate::server::AppState,
) -> anyhow::Result<async_nats::jetstream::kv::Store> {
match state.jetstream.get_key_value(BUCKET).await {
Ok(store) => Ok(store),
Err(_) => Ok(state
.jetstream
.create_key_value(async_nats::jetstream::kv::Config { bucket: BUCKET.to_string(), ..Default::default() })
.create_key_value(async_nats::jetstream::kv::Config {
bucket: BUCKET.to_string(),
..Default::default()
})
.await?),
}
}
/// Every membership record.
pub async fn all(state: &crate::server::AppState) -> Vec<Answer> {
let Ok(store) = state.jetstream.get_key_value(BUCKET).await else { return Vec::new() };
let Ok(store) = state.jetstream.get_key_value(BUCKET).await else {
return Vec::new();
};
let keys: Vec<String> = {
use futures::TryStreamExt;
match store.keys().await {
@@ -205,18 +263,36 @@ pub async fn all(state: &crate::server::AppState) -> Vec<Answer> {
/// Moves one record the way a desk button would: through the event
/// log, onto the projection, out on the answers subject.
async fn move_to(state: &crate::server::AppState, store: &async_nats::jetstream::kv::Store, mut answer: Answer, target: &str, by: &str) -> anyhow::Result<Answer> {
async fn move_to(
state: &crate::server::AppState,
store: &async_nats::jetstream::kv::Store,
mut answer: Answer,
target: &str,
by: &str,
) -> anyhow::Result<Answer> {
let aggregates = state.aggregates.load();
let schema = aggregates.get(BUCKET).ok_or_else(|| anyhow::anyhow!("no {BUCKET} schema"))?;
let schema = aggregates
.get(BUCKET)
.ok_or_else(|| anyhow::anyhow!("no {BUCKET} schema"))?;
let now_ms = chrono::Utc::now().timestamp_millis();
let payload = serde_json::json!({ "to": target, "item": answer.id, "by": by });
crate::answers::transition_or_reseed(&state.jetstream, schema, &answer.id, target, &answer.state, payload.clone(), now_ms)
crate::answers::transition_or_reseed(
&state.jetstream,
schema,
&answer.id,
target,
&answer.state,
payload.clone(),
now_ms,
)
.await
.map_err(|e| anyhow::anyhow!("{e}"))?;
answer.state = target.to_string();
answer.decided_ms = Some(now_ms);
answer.decided_by = Some(by.to_string());
store.put(&answer.id, serde_json::to_vec(&answer)?.into()).await?;
store
.put(&answer.id, serde_json::to_vec(&answer)?.into())
.await?;
let parent_hashes = vec![answer.id.clone()];
let chain_hash = crate::chain::hash_node(&answer.question_id, &parent_hashes, &payload, now_ms);
let event = crate::events::AnswerSubmitted {
@@ -236,15 +312,29 @@ async fn move_to(state: &crate::server::AppState, store: &async_nats::jetstream:
/// `invited`; a removed one is invited again; an invited or active one
/// is left as it is, so a second grant costs nothing. `created` in
/// what comes back says whether the person had a place already.
pub async fn grant(state: &crate::server::AppState, group: &str, name: &str, who: &Identity, by: &str) -> anyhow::Result<crate::people::Invited> {
let key = who.key().ok_or_else(|| anyhow::anyhow!("a membership needs a phone number or an address to know the person by"))?;
pub async fn grant(
state: &crate::server::AppState,
group: &str,
name: &str,
who: &Identity,
by: &str,
) -> anyhow::Result<crate::people::Invited> {
let key = who.key().ok_or_else(|| {
anyhow::anyhow!("a membership needs a phone number or an address to know the person by")
})?;
let store = store(state).await?;
// The person's record in this group, however it was keyed: a desk
// that invited by address and a decision that granted by phone
// number are talking about one person once a sign-in has joined
// the two, and must not make two.
let existing: Option<Answer> = all(state).await.into_iter().find(|a| group_of(a) == group.trim() && identity_of(a).matches(who));
let id = existing.as_ref().map(|a| a.id.clone()).unwrap_or_else(|| record_id(group, &key));
let existing: Option<Answer> = all(state)
.await
.into_iter()
.find(|a| group_of(a) == group.trim() && identity_of(a).matches(who));
let id = existing
.as_ref()
.map(|a| a.id.clone())
.unwrap_or_else(|| record_id(group, &key));
let now_ms = chrono::Utc::now().timestamp_millis();
let invited = |created: bool| crate::people::Invited {
username: key.clone(),
@@ -269,7 +359,18 @@ pub async fn grant(state: &crate::server::AppState, group: &str, name: &str, who
"name": name, "group": group, "by": by,
"phone": who.phone, "email": who.email, "sub": who.sub,
});
crate::answers::store_answer(&state.jetstream, &state.aggregates.load(), BUCKET, id, "/", "Membership", &responses, now_ms, 0).await?;
crate::answers::store_answer(
&state.jetstream,
&state.aggregates.load(),
BUCKET,
id,
"/",
"Membership",
&responses,
now_ms,
0,
)
.await?;
Ok(invited(true))
}
}
@@ -277,11 +378,22 @@ pub async fn grant(state: &crate::server::AppState, group: &str, name: &str, who
/// Takes a person out of a group: the record moves to `removed` and
/// stays, so the desk can see who was in and put them back.
pub async fn revoke(state: &crate::server::AppState, group: &str, who: &Identity, by: &str) -> anyhow::Result<bool> {
let Some(key) = who.key() else { return Ok(false) };
pub async fn revoke(
state: &crate::server::AppState,
group: &str,
who: &Identity,
by: &str,
) -> anyhow::Result<bool> {
let Some(key) = who.key() else {
return Ok(false);
};
let store = store(state).await?;
let id = record_id(group, &key);
let Some(answer) = store.get(&id).await?.and_then(|b| serde_json::from_slice::<Answer>(&b).ok()) else {
let Some(answer) = store
.get(&id)
.await?
.and_then(|b| serde_json::from_slice::<Answer>(&b).ok())
else {
return Ok(false);
};
if answer.state == REMOVED {
@@ -296,10 +408,15 @@ pub async fn revoke(state: &crate::server::AppState, group: &str, who: &Identity
/// and their active groups come back to be their groups for the
/// session. Their `sub` is remembered, so the next login matches on
/// it before anything else.
pub async fn on_signed_in(state: &crate::server::AppState, user: &crate::auth::User) -> Vec<String> {
pub async fn on_signed_in(
state: &crate::server::AppState,
user: &crate::auth::User,
) -> Vec<String> {
let who = Identity::from_user(user);
let records = all(state).await;
let Ok(store) = store(state).await else { return groups_of(&records, &who) };
let Ok(store) = store(state).await else {
return groups_of(&records, &who);
};
let mut settled = Vec::with_capacity(records.len());
for answer in records {
let held = identity_of(&answer);
@@ -341,7 +458,13 @@ pub async fn on_signed_in(state: &crate::server::AppState, user: &crate::auth::U
mod tests {
use super::*;
fn record(group: &str, state: &str, phone: Option<&str>, email: Option<&str>, sub: Option<&str>) -> Answer {
fn record(
group: &str,
state: &str,
phone: Option<&str>,
email: Option<&str>,
sub: Option<&str>,
) -> Answer {
Answer {
id: record_id(group, "k"),
question_id: "/".into(),
@@ -359,39 +482,89 @@ mod tests {
#[test]
fn a_phone_number_is_one_string_for_everyone() {
assert_eq!(normalize_phone("912 34 567", "+47"), Some("+4791234567".into()));
assert_eq!(normalize_phone("+47 912-34-567", "+47"), Some("+4791234567".into()));
assert_eq!(normalize_phone("004791234567", "+47"), Some("+4791234567".into()));
assert_eq!(normalize_phone("91234567", "+46"), Some("+4691234567".into()));
assert_eq!(
normalize_phone("912 34 567", "+47"),
Some("+4791234567".into())
);
assert_eq!(
normalize_phone("+47 912-34-567", "+47"),
Some("+4791234567".into())
);
assert_eq!(
normalize_phone("004791234567", "+47"),
Some("+4791234567".into())
);
assert_eq!(
normalize_phone("91234567", "+46"),
Some("+4691234567".into())
);
assert_eq!(normalize_phone("kari@vel.no", "+47"), None);
assert_eq!(normalize_phone("123", "+47"), None);
}
#[test]
fn a_person_is_their_phone_first_and_their_address_second() {
let both = Identity::from_responses(&serde_json::json!({ "name": "Kari", "telefon": "912 34 567", "email": "Kari@Vel.no" }));
let both = Identity::from_responses(
&serde_json::json!({ "name": "Kari", "telefon": "912 34 567", "email": "Kari@Vel.no" }),
);
assert_eq!(both.key().as_deref(), Some("_4791234567"));
assert_eq!(both.email.as_deref(), Some("kari@vel.no"));
let mail_only = Identity::from_responses(&serde_json::json!({ "epost": "Kari@Vel.no" }));
assert_eq!(mail_only.key().as_deref(), Some("kari_vel_no"));
assert_eq!(Identity::from_contact("mailto:Post@Vel.no").email.as_deref(), Some("post@vel.no"));
assert_eq!(Identity::from_contact("912 34 567").phone.as_deref(), Some("+4791234567"));
assert_eq!(
Identity::from_contact("mailto:Post@Vel.no")
.email
.as_deref(),
Some("post@vel.no")
);
assert_eq!(
Identity::from_contact("912 34 567").phone.as_deref(),
Some("+4791234567")
);
assert!(Identity::default().key().is_none());
}
#[test]
fn a_record_is_about_a_person_by_any_one_of_sub_phone_or_email() {
let kari = Identity { sub: Some("abc".into()), phone: Some("+4791234567".into()), email: Some("kari@vel.no".into()) };
assert!(Identity { sub: None, phone: Some("+4791234567".into()), email: None }.matches(&kari));
assert!(Identity { sub: None, phone: None, email: Some("kari@vel.no".into()) }.matches(&kari));
assert!(Identity { sub: Some("abc".into()), phone: None, email: Some("old@vel.no".into()) }.matches(&kari));
assert!(!Identity { sub: None, phone: Some("+4799999999".into()), email: Some("ola@vel.no".into()) }.matches(&kari));
let kari = Identity {
sub: Some("abc".into()),
phone: Some("+4791234567".into()),
email: Some("kari@vel.no".into()),
};
assert!(Identity {
sub: None,
phone: Some("+4791234567".into()),
email: None
}
.matches(&kari));
assert!(Identity {
sub: None,
phone: None,
email: Some("kari@vel.no".into())
}
.matches(&kari));
assert!(Identity {
sub: Some("abc".into()),
phone: None,
email: Some("old@vel.no".into())
}
.matches(&kari));
assert!(!Identity {
sub: None,
phone: Some("+4799999999".into()),
email: Some("ola@vel.no".into())
}
.matches(&kari));
assert!(!Identity::default().matches(&kari));
}
#[test]
fn only_active_memberships_are_groups() {
let kari = Identity { sub: None, phone: Some("+4791234567".into()), email: Some("kari@vel.no".into()) };
let kari = Identity {
sub: None,
phone: Some("+4791234567".into()),
email: Some("kari@vel.no".into()),
};
let records = vec![
record("styret", ACTIVE, Some("+4791234567"), None, None),
record("kasserer", INVITED, None, Some("kari@vel.no"), None),
+331 -68
View File
@@ -37,14 +37,25 @@ pub enum People {
impl People {
/// `None` only when Kanidm is asked for and not configured.
pub fn from_env() -> Option<Self> {
match std::env::var("PEOPLE_BACKEND").ok().map(|b| b.trim().to_lowercase()).as_deref() {
match std::env::var("PEOPLE_BACKEND")
.ok()
.map(|b| b.trim().to_lowercase())
.as_deref()
{
Some("memberships") => Some(People::Memberships),
Some("kanidm") => Kanidm::from_env().map(People::Kanidm),
Some(other) => {
tracing::warn!(backend = other, "PEOPLE_BACKEND is neither kanidm nor memberships; using memberships");
tracing::warn!(
backend = other,
"PEOPLE_BACKEND is neither kanidm nor memberships; using memberships"
);
Some(People::Memberships)
}
None => Some(Kanidm::from_env().map(People::Kanidm).unwrap_or(People::Memberships)),
None => Some(
Kanidm::from_env()
.map(People::Kanidm)
.unwrap_or(People::Memberships),
),
}
}
@@ -55,10 +66,19 @@ impl People {
/// through Kanidm an account and a one-time link, through
/// memberships a record that becomes theirs when they sign in.
/// `by` is who asked, for the record.
pub async fn invite(&self, state: &crate::server::AppState, name: &str, who: &crate::memberships::Identity, group: &str, by: &str) -> anyhow::Result<Invited> {
pub async fn invite(
&self,
state: &crate::server::AppState,
name: &str,
who: &crate::memberships::Identity,
group: &str,
by: &str,
) -> anyhow::Result<Invited> {
match self {
People::Kanidm(kanidm) => {
let email = who.email.as_deref().ok_or_else(|| anyhow::anyhow!("Kanidm knows people by their address, and this one has none"))?;
let email = who.email.as_deref().ok_or_else(|| {
anyhow::anyhow!("Kanidm knows people by their address, and this one has none")
})?;
kanidm.invite(name, email, group).await
}
People::Memberships => {
@@ -75,7 +95,9 @@ impl People {
invited.link_expires_ms = account.link_expires_ms;
invited.created = account.created;
}
Err(e) => tracing::warn!(group, %email, "the membership is on file, but no Kanidm account could be made: {e}"),
Err(e) => {
tracing::warn!(group, %email, "the membership is on file, but no Kanidm account could be made: {e}")
}
}
}
Ok(invited)
@@ -85,10 +107,19 @@ impl People {
/// Takes a person out of a group. Through Kanidm only what portal
/// put there (see `Invited.already_member`); memberships remember.
pub async fn remove(&self, state: &crate::server::AppState, who: &crate::memberships::Identity, username: &str, group: &str, by: &str) -> anyhow::Result<()> {
pub async fn remove(
&self,
state: &crate::server::AppState,
who: &crate::memberships::Identity,
username: &str,
group: &str,
by: &str,
) -> anyhow::Result<()> {
match self {
People::Kanidm(kanidm) => kanidm.remove_member(group, username).await,
People::Memberships => crate::memberships::revoke(state, group, who, by).await.map(|_| ()),
People::Memberships => crate::memberships::revoke(state, group, who, by)
.await
.map(|_| ()),
}
}
@@ -140,7 +171,9 @@ pub struct Invited {
fn expiry_ms(value: &serde_json::Value) -> Option<i64> {
match value {
serde_json::Value::Number(n) => n.as_i64().map(|s| s * 1000),
serde_json::Value::String(s) => chrono::DateTime::parse_from_rfc3339(s).ok().map(|t| t.timestamp_millis()),
serde_json::Value::String(s) => chrono::DateTime::parse_from_rfc3339(s)
.ok()
.map(|t| t.timestamp_millis()),
_ => None,
}
}
@@ -163,7 +196,11 @@ fn members_from(body: &str) -> Vec<String> {
fn has_credentials_from(body: &str) -> bool {
serde_json::from_str::<serde_json::Value>(body)
.ok()
.and_then(|v| v.get("creds").and_then(|c| c.as_array()).map(|c| !c.is_empty()))
.and_then(|v| {
v.get("creds")
.and_then(|c| c.as_array())
.map(|c| !c.is_empty())
})
.unwrap_or(false)
}
@@ -184,7 +221,15 @@ fn person_lookup_found(status: u16, body: &str) -> Option<bool> {
/// `{"entries": [{"attrs": {"name": ["…"], …}}]}`.
fn name_from_search(body: &str) -> Option<String> {
let v: serde_json::Value = serde_json::from_str(body).ok()?;
v.get("entries")?.as_array()?.first()?.get("attrs")?.get("name")?.as_array()?.first()?.as_str().map(str::to_string)
v.get("entries")?
.as_array()?
.first()?
.get("attrs")?
.get("name")?
.as_array()?
.first()?
.as_str()
.map(str::to_string)
}
pub struct Kanidm {
@@ -202,13 +247,30 @@ impl Kanidm {
/// `None` when `KANIDM_API_TOKEN` is unset: invites fail closed with
/// a clear message instead of a half-made account.
pub fn from_env() -> Option<Self> {
let base = std::env::var("KANIDM_URL").ok()?.trim_end_matches('/').to_string();
let token = std::env::var("KANIDM_API_TOKEN").ok().filter(|t| !t.is_empty())?;
let group_prefix = std::env::var("KANIDM_GROUP_PREFIX").unwrap_or_default().trim().to_string();
Some(Kanidm { base, token, group_prefix })
let base = std::env::var("KANIDM_URL")
.ok()?
.trim_end_matches('/')
.to_string();
let token = std::env::var("KANIDM_API_TOKEN")
.ok()
.filter(|t| !t.is_empty())?;
let group_prefix = std::env::var("KANIDM_GROUP_PREFIX")
.unwrap_or_default()
.trim()
.to_string();
Some(Kanidm {
base,
token,
group_prefix,
})
}
async fn call(&self, method: openidconnect::reqwest::Method, path: &str, body: Option<serde_json::Value>) -> anyhow::Result<(u16, String)> {
async fn call(
&self,
method: openidconnect::reqwest::Method,
path: &str,
body: Option<serde_json::Value>,
) -> anyhow::Result<(u16, String)> {
let mut req = crate::http::client()
.request(method, format!("{}{path}", self.base))
.bearer_auth(&self.token)
@@ -223,8 +285,15 @@ impl Kanidm {
}
async fn person_exists(&self, username: &str) -> anyhow::Result<bool> {
let (status, text) = self.call(openidconnect::reqwest::Method::GET, &format!("/v1/person/{username}"), None).await?;
person_lookup_found(status, &text).ok_or_else(|| anyhow::anyhow!("kanidm: looking up {username}: http {status}"))
let (status, text) = self
.call(
openidconnect::reqwest::Method::GET,
&format!("/v1/person/{username}"),
None,
)
.await?;
person_lookup_found(status, &text)
.ok_or_else(|| anyhow::anyhow!("kanidm: looking up {username}: http {status}"))
}
/// The account that already carries this email, if the token may
@@ -275,7 +344,13 @@ impl Kanidm {
/// Who is in a group, by account name.
pub async fn members(&self, group: &str) -> anyhow::Result<Vec<String>> {
let (status, text) = self.call(openidconnect::reqwest::Method::GET, &format!("/v1/group/{}{group}/_attr/member", self.group_prefix), None).await?;
let (status, text) = self
.call(
openidconnect::reqwest::Method::GET,
&format!("/v1/group/{}{group}/_attr/member", self.group_prefix),
None,
)
.await?;
if !(200..300).contains(&status) {
anyhow::bail!("kanidm: reading the members of {group}: http {status}: {text}");
}
@@ -286,7 +361,11 @@ impl Kanidm {
/// service account manages, like adding them was.
pub async fn remove_member(&self, group: &str, username: &str) -> anyhow::Result<()> {
let (status, text) = self
.call(openidconnect::reqwest::Method::DELETE, &format!("/v1/group/{}{group}/_attr/member", self.group_prefix), Some(serde_json::json!([username])))
.call(
openidconnect::reqwest::Method::DELETE,
&format!("/v1/group/{}{group}/_attr/member", self.group_prefix),
Some(serde_json::json!([username])),
)
.await?;
if !(200..300).contains(&status) {
anyhow::bail!("kanidm: removing {username} from {group}: http {status}: {text}");
@@ -297,7 +376,13 @@ impl Kanidm {
/// Whether Kanidm shows a credential for this person. `false` is
/// "not that it can see", never "none": see `has_credentials_from`.
pub async fn has_credentials(&self, username: &str) -> anyhow::Result<bool> {
let (status, text) = self.call(openidconnect::reqwest::Method::GET, &format!("/v1/person/{username}/_credential/_status"), None).await?;
let (status, text) = self
.call(
openidconnect::reqwest::Method::GET,
&format!("/v1/person/{username}/_credential/_status"),
None,
)
.await?;
if !(200..300).contains(&status) {
anyhow::bail!("kanidm: credential status of {username}: http {status}");
}
@@ -308,7 +393,11 @@ impl Kanidm {
/// passkey, and when it runs out.
pub async fn reset_link(&self, username: &str) -> anyhow::Result<(String, Option<i64>)> {
let (status, text) = self
.call(openidconnect::reqwest::Method::GET, &format!("/v1/person/{username}/_credential/_update_intent/{LINK_SECONDS}"), None)
.call(
openidconnect::reqwest::Method::GET,
&format!("/v1/person/{username}/_credential/_update_intent/{LINK_SECONDS}"),
None,
)
.await?;
if !(200..300).contains(&status) {
anyhow::bail!("kanidm: reset token for {username}: http {status}: {text}");
@@ -320,10 +409,18 @@ impl Kanidm {
expiry_time: Option<serde_json::Value>,
}
let intent: Intent = serde_json::from_str(&text)?;
Ok((format!("{}/ui/reset?token={}", self.base, intent.token), intent.expiry_time.as_ref().and_then(expiry_ms)))
Ok((
format!("{}/ui/reset?token={}", self.base, intent.token),
intent.expiry_time.as_ref().and_then(expiry_ms),
))
}
async fn onboard(&self, name: &str, email: &str, group: Option<&str>) -> anyhow::Result<Invited> {
async fn onboard(
&self,
name: &str,
email: &str,
group: Option<&str>,
) -> anyhow::Result<Invited> {
let existing = self.find_by_mail(email).await?;
let username = existing.clone().unwrap_or_else(|| username_for(email));
let created = if existing.is_some() || self.person_exists(&username).await? {
@@ -351,9 +448,17 @@ impl Kanidm {
// were here before is what says whose the membership is; a
// failed read says "not before", which errs toward portal
// tidying up after itself.
already_member = !created && self.members(group).await.is_ok_and(|m| m.contains(&username));
already_member = !created
&& self
.members(group)
.await
.is_ok_and(|m| m.contains(&username));
let (status, text) = self
.call(openidconnect::reqwest::Method::POST, &format!("/v1/group/{}{group}/_attr/member", self.group_prefix), Some(serde_json::json!([username])))
.call(
openidconnect::reqwest::Method::POST,
&format!("/v1/group/{}{group}/_attr/member", self.group_prefix),
Some(serde_json::json!([username])),
)
.await?;
if !(200..300).contains(&status) {
anyhow::bail!("kanidm: adding {username} to {group}: http {status}: {text}");
@@ -393,7 +498,10 @@ fn link_runs_out(answer: &crate::answers::Answer) -> i64 {
/// The account an invite record is about.
fn invited_of(answer: &crate::answers::Answer) -> Option<Invited> {
answer.responses.get("invited").and_then(|v| serde_json::from_value(v.clone()).ok())
answer
.responses
.get("invited")
.and_then(|v| serde_json::from_value(v.clone()).ok())
}
/// Where a waiting invite goes now, if anywhere. Pure: what is known
@@ -401,7 +509,11 @@ fn invited_of(answer: &crate::answers::Answer) -> Option<Invited> {
///
/// An invite that made no account has no link to wait for: the person
/// had their way in already, and was only given a place.
pub fn invite_due(answer: &crate::answers::Answer, is_in: bool, now_ms: i64) -> Option<&'static str> {
pub fn invite_due(
answer: &crate::answers::Answer,
is_in: bool,
now_ms: i64,
) -> Option<&'static str> {
if answer.state != crate::answers::OPEN_STATE {
return None;
}
@@ -418,7 +530,11 @@ pub fn invite_due(answer: &crate::answers::Answer, is_in: bool, now_ms: i64) ->
/// An invite nobody at a desk made - the first person, someone the
/// content names as responsible - on file like any other, so the desk
/// that reads `portal_invites` sees it, and can send it again.
pub async fn record(state: &crate::server::AppState, invited: &Invited, responsible_for: &[String]) {
pub async fn record(
state: &crate::server::AppState,
invited: &Invited,
responsible_for: &[String],
) {
let now_ms = chrono::Utc::now().timestamp_millis();
let mut responses = serde_json::json!({
"name": invited.display_name,
@@ -429,23 +545,53 @@ pub async fn record(state: &crate::server::AppState, invited: &Invited, responsi
responses["responsible_for"] = serde_json::json!(responsible_for);
}
let id = crate::chain::hash_node("/", &[], &responses, now_ms);
if let Err(e) = crate::answers::store_answer(&state.jetstream, &state.aggregates.load(), INVITES_BUCKET, id, "/", SEED_ALTERNATIVE, &responses, now_ms, 0).await {
if let Err(e) = crate::answers::store_answer(
&state.jetstream,
&state.aggregates.load(),
INVITES_BUCKET,
id,
"/",
SEED_ALTERNATIVE,
&responses,
now_ms,
0,
)
.await
{
tracing::warn!(user = %invited.username, "the invite is sent, but could not be put on file for a desk: {e}");
}
}
/// Moves an invite record, the way a desk button would: through the
/// event log, onto the projection, and out on the answers subject.
async fn move_invite(state: &crate::server::AppState, store: &async_nats::jetstream::kv::Store, mut answer: crate::answers::Answer, target: &str) {
async fn move_invite(
state: &crate::server::AppState,
store: &async_nats::jetstream::kv::Store,
mut answer: crate::answers::Answer,
target: &str,
) {
let aggregates = state.aggregates.load();
let Some(schema) = aggregates.get(INVITES_BUCKET).filter(|s| s.allowed(&answer.state).iter().any(|t| t == target)) else {
let Some(schema) = aggregates
.get(INVITES_BUCKET)
.filter(|s| s.allowed(&answer.state).iter().any(|t| t == target))
else {
// A content repo may declare the bucket itself, without this
// state. Then the invite stays where its own graph has it.
return;
};
let now_ms = chrono::Utc::now().timestamp_millis();
let payload = serde_json::json!({ "to": target, "item": answer.id, "by": BY });
if let Err(e) = crate::answers::transition_or_reseed(&state.jetstream, schema, &answer.id, target, &answer.state, payload.clone(), now_ms).await {
if let Err(e) = crate::answers::transition_or_reseed(
&state.jetstream,
schema,
&answer.id,
target,
&answer.state,
payload.clone(),
now_ms,
)
.await
{
tracing::warn!(id = %answer.id, target, "an invite could not be moved: {e}");
return;
}
@@ -454,7 +600,11 @@ async fn move_invite(state: &crate::server::AppState, store: &async_nats::jetstr
answer.decided_by = Some(BY.to_string());
// The link is a way in for as long as it is on file; one that is
// used or run out is only something to leak.
if let Some(invited) = answer.responses.get_mut("invited").and_then(|i| i.as_object_mut()) {
if let Some(invited) = answer
.responses
.get_mut("invited")
.and_then(|i| i.as_object_mut())
{
invited.insert("reset_link".into(), serde_json::Value::Null);
}
if let Ok(data) = serde_json::to_vec(&answer) {
@@ -478,17 +628,33 @@ async fn move_invite(state: &crate::server::AppState, store: &async_nats::jetstr
}
/// Every invite on file that is still waiting in one of `states`.
async fn invites_in(state: &crate::server::AppState, states: &[&str]) -> Option<(async_nats::jetstream::kv::Store, Vec<crate::answers::Answer>)> {
async fn invites_in(
state: &crate::server::AppState,
states: &[&str],
) -> Option<(
async_nats::jetstream::kv::Store,
Vec<crate::answers::Answer>,
)> {
// Made on first write: a site that has invited nobody has no bucket.
let store = state.jetstream.get_key_value(INVITES_BUCKET).await.ok()?;
let keys: Vec<String> = {
use futures::TryStreamExt;
store.keys().await.ok()?.try_collect().await.unwrap_or_default()
store
.keys()
.await
.ok()?
.try_collect()
.await
.unwrap_or_default()
};
let mut found = Vec::new();
for id in keys {
let Ok(Some(bytes)) = store.get(&id).await else { continue };
let Ok(answer) = serde_json::from_slice::<crate::answers::Answer>(&bytes) else { continue };
let Ok(Some(bytes)) = store.get(&id).await else {
continue;
};
let Ok(answer) = serde_json::from_slice::<crate::answers::Answer>(&bytes) else {
continue;
};
if states.contains(&answer.state.as_str()) {
found.push(answer);
}
@@ -509,12 +675,17 @@ pub async fn run(state: crate::server::AppState) {
/// One pass over the waiting invites: done for whoever Kanidm shows a
/// credential for, expired for a link that ran out.
pub async fn sweep_invites(state: &crate::server::AppState) {
let Some((store, waiting)) = invites_in(state, &[crate::answers::OPEN_STATE]).await else { return };
let Some((store, waiting)) = invites_in(state, &[crate::answers::OPEN_STATE]).await else {
return;
};
let kanidm = Kanidm::from_env();
let now_ms = chrono::Utc::now().timestamp_millis();
for answer in waiting {
let is_in = match (&kanidm, invited_of(&answer)) {
(Some(kanidm), Some(invited)) if invited.created => kanidm.has_credentials(&invited.username).await.unwrap_or(false),
(Some(kanidm), Some(invited)) if invited.created => kanidm
.has_credentials(&invited.username)
.await
.unwrap_or(false),
_ => false,
};
if let Some(target) = invite_due(&answer, is_in, now_ms) {
@@ -527,14 +698,23 @@ pub async fn sweep_invites(state: &crate::server::AppState) {
/// address it went to for someone whose account goes by another name.
pub fn is_for(invited: &Invited, user: &crate::auth::User) -> bool {
let address = invited.email.trim().to_lowercase();
invited.username == user.username || (!address.is_empty() && user.email.as_deref().is_some_and(|e| e.trim().to_lowercase() == address))
invited.username == user.username
|| (!address.is_empty()
&& user
.email
.as_deref()
.is_some_and(|e| e.trim().to_lowercase() == address))
}
/// Someone signed in: whatever invites were waiting for them, or had
/// run out, are done. This is the one sure sign a link was used -
/// they are here.
pub async fn on_signed_in(state: crate::server::AppState, user: crate::auth::User) {
let Some((store, waiting)) = invites_in(&state, &[crate::answers::OPEN_STATE, INVITE_EXPIRED]).await else { return };
let Some((store, waiting)) =
invites_in(&state, &[crate::answers::OPEN_STATE, INVITE_EXPIRED]).await
else {
return;
};
for answer in waiting {
if invited_of(&answer).is_some_and(|invited| is_for(&invited, &user)) {
move_invite(&state, &store, answer, INVITE_DONE).await;
@@ -547,11 +727,20 @@ pub async fn on_signed_in(state: crate::server::AppState, user: crate::auth::Use
/// time, which for someone the content names as responsible is the one
/// that says so.
pub async fn on_reopened(state: &crate::server::AppState, id: &str) {
let Ok(store) = state.jetstream.get_key_value(INVITES_BUCKET).await else { return };
let Ok(Some(bytes)) = store.get(id).await else { return };
let Ok(mut answer) = serde_json::from_slice::<crate::answers::Answer>(&bytes) else { return };
let Ok(store) = state.jetstream.get_key_value(INVITES_BUCKET).await else {
return;
};
let Ok(Some(bytes)) = store.get(id).await else {
return;
};
let Ok(mut answer) = serde_json::from_slice::<crate::answers::Answer>(&bytes) else {
return;
};
let Some(mut invited) = invited_of(&answer) else {
tracing::warn!(id, "an invite was sent again, but the record names no account - nothing sent");
tracing::warn!(
id,
"an invite was sent again, but the record names no account - nothing sent"
);
return;
};
let Some(kanidm) = Kanidm::from_env() else {
@@ -568,7 +757,10 @@ pub async fn on_reopened(state: &crate::server::AppState, id: &str) {
invited.reset_link = Some(link);
invited.link_expires_ms = expires;
if let Some(map) = answer.responses.as_object_mut() {
map.insert("invited".to_string(), serde_json::to_value(&invited).unwrap_or_default());
map.insert(
"invited".to_string(),
serde_json::to_value(&invited).unwrap_or_default(),
);
}
if let Ok(data) = serde_json::to_vec(&answer) {
let _ = store.put(id, data.into()).await;
@@ -592,7 +784,13 @@ pub fn username_for(email: &str) -> String {
let local = email.split('@').next().unwrap_or(email).to_lowercase();
let mut out: String = local
.chars()
.map(|c| if c.is_ascii_lowercase() || c.is_ascii_digit() || matches!(c, '.' | '_' | '-') { c } else { '-' })
.map(|c| {
if c.is_ascii_lowercase() || c.is_ascii_digit() || matches!(c, '.' | '_' | '-') {
c
} else {
'-'
}
})
.collect();
while out.starts_with(|c: char| !c.is_ascii_lowercase()) && !out.is_empty() {
out.remove(0);
@@ -645,17 +843,33 @@ pub async fn on_state(
return;
};
let Some(people) = People::from_env() else {
tracing::warn!(bucket, id, to_state, group, "this state grants a group, but {} - nobody was added", People::NOT_CONFIGURED);
tracing::warn!(
bucket,
id,
to_state,
group,
"this state grants a group, but {} - nobody was added",
People::NOT_CONFIGURED
);
return;
};
let Some((name, who)) = who_for(responses) else {
tracing::warn!(bucket, id, to_state, group, "this state grants a group, but the record carries no name and no phone number or email - nobody was added");
return;
};
let invited = match people.invite(state, &name, &who, &group, &format!("{bucket}:{id}")).await {
let invited = match people
.invite(state, &name, &who, &group, &format!("{bucket}:{id}"))
.await
{
Ok(invited) => invited,
Err(e) => {
tracing::error!(bucket, id, to_state, group, "granting the group failed: {e}");
tracing::error!(
bucket,
id,
to_state,
group,
"granting the group failed: {e}"
);
return;
}
};
@@ -673,14 +887,20 @@ mod tests {
#[test]
fn usernames_are_the_email_local_part_made_safe() {
assert_eq!(username_for("jorn@westra-engineering.no"), "jorn");
assert_eq!(username_for("Ole.Nordmann+site@Example.com"), "ole.nordmann-site");
assert_eq!(
username_for("Ole.Nordmann+site@Example.com"),
"ole.nordmann-site"
);
assert_eq!(username_for("42@x.no"), "person");
assert_eq!(username_for("Ærlig@x.no"), "rlig");
}
#[test]
fn a_search_by_mail_yields_the_accounts_own_name() {
assert_eq!(name_from_search(r#"{"entries":[{"attrs":{"name":["ole.n"],"mail":["ole@x.no"]}}]}"#), Some("ole.n".into()));
assert_eq!(
name_from_search(r#"{"entries":[{"attrs":{"name":["ole.n"],"mail":["ole@x.no"]}}]}"#),
Some("ole.n".into())
);
assert_eq!(name_from_search(r#"{"entries":[]}"#), None);
assert_eq!(name_from_search("null"), None);
assert_eq!(name_from_search("not json"), None);
@@ -688,21 +908,32 @@ mod tests {
#[test]
fn a_link_runs_out_when_kanidm_says_in_either_spelling() {
assert_eq!(expiry_ms(&serde_json::json!(1790793550)), Some(1790793550000));
assert_eq!(expiry_ms(&serde_json::json!("2026-09-30T12:00:00Z")), Some(1790769600000));
assert_eq!(
expiry_ms(&serde_json::json!(1790793550)),
Some(1790793550000)
);
assert_eq!(
expiry_ms(&serde_json::json!("2026-09-30T12:00:00Z")),
Some(1790769600000)
);
assert_eq!(expiry_ms(&serde_json::json!(null)), None);
}
#[test]
fn members_are_account_names_and_an_empty_group_is_null() {
assert_eq!(members_from(r#"["kari@id.vel.no","ola@id.vel.no"]"#), vec!["kari", "ola"]);
assert_eq!(
members_from(r#"["kari@id.vel.no","ola@id.vel.no"]"#),
vec!["kari", "ola"]
);
assert!(members_from("null").is_empty());
assert!(members_from("nonsense").is_empty());
}
#[test]
fn only_a_listed_credential_counts() {
assert!(has_credentials_from(r#"{"creds":[{"type_":"PasswordMfa"}]}"#));
assert!(has_credentials_from(
r#"{"creds":[{"type_":"PasswordMfa"}]}"#
));
assert!(!has_credentials_from(r#"{"creds":[]}"#));
assert!(!has_credentials_from("null"));
}
@@ -730,24 +961,53 @@ mod tests {
fn a_waiting_invite_is_done_when_they_are_in_and_expired_when_the_link_ran_out() {
let day = LINK_SECONDS * 1000;
// Still within the link's life, and nobody has seen them: wait.
assert_eq!(invite_due(&invite("open", true, Some(5_000)), false, 4_999), None);
assert_eq!(invite_due(&invite("open", true, Some(5_000)), false, 5_000), Some(INVITE_EXPIRED));
assert_eq!(
invite_due(&invite("open", true, Some(5_000)), false, 4_999),
None
);
assert_eq!(
invite_due(&invite("open", true, Some(5_000)), false, 5_000),
Some(INVITE_EXPIRED)
);
// In is in, however long the link had left.
assert_eq!(invite_due(&invite("open", true, Some(5_000)), true, 10), Some(INVITE_DONE));
assert_eq!(invite_due(&invite("open", true, Some(5_000)), true, 9_000), Some(INVITE_DONE));
assert_eq!(
invite_due(&invite("open", true, Some(5_000)), true, 10),
Some(INVITE_DONE)
);
assert_eq!(
invite_due(&invite("open", true, Some(5_000)), true, 9_000),
Some(INVITE_DONE)
);
// A record from before links carried their end: a day from when it was sent.
assert_eq!(invite_due(&invite("open", true, None), false, 1_000 + day - 1), None);
assert_eq!(invite_due(&invite("open", true, None), false, 1_000 + day), Some(INVITE_EXPIRED));
assert_eq!(
invite_due(&invite("open", true, None), false, 1_000 + day - 1),
None
);
assert_eq!(
invite_due(&invite("open", true, None), false, 1_000 + day),
Some(INVITE_EXPIRED)
);
// Someone who had an account was only given a place: nothing to wait for.
assert_eq!(invite_due(&invite("open", false, None), false, 10), Some(INVITE_DONE));
assert_eq!(
invite_due(&invite("open", false, None), false, 10),
Some(INVITE_DONE)
);
// Only a waiting invite moves.
assert_eq!(invite_due(&invite("expired", true, Some(5_000)), false, 9_000), None);
assert_eq!(invite_due(&invite("done", true, Some(5_000)), true, 9_000), None);
assert_eq!(
invite_due(&invite("expired", true, Some(5_000)), false, 9_000),
None
);
assert_eq!(
invite_due(&invite("done", true, Some(5_000)), true, 9_000),
None
);
}
#[test]
fn an_invite_is_theirs_by_account_or_by_address() {
let invited: Invited = serde_json::from_value(invite("open", true, None).responses["invited"].clone()).unwrap();
let invited: Invited =
serde_json::from_value(invite("open", true, None).responses["invited"].clone())
.unwrap();
let user = |username: &str, email: Option<&str>| crate::auth::User {
sub: "s".into(),
username: username.into(),
@@ -766,7 +1026,10 @@ mod tests {
fn a_null_person_answer_means_nobody_there() {
assert_eq!(person_lookup_found(200, "null"), Some(false));
assert_eq!(person_lookup_found(200, "null\n"), Some(false));
assert_eq!(person_lookup_found(200, r#"{"attrs":{"name":["bl"]}}"#), Some(true));
assert_eq!(
person_lookup_found(200, r#"{"attrs":{"name":["bl"]}}"#),
Some(true)
);
assert_eq!(person_lookup_found(404, ""), Some(false));
assert_eq!(person_lookup_found(401, "unauthorized"), None);
}
+435
View File
@@ -0,0 +1,435 @@
//! An Apple configuration profile, rendered from the mail server's own
//! PACC document.
//!
//! iOS has no autoconfiguration for IMAP accounts: Apple Mail reads neither
//! Mozilla autoconfig nor Microsoft Autodiscover (those are for ActiveSync),
//! so adding an account by hand means the device probing for ports. A probe
//! that meets a *filtered* port - a firewall that drops rather than refuses -
//! waits out a TCP timeout and the account is reported as unverifiable, which
//! reads to the person as "wrong password". A profile ends the guessing: it
//! states the hosts and the ports, and carries mail, calendars and contacts in
//! one install.
//!
//! What it does not do is invent any of those values. `site.yaml` names a
//! **PACC** document (`/.well-known/user-agent-configuration.json`, which
//! Stalwart generates and which is defined to live on the mail domain), and
//! every host, port and collection path in the profile comes from there. So
//! the ports are never written down in two places, and a server that moves a
//! collection moves it in the profile too.
//!
//! The profile asks for the address and the password at install time rather
//! than embedding them: it holds nothing personal, so one document serves
//! everyone with an account and it can be fetched by anyone - it says no more
//! than the PACC document it is built from already says in public.
#[cfg(feature = "ssr")]
use crate::content::ProfileConfig;
/// The parts of a PACC document this renders from. Unknown protocols and
/// members are ignored: PACC grows, and a profile only needs what iOS can
/// be told about.
#[cfg(feature = "ssr")]
#[derive(serde::Deserialize, Default)]
struct Pacc {
#[serde(default)]
protocols: Protocols,
}
#[cfg(feature = "ssr")]
#[derive(serde::Deserialize, Default)]
struct Protocols {
#[serde(default)]
imap: Option<HostPort>,
#[serde(default)]
smtp: Option<HostPort>,
#[serde(default)]
caldav: Option<Endpoint>,
#[serde(default)]
carddav: Option<Endpoint>,
}
/// A protocol spoken on a host. PACC may state the port; when it does not,
/// the caller supplies the registered implicit-TLS default.
#[cfg(feature = "ssr")]
#[derive(serde::Deserialize)]
struct HostPort {
host: String,
#[serde(default)]
port: Option<u16>,
}
#[cfg(feature = "ssr")]
#[derive(serde::Deserialize)]
struct Endpoint {
url: String,
}
/// Host, port and path of a DAV collection URL. A collection whose URL
/// cannot be parsed is left out of the profile rather than guessed at.
#[cfg(feature = "ssr")]
fn split_url(url: &str) -> Option<(String, u16, String)> {
let rest = url.strip_prefix("https://")?;
let (authority, path) = match rest.find('/') {
Some(i) => (&rest[..i], &rest[i..]),
None => (rest, "/"),
};
let (host, port) = match authority.rsplit_once(':') {
Some((h, p)) => (h.to_string(), p.parse().ok()?),
None => (authority.to_string(), 443),
};
if host.is_empty() {
return None;
}
Some((host, port, path.to_string()))
}
/// A UUID derived from what the payload is for, not drawn at random: iOS
/// replaces an installed profile whose identifier and UUID it already has, and
/// stacks a duplicate when they differ. Rendering the document twice has to
/// produce the same ids, so reinstalling updates the account instead of adding
/// a second one beside it.
#[cfg(feature = "ssr")]
fn stable_uuid(seed: &str) -> String {
use sha2::{Digest, Sha256};
let h = Sha256::digest(seed.as_bytes());
let b: Vec<String> = h.iter().take(16).map(|x| format!("{x:02X}")).collect();
let s = b.join("");
format!(
"{}-{}-{}-{}-{}",
&s[0..8],
&s[8..12],
&s[12..16],
&s[16..20],
&s[20..32]
)
}
#[cfg(feature = "ssr")]
fn esc(s: &str) -> String {
s.replace('&', "&amp;")
.replace('<', "&lt;")
.replace('>', "&gt;")
}
#[cfg(feature = "ssr")]
fn string(key: &str, value: &str) -> String {
format!(
"\t\t\t<key>{key}</key>\n\t\t\t<string>{}</string>\n",
esc(value)
)
}
#[cfg(feature = "ssr")]
fn integer(key: &str, value: u16) -> String {
format!("\t\t\t<key>{key}</key>\n\t\t\t<integer>{value}</integer>\n")
}
#[cfg(feature = "ssr")]
fn boolean(key: &str, value: bool) -> String {
format!(
"\t\t\t<key>{key}</key>\n\t\t\t<{}/>\n",
if value { "true" } else { "false" }
)
}
/// Renders the profile. `domain` is the site's own host, which seeds the
/// identifiers and names the accounts on the device.
#[cfg(feature = "ssr")]
pub fn render(pacc: &str, domain: &str, cfg: &ProfileConfig) -> anyhow::Result<String> {
let pacc: Pacc = serde_json::from_str(pacc)?;
let name = cfg.name.clone().unwrap_or_else(|| domain.to_string());
let base = format!("{domain}.profile");
let mut payloads = String::new();
let mut count = 0;
// Mail. Both halves have to be there: a profile with an incoming server
// and no outgoing one installs an account that cannot send.
if let (Some(imap), Some(smtp)) = (&pacc.protocols.imap, &pacc.protocols.smtp) {
count += 1;
payloads.push_str("\t\t<dict>\n");
payloads.push_str(&string("PayloadType", "com.apple.mail.managed"));
payloads.push_str(&string("PayloadVersion", "1"));
payloads.push_str(&string("PayloadIdentifier", &format!("{base}.mail")));
payloads.push_str(&string(
"PayloadUUID",
&stable_uuid(&format!("{base}.mail")),
));
payloads.push_str(&string("PayloadDisplayName", &format!("{name} mail")));
payloads.push_str(&string("EmailAccountType", "EmailTypeIMAP"));
payloads.push_str(&string("EmailAccountDescription", &name));
payloads.push_str(&string("IncomingMailServerHostName", &imap.host));
payloads.push_str(&integer(
"IncomingMailServerPortNumber",
imap.port.unwrap_or(993),
));
payloads.push_str(&boolean("IncomingMailServerUseSSL", true));
payloads.push_str(&string(
"IncomingMailServerAuthentication",
"EmailAuthPassword",
));
payloads.push_str(&string("OutgoingMailServerHostName", &smtp.host));
payloads.push_str(&integer(
"OutgoingMailServerPortNumber",
smtp.port.unwrap_or(465),
));
payloads.push_str(&boolean("OutgoingMailServerUseSSL", true));
payloads.push_str(&string(
"OutgoingMailServerAuthentication",
"EmailAuthPassword",
));
// The one password is the same both ways, so the person types it once.
payloads.push_str(&boolean("OutgoingPasswordSameAsIncomingPassword", true));
payloads.push_str("\t\t</dict>\n");
}
for (proto, endpoint, kind, label) in [
(
"caldav",
pacc.protocols.caldav.as_ref(),
"com.apple.caldav.account",
"calendars",
),
(
"carddav",
pacc.protocols.carddav.as_ref(),
"com.apple.carddav.account",
"contacts",
),
] {
let Some((host, port, path)) = endpoint.and_then(|e| split_url(&e.url)) else {
continue;
};
count += 1;
let prefix = if proto == "caldav" {
"CalDAV"
} else {
"CardDAV"
};
payloads.push_str("\t\t<dict>\n");
payloads.push_str(&string("PayloadType", kind));
payloads.push_str(&string("PayloadVersion", "1"));
payloads.push_str(&string("PayloadIdentifier", &format!("{base}.{proto}")));
payloads.push_str(&string(
"PayloadUUID",
&stable_uuid(&format!("{base}.{proto}")),
));
payloads.push_str(&string("PayloadDisplayName", &format!("{name} {label}")));
payloads.push_str(&string(&format!("{prefix}AccountDescription"), &name));
payloads.push_str(&string(&format!("{prefix}HostName"), &host));
payloads.push_str(&integer(&format!("{prefix}Port"), port));
payloads.push_str(&boolean(&format!("{prefix}UseSSL"), true));
payloads.push_str(&string(&format!("{prefix}PrincipalURL"), &path));
payloads.push_str("\t\t</dict>\n");
}
if count == 0 {
anyhow::bail!("PACC document describes no protocol iOS can be configured with");
}
let organization = cfg.organization.clone().unwrap_or_else(|| name.clone());
Ok(format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\
<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n\
<plist version=\"1.0\">\n\
<dict>\n\
\t<key>PayloadType</key>\n\t<string>Configuration</string>\n\
\t<key>PayloadVersion</key>\n\t<integer>1</integer>\n\
\t<key>PayloadIdentifier</key>\n\t<string>{base}</string>\n\
\t<key>PayloadUUID</key>\n\t<string>{uuid}</string>\n\
\t<key>PayloadDisplayName</key>\n\t<string>{display}</string>\n\
\t<key>PayloadOrganization</key>\n\t<string>{org}</string>\n\
\t<key>PayloadDescription</key>\n\t<string>{desc}</string>\n\
\t<key>PayloadRemovalDisallowed</key>\n\t<false/>\n\
\t<key>PayloadContent</key>\n\t<array>\n{payloads}\t</array>\n\
</dict>\n</plist>\n",
base = esc(&base),
uuid = stable_uuid(&base),
display = esc(&name),
org = esc(&organization),
desc = esc(&format!(
"Sets up {name} on this device. You will be asked for your address and password."
)),
payloads = payloads,
))
}
/// `GET /profile.mobileconfig` - the profile for this site, or 404 when
/// `site.yaml` declares no `profile:`. Served as
/// `application/x-apple-aspen-config`, without which iOS downloads the file
/// instead of offering to install it.
#[cfg(feature = "ssr")]
pub async fn handler(
axum::extract::State(state): axum::extract::State<crate::server::AppState>,
) -> axum::response::Response {
use axum::http::{header, StatusCode};
use axum::response::IntoResponse;
let site = state.site.load();
let Some(cfg) = site.profile.clone() else {
return (
StatusCode::NOT_FOUND,
"this site offers no configuration profile",
)
.into_response();
};
let doc = match crate::http::client()
.get(&cfg.pacc)
.send()
.await
.and_then(|r| r.error_for_status())
{
Ok(r) => r,
Err(e) => {
tracing::warn!("profile: fetching {}: {e}", cfg.pacc);
return (StatusCode::BAD_GATEWAY, "cannot reach the mail server").into_response();
}
};
let body = match crate::http::bytes(doc, crate::http::ASSET_CAP).await {
Ok(b) => b,
Err(e) => return (StatusCode::BAD_GATEWAY, e.to_string()).into_response(),
};
let text = match String::from_utf8(body) {
Ok(t) => t,
Err(_) => return (StatusCode::BAD_GATEWAY, "PACC document is not UTF-8").into_response(),
};
let domain = crate::mail::public_url();
let domain = domain.trim_start_matches("https://").trim_end_matches('/');
match render(&text, domain, &cfg) {
Ok(plist) => (
[
(header::CONTENT_TYPE, "application/x-apple-aspen-config"),
(header::CACHE_CONTROL, "no-cache"),
],
plist,
)
.into_response(),
Err(e) => {
tracing::warn!("profile: {e}");
(StatusCode::BAD_GATEWAY, e.to_string()).into_response()
}
}
}
#[cfg(all(test, feature = "ssr"))]
mod tests {
use super::*;
/// Stalwart 0.16's own PACC document, as served. Note what it does *not*
/// say: imap and smtp carry a host and no port, which is why the renderer
/// has to supply the registered implicit-TLS defaults.
const PACC: &str = r#"{"protocols":{"jmap":{"url":"https://mail.example.no/jmap/session"},
"imap":{"host":"mail.example.no"},"pop3":{"host":"mail.example.no"},
"smtp":{"host":"mail.example.no"},"caldav":{"url":"https://mail.example.no/dav/cal/"},
"carddav":{"url":"https://mail.example.no/dav/card/"},
"webdav":{"url":"https://mail.example.no/dav/file/"},
"managesieve":{"host":"mail.example.no"}},
"authentication":{"oauth-public":{"issuer":"https://mail.example.no"},"password":true},
"info":{"provider":{"name":"Stalwart"}}}"#;
fn cfg() -> ProfileConfig {
ProfileConfig {
pacc: "https://mail.example.no/.well-known/user-agent-configuration.json".into(),
name: Some("Example".into()),
organization: None,
}
}
#[test]
fn every_value_comes_from_the_pacc_document() {
let p = render(PACC, "example.no", &cfg()).unwrap();
assert!(p.contains("<string>com.apple.mail.managed</string>"));
assert!(p.contains("<string>com.apple.caldav.account</string>"));
assert!(p.contains("<string>com.apple.carddav.account</string>"));
assert!(p.contains(
"<key>IncomingMailServerHostName</key>\n\t\t\t<string>mail.example.no</string>"
));
// The ports PACC left out, filled in from the registry, not invented.
assert!(p.contains("<key>IncomingMailServerPortNumber</key>\n\t\t\t<integer>993</integer>"));
assert!(p.contains("<key>OutgoingMailServerPortNumber</key>\n\t\t\t<integer>465</integer>"));
// A DAV URL becomes a host, a port and the collection's own path.
assert!(p.contains("<key>CalDAVHostName</key>\n\t\t\t<string>mail.example.no</string>"));
assert!(p.contains("<key>CalDAVPrincipalURL</key>\n\t\t\t<string>/dav/cal/</string>"));
assert!(p.contains("<key>CardDAVPrincipalURL</key>\n\t\t\t<string>/dav/card/</string>"));
assert!(p.contains("<key>CalDAVPort</key>\n\t\t\t<integer>443</integer>"));
}
/// Nothing personal in the document: no address and no username, so iOS
/// asks at install and one profile serves everyone with an account.
#[test]
fn the_profile_holds_no_credentials() {
let p = render(PACC, "example.no", &cfg()).unwrap();
for key in [
"EmailAddress",
"IncomingMailServerUsername",
"OutgoingMailServerUsername",
"CalDAVUsername",
"CardDAVUsername",
"CalDAVPassword",
"Password",
] {
assert!(
!p.contains(&format!("<key>{key}</key>")),
"{key} must not be in the profile"
);
}
}
/// Rendering twice has to give the same ids, or a reinstall adds a second
/// account beside the first instead of replacing it.
#[test]
fn identifiers_are_stable_across_renders() {
let a = render(PACC, "example.no", &cfg()).unwrap();
let b = render(PACC, "example.no", &cfg()).unwrap();
assert_eq!(a, b);
// ...and belong to this site alone.
let other = render(PACC, "other.no", &cfg()).unwrap();
assert_ne!(a, other);
}
/// A PACC document that describes a port explicitly wins over the default.
#[test]
fn a_stated_port_is_used() {
let pacc = r#"{"protocols":{"imap":{"host":"m.example.no","port":1993},
"smtp":{"host":"m.example.no","port":1465}}}"#;
let p = render(pacc, "example.no", &cfg()).unwrap();
assert!(p.contains("<integer>1993</integer>"));
assert!(p.contains("<integer>1465</integer>"));
}
/// Half a mail account is worse than none: an account that cannot send is
/// not installed, and a document with nothing iOS understands is an error
/// rather than an empty profile that looks like it worked.
#[test]
fn an_incomplete_document_is_refused() {
let incoming_only = r#"{"protocols":{"imap":{"host":"m.example.no"}}}"#;
assert!(render(incoming_only, "example.no", &cfg()).is_err());
let jmap_only = r#"{"protocols":{"jmap":{"url":"https://m.example.no/jmap/session"}}}"#;
assert!(render(jmap_only, "example.no", &cfg()).is_err());
}
/// A name with an ampersand in it must not break the plist.
#[test]
fn names_are_escaped() {
let c = ProfileConfig {
name: Some("Smith & Co <mail>".into()),
..cfg()
};
let p = render(PACC, "example.no", &c).unwrap();
assert!(p.contains("Smith &amp; Co &lt;mail&gt;"));
assert!(!p.contains("Smith & Co"));
}
#[test]
fn a_dav_url_that_cannot_be_parsed_is_left_out() {
assert_eq!(
split_url("https://m.example.no:8443/dav/cal/"),
Some(("m.example.no".into(), 8443, "/dav/cal/".into()))
);
assert_eq!(
split_url("https://m.example.no"),
Some(("m.example.no".into(), 443, "/".into()))
);
assert_eq!(split_url("http://m.example.no/dav/cal/"), None);
assert_eq!(split_url("https://"), None);
}
}
+21 -6
View File
@@ -23,14 +23,19 @@ pub struct Report {
}
pub async fn gather(state: &crate::server::AppState) -> Report {
let mut report = Report { storylines: state.questions.load().arity(), buckets: BTreeMap::new() };
let mut report = Report {
storylines: state.questions.load().arity(),
buckets: BTreeMap::new(),
};
let aggregates = state.aggregates.load();
for bucket in aggregates.keys() {
// The runtime's own buckets are not the site's offers.
if bucket.starts_with("portal_") {
continue;
}
let Ok(store) = state.jetstream.get_key_value(bucket).await else { continue };
let Ok(store) = state.jetstream.get_key_value(bucket).await else {
continue;
};
let keys: Vec<String> = {
use futures::TryStreamExt;
match store.keys().await {
@@ -40,14 +45,24 @@ pub async fn gather(state: &crate::server::AppState) -> Report {
};
let counts = report.buckets.entry(bucket.clone()).or_default();
for key in keys {
let Ok(Some(bytes)) = store.get(&key).await else { continue };
let Ok(answer) = serde_json::from_slice::<crate::answers::Answer>(&bytes) else { continue };
*counts.entry(answer.storyline.to_string()).or_default().entry(answer.state).or_default() += 1;
let Ok(Some(bytes)) = store.get(&key).await else {
continue;
};
let Ok(answer) = serde_json::from_slice::<crate::answers::Answer>(&bytes) else {
continue;
};
*counts
.entry(answer.storyline.to_string())
.or_default()
.entry(answer.state)
.or_default() += 1;
}
}
report
}
pub async fn handler(axum::extract::State(state): axum::extract::State<crate::server::AppState>) -> axum::Json<Report> {
pub async fn handler(
axum::extract::State(state): axum::extract::State<crate::server::AppState>,
) -> axum::Json<Report> {
axum::Json(gather(&state).await)
}
+179 -48
View File
@@ -27,13 +27,18 @@ pub async fn get_resource(
// `Resource::refetch()` after a fully-empty-params call failed with
// "missing field `params`", even though the very first load (always
// server-resolved) never did.
#[server(default)]
params: std::collections::HashMap<String, String>,
#[server(default)] params: std::collections::HashMap<String, String>,
) -> Result<serde_json::Value, ServerFnError> {
use crate::server::AppState;
let state = expect_context::<AppState>();
let feature = find_feature(&state, crate::server::storyline::current().await, &question_id, &alternative, &feature_name)?;
let feature = find_feature(
&state,
crate::server::storyline::current().await,
&question_id,
&alternative,
&feature_name,
)?;
let resource = feature
.resource
.as_ref()
@@ -56,13 +61,18 @@ pub async fn get_requirement_options(
feature_name: String,
requirement_name: String,
// See get_resource's `params` for why this is needed.
#[server(default)]
params: std::collections::HashMap<String, String>,
#[server(default)] params: std::collections::HashMap<String, String>,
) -> Result<serde_json::Value, ServerFnError> {
use crate::server::AppState;
let state = expect_context::<AppState>();
let feature = find_feature(&state, crate::server::storyline::current().await, &question_id, &alternative, &feature_name)?;
let feature = find_feature(
&state,
crate::server::storyline::current().await,
&question_id,
&alternative,
&feature_name,
)?;
let requirement = feature
.requirements
.iter()
@@ -82,7 +92,13 @@ pub async fn get_requirement_options(
.resource
.as_ref()
.ok_or_else(|| ServerFnError::new("requirement has no resource"))?;
authorize_resource(&state, &question_id, &alternative, &format!("{feature_name}|{requirement_name}")).await?;
authorize_resource(
&state,
&question_id,
&alternative,
&format!("{feature_name}|{requirement_name}"),
)
.await?;
fetch_resource_value(&state, resource, &as_the_person_asking(params).await).await
}
@@ -97,13 +113,18 @@ pub async fn get_requirement_binding(
feature_name: String,
requirement_name: String,
// See get_resource's `params` for why this is needed.
#[server(default)]
params: std::collections::HashMap<String, String>,
#[server(default)] params: std::collections::HashMap<String, String>,
) -> Result<serde_json::Value, ServerFnError> {
use crate::server::AppState;
let state = expect_context::<AppState>();
let feature = find_feature(&state, crate::server::storyline::current().await, &question_id, &alternative, &feature_name)?;
let feature = find_feature(
&state,
crate::server::storyline::current().await,
&question_id,
&alternative,
&feature_name,
)?;
let requirement = feature
.requirements
.iter()
@@ -113,7 +134,13 @@ pub async fn get_requirement_binding(
.bind
.as_ref()
.ok_or_else(|| ServerFnError::new("requirement has no binding"))?;
authorize_resource(&state, &question_id, &alternative, &format!("{feature_name}|{requirement_name}|bind")).await?;
authorize_resource(
&state,
&question_id,
&alternative,
&format!("{feature_name}|{requirement_name}|bind"),
)
.await?;
fetch_resource_value(&state, &bind.resource, &as_the_person_asking(params).await).await
}
@@ -138,7 +165,8 @@ fn find_feature(
// also what substitutes the URL segment into the page's resource
// keys - so this lookup is where a `/review/<record>` page's
// feature acquires its record-specific key.
let question = crate::content::resolve_question(state.questions.load().at(storyline), question_id)
let question =
crate::content::resolve_question(state.questions.load().at(storyline), question_id)
.ok_or_else(|| ServerFnError::new("unknown question"))?;
question
.alternatives
@@ -232,9 +260,16 @@ pub(crate) fn matches_filter(
} else {
filter.equals.clone()
};
let wanted = crate::content::template_action(&equals, &serde_json::json!(params)).unwrap_or(equals);
let wanted =
crate::content::template_action(&equals, &serde_json::json!(params)).unwrap_or(equals);
// Addresses compare without case; everything else as written.
let same = |found: &str| if filter.equals.contains("{user.email}") { found.trim().eq_ignore_ascii_case(&wanted) } else { found == wanted };
let same = |found: &str| {
if filter.equals.contains("{user.email}") {
found.trim().eq_ignore_ascii_case(&wanted)
} else {
found == wanted
}
};
let found = value
.get("responses")
.and_then(|r| r.get(&filter.field))
@@ -254,16 +289,24 @@ pub(crate) const USER_EMAIL: &str = "__user_email";
/// The caller's parameters with the session's person put in and
/// anything pretending to be them taken out.
#[cfg(feature = "ssr")]
pub(crate) fn with_person(mut params: std::collections::HashMap<String, String>, user: Option<&crate::auth::User>) -> std::collections::HashMap<String, String> {
pub(crate) fn with_person(
mut params: std::collections::HashMap<String, String>,
user: Option<&crate::auth::User>,
) -> std::collections::HashMap<String, String> {
params.retain(|k, _| !k.starts_with("__"));
if let Some(email) = user.and_then(|u| u.email.as_deref()).filter(|e| !e.trim().is_empty()) {
if let Some(email) = user
.and_then(|u| u.email.as_deref())
.filter(|e| !e.trim().is_empty())
{
params.insert(USER_EMAIL.to_string(), email.trim().to_lowercase());
}
params
}
#[cfg(feature = "ssr")]
async fn as_the_person_asking(params: std::collections::HashMap<String, String>) -> std::collections::HashMap<String, String> {
async fn as_the_person_asking(
params: std::collections::HashMap<String, String>,
) -> std::collections::HashMap<String, String> {
use crate::auth::{User, SESSION_USER_KEY};
let user = match leptos_axum::extract::<tower_sessions::Session>().await {
Ok(session) => session.get::<User>(SESSION_USER_KEY).await.ok().flatten(),
@@ -300,8 +343,8 @@ async fn fetch_resource_value(
.await
.map_err(|e| ServerFnError::new(e.to_string()))?
.ok_or_else(|| ServerFnError::new("resource key not found"))?;
let value: serde_json::Value =
serde_json::from_slice(&bytes).map_err(|e| ServerFnError::new(e.to_string()))?;
let value: serde_json::Value = serde_json::from_slice(&bytes)
.map_err(|e| ServerFnError::new(e.to_string()))?;
// A state filter holds for one record as well as a
// listing, or naming the key would walk around it.
if !shows_state(resource, &value) || !matches_filter(resource, &value, params) {
@@ -322,7 +365,9 @@ async fn fetch_resource_value(
for key in keys {
if let Ok(Some(bytes)) = store.get(&key).await {
if let Ok(value) = serde_json::from_slice::<serde_json::Value>(&bytes) {
if shows_state(resource, &value) && matches_filter(resource, &value, params) {
if shows_state(resource, &value)
&& matches_filter(resource, &value, params)
{
items.push(value);
}
}
@@ -341,7 +386,11 @@ async fn fetch_resource_value(
fetch_gitea_json(state, &format!("/api/v1/users/{username}/starred"), params).await?
}
ResourceSource::GiteaReleases { owner, repo } => {
fetch_gitea_json(state, &format!("/api/v1/repos/{owner}/{repo}/releases"), params)
fetch_gitea_json(
state,
&format!("/api/v1/repos/{owner}/{repo}/releases"),
params,
)
.await?
}
ResourceSource::Url { url } => fetch_url_json(url, params).await?,
@@ -380,7 +429,10 @@ pub const FINISHED_SHOWN: usize = 3;
#[cfg(feature = "ssr")]
fn finished_wanted(params: &std::collections::HashMap<String, String>) -> usize {
params.get(FINISHED_PARAM).and_then(|n| n.parse().ok()).unwrap_or(FINISHED_SHOWN)
params
.get(FINISHED_PARAM)
.and_then(|n| n.parse().ok())
.unwrap_or(FINISHED_SHOWN)
}
/// A desk is for the cases still moving. Every record still open stays;
@@ -389,12 +441,21 @@ fn finished_wanted(params: &std::collections::HashMap<String, String>) -> usize
/// ones to offer. Each finished record is marked, so the page can tell
/// them apart without knowing the state machine.
#[cfg(feature = "ssr")]
pub(crate) fn cap_finished(items: Vec<serde_json::Value>, schema: &crate::aggregates::AggregateSchema, shown: usize) -> Vec<serde_json::Value> {
pub(crate) fn cap_finished(
items: Vec<serde_json::Value>,
schema: &crate::aggregates::AggregateSchema,
shown: usize,
) -> Vec<serde_json::Value> {
let is_final = |v: &serde_json::Value| {
v.get("state").and_then(|s| s.as_str()).is_some_and(|s| schema.has_state(s) && schema.allowed(s).is_empty())
v.get("state")
.and_then(|s| s.as_str())
.is_some_and(|s| schema.has_state(s) && schema.allowed(s).is_empty())
};
let when = |v: &serde_json::Value| {
v.get("decided_ms").and_then(|d| d.as_i64()).or_else(|| v.get("submitted_ms").and_then(|d| d.as_i64())).unwrap_or(0)
v.get("decided_ms")
.and_then(|d| d.as_i64())
.or_else(|| v.get("submitted_ms").and_then(|d| d.as_i64()))
.unwrap_or(0)
};
let (mut finished, mut open): (Vec<_>, Vec<_>) = items.into_iter().partition(is_final);
finished.sort_by_key(|v| std::cmp::Reverse(when(v)));
@@ -591,9 +652,16 @@ pub fn apply_jq(
let input_val = read::parse_single(input_text.as_bytes())
.map_err(|e| ServerFnError::new(format!("internal: re-parsing json for jq: {e:?}")))?;
let program = File { code: filter_src, path: () };
let defs = jaq_core::defs().chain(jaq_std::defs()).chain(jaq_json::defs());
let funs = jaq_core::funs().chain(jaq_std::funs()).chain(jaq_json::funs());
let program = File {
code: filter_src,
path: (),
};
let defs = jaq_core::defs()
.chain(jaq_std::defs())
.chain(jaq_json::defs());
let funs = jaq_core::funs()
.chain(jaq_std::funs())
.chain(jaq_json::funs());
let loader = Loader::new(defs);
let arena = Arena::default();
let modules = loader
@@ -607,7 +675,8 @@ pub fn apply_jq(
let mut outputs = Vec::new();
for result in filter.id.run((ctx, input_val)).map(unwrap_valr) {
let val = result.map_err(|e| ServerFnError::new(format!("jq filter runtime error: {e:?}")))?;
let val =
result.map_err(|e| ServerFnError::new(format!("jq filter runtime error: {e:?}")))?;
let text = val.to_string();
let json: serde_json::Value = serde_json::from_str(&text)
.map_err(|e| ServerFnError::new(format!("internal: parsing jq output: {e}")))?;
@@ -633,16 +702,30 @@ mod tests {
#[test]
fn a_listing_can_be_narrowed_to_the_states_worth_showing() {
let all = spec("source: { kind: kv, bucket: cases }\npublic: true\n");
let decided = spec("source: { kind: kv, bucket: cases }\npublic: true\nstates: [approved, refused]\n");
let decided = spec(
"source: { kind: kv, bucket: cases }\npublic: true\nstates: [approved, refused]\n",
);
for state in ["approved", "refused", "withdrawn", "open"] {
assert!(shows_state(&all, &record(state, serde_json::json!({}))));
}
assert!(shows_state(&decided, &record("approved", serde_json::json!({}))));
assert!(shows_state(&decided, &record("refused", serde_json::json!({}))));
assert!(!shows_state(&decided, &record("withdrawn", serde_json::json!({}))));
assert!(shows_state(
&decided,
&record("approved", serde_json::json!({}))
));
assert!(shows_state(
&decided,
&record("refused", serde_json::json!({}))
));
assert!(!shows_state(
&decided,
&record("withdrawn", serde_json::json!({}))
));
// Something that is not an aggregate has no state to filter on
// and is left alone.
assert!(shows_state(&decided, &serde_json::json!({ "name": "a file" })));
assert!(shows_state(
&decided,
&serde_json::json!({ "name": "a file" })
));
}
// The other half of a record belonging to a record: the
@@ -651,9 +734,15 @@ mod tests {
fn a_listing_can_be_narrowed_to_the_children_of_one_record() {
let children = spec("source: { kind: kv, bucket: objections }\nrequires_group: case_officers\nwhere: { field: case, equals: \"{key}\" }\n");
let params = std::collections::HashMap::from([("key".to_string(), "abc123".to_string())]);
let mine = record("open", serde_json::json!({ "case": "abc123", "your_objection": "the sun" }));
let mine = record(
"open",
serde_json::json!({ "case": "abc123", "your_objection": "the sun" }),
);
let theirs = record("open", serde_json::json!({ "case": "def456" }));
let neither = record("open", serde_json::json!({ "your_objection": "no case named" }));
let neither = record(
"open",
serde_json::json!({ "your_objection": "no case named" }),
);
assert!(matches_filter(&children, &mine, &params));
assert!(!matches_filter(&children, &theirs, &params));
// A record that did not say is not everyone's child.
@@ -663,13 +752,25 @@ mod tests {
let own = spec("source: { kind: kv, bucket: cases }\nrequires_group: customers\nwhere: { field: email, equals: \"{user.email}\" }\n");
let hers = serde_json::json!({ "responses": { "email": "Kari@Vel.no" } });
let his = serde_json::json!({ "responses": { "email": "ola@vel.no" } });
let kari = crate::auth::User { sub: "s".into(), username: "kari".into(), display_name: "Kari".into(), groups: vec![], phone: None, email: Some("kari@vel.no".into()) };
let forged: std::collections::HashMap<String, String> = [(USER_EMAIL.to_string(), "ola@vel.no".to_string())].into_iter().collect();
let kari = crate::auth::User {
sub: "s".into(),
username: "kari".into(),
display_name: "Kari".into(),
groups: vec![],
phone: None,
email: Some("kari@vel.no".into()),
};
let forged: std::collections::HashMap<String, String> =
[(USER_EMAIL.to_string(), "ola@vel.no".to_string())]
.into_iter()
.collect();
let asked = with_person(forged.clone(), Some(&kari));
assert!(matches_filter(&own, &hers, &asked) && !matches_filter(&own, &his, &asked));
// Nobody signed in: the forged parameter is dropped, and nothing matches.
let anonymous = with_person(forged, None);
assert!(!matches_filter(&own, &hers, &anonymous) && !matches_filter(&own, &his, &anonymous));
assert!(
!matches_filter(&own, &hers, &anonymous) && !matches_filter(&own, &his, &anonymous)
);
// No filter, no narrowing: every resource written before this
// existed still lists its whole bucket.
let plain = spec("source: { kind: kv, bucket: objections }\npublic: true\n");
@@ -680,8 +781,22 @@ mod tests {
fn a_filter_can_name_a_value_outright() {
let fixed = spec("source: { kind: kv, bucket: cases }\npublic: true\nwhere: { field: which_authority, equals: The county }\n");
let params = std::collections::HashMap::new();
assert!(matches_filter(&fixed, &record("open", serde_json::json!({ "which_authority": "The county" })), &params));
assert!(!matches_filter(&fixed, &record("open", serde_json::json!({ "which_authority": "Fire service" })), &params));
assert!(matches_filter(
&fixed,
&record(
"open",
serde_json::json!({ "which_authority": "The county" })
),
&params
));
assert!(!matches_filter(
&fixed,
&record(
"open",
serde_json::json!({ "which_authority": "Fire service" })
),
&params
));
}
/// `limit` caps what the page renders, and does it after `jq` so
@@ -702,12 +817,19 @@ mod tests {
let shaped = apply_jq(filter, &releases).expect("filter runs");
let all = apply_limit(None, shaped.clone());
assert_eq!(all.as_array().map(Vec::len), Some(3), "no limit shows everything");
assert_eq!(
all.as_array().map(Vec::len),
Some(3),
"no limit shows everything"
);
let latest = apply_limit(Some(1), shaped.clone());
let rows = latest.as_array().expect("still a list");
assert_eq!(rows.len(), 1);
assert_eq!(rows[0]["tag"], "v0.1.17", "and it is the newest, which Gitea returns first");
assert_eq!(
rows[0]["tag"], "v0.1.17",
"and it is the newest, which Gitea returns first"
);
let two = apply_limit(Some(2), shaped.clone());
assert_eq!(two.as_array().map(Vec::len), Some(2));
@@ -823,7 +945,11 @@ mod tests {
assert_eq!(items[0]["name"], "varde bring-up");
assert_eq!(items[0]["tag"], "v0.3.0");
assert_eq!(items[0]["description"], "Embedded varde-core on iOS.");
assert_eq!(items[0]["url"], serde_json::Value::Null, "no public link while private");
assert_eq!(
items[0]["url"],
serde_json::Value::Null,
"no public link while private"
);
assert!(items[0].get("html_url").is_none());
}
@@ -845,8 +971,7 @@ mod tests {
("path".to_string(), "questions/a b.yaml".to_string()),
("ref".to_string(), "main".to_string()),
]);
let (url, leftover) =
template_url("https://x.example/api/contents/{path}", &params);
let (url, leftover) = template_url("https://x.example/api/contents/{path}", &params);
// `/` survives, the space percent-encodes, and the param that
// matched a placeholder is consumed.
assert_eq!(url, "https://x.example/api/contents/questions/a%20b.yaml");
@@ -878,8 +1003,14 @@ mod finished_tests {
.unwrap();
let rec = |id: &str, state: &str, decided: i64| serde_json::json!({ "id": id, "state": state, "submitted_ms": 0, "decided_ms": decided });
let mut items = vec![rec("o1", "open", 0), rec("o2", "open", 0)];
items.extend((1..=6).map(|n| rec(&format!("f{n}"), if n % 2 == 0 { "won" } else { "lost" }, n)));
let ids = |v: &[serde_json::Value]| v.iter().map(|x| x["id"].as_str().unwrap().to_string()).collect::<Vec<_>>();
items.extend(
(1..=6).map(|n| rec(&format!("f{n}"), if n % 2 == 0 { "won" } else { "lost" }, n)),
);
let ids = |v: &[serde_json::Value]| {
v.iter()
.map(|x| x["id"].as_str().unwrap().to_string())
.collect::<Vec<_>>()
};
let shown = cap_finished(items.clone(), &schema, 3);
// Both open cases, then the three newest finished and one more
+195 -42
View File
@@ -58,10 +58,15 @@ pub const SEED_BUCKET: &str = "portal_seed";
pub fn most_privileged_group(questions: &HashMap<String, Question>) -> Option<String> {
let mut reach: BTreeMap<&str, BTreeSet<&str>> = BTreeMap::new();
for q in questions.values() {
let Some(group) = q.qualifies.as_deref() else { continue };
let Some(group) = q.qualifies.as_deref() else {
continue;
};
for alt in &q.alternatives {
if let Some(invite) = &alt.invite {
reach.entry(group).or_default().insert(invite.group.as_str());
reach
.entry(group)
.or_default()
.insert(invite.group.as_str());
}
}
}
@@ -73,7 +78,19 @@ pub fn most_privileged_group(questions: &HashMap<String, Question>) -> Option<St
/// The bucket key for one seed: stable, and within NATS key rules.
pub fn seed_key(email: &str, group: &str) -> String {
let clean = |s: &str| s.trim().to_lowercase().chars().map(|c| if c.is_ascii_alphanumeric() || c == '-' || c == '_' { c } else { '_' }).collect::<String>();
let clean = |s: &str| {
s.trim()
.to_lowercase()
.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '-' || c == '_' {
c
} else {
'_'
}
})
.collect::<String>()
};
format!("{}.{}", clean(group), clean(email))
}
@@ -82,7 +99,9 @@ pub fn seed_key(email: &str, group: &str) -> String {
/// questions they answer for, in page order. A page a group answers
/// for (`responsible.group`) names nobody: the board's inbox is not a
/// person, and its members have their accounts through the desk.
pub fn responsible_people(questions: &HashMap<String, Question>) -> BTreeMap<String, (String, Vec<String>)> {
pub fn responsible_people(
questions: &HashMap<String, Question>,
) -> BTreeMap<String, (String, Vec<String>)> {
let mut ids: Vec<&String> = questions.keys().collect();
ids.sort();
let mut out: BTreeMap<String, (String, Vec<String>)> = BTreeMap::new();
@@ -94,7 +113,9 @@ pub fn responsible_people(questions: &HashMap<String, Question>) -> BTreeMap<Str
}
let email = contact_email(&r.contact);
let Some(email) = email else { continue };
let entry = out.entry(email).or_insert_with(|| (r.name.trim().to_string(), Vec::new()));
let entry = out
.entry(email)
.or_insert_with(|| (r.name.trim().to_string(), Vec::new()));
if !entry.1.contains(&q.name) {
entry.1.push(q.name.clone());
}
@@ -105,7 +126,11 @@ pub fn responsible_people(questions: &HashMap<String, Question>) -> BTreeMap<Str
/// A responsible `contact` as an address: `mailto:` off, lowercased;
/// `None` for a phone number or anything else that is not one.
pub fn contact_email(contact: &str) -> Option<String> {
let c = contact.trim().trim_start_matches("mailto:").trim().to_lowercase();
let c = contact
.trim()
.trim_start_matches("mailto:")
.trim()
.to_lowercase();
(c.contains('@') && !c.contains(char::is_whitespace)).then_some(c)
}
@@ -126,7 +151,10 @@ static PASS: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
pub async fn run(state: crate::server::AppState) {
// Subscribed before the first pass, so content that changes while
// it runs is not missed.
let reloads = state.nats.subscribe(crate::content::CONTENT_RELOADED_SUBJECT).await;
let reloads = state
.nats
.subscribe(crate::content::CONTENT_RELOADED_SUBJECT)
.await;
tokio::join!(seed_admin(&state), responsible_until_done(&state), async {
let mut reloads = match reloads {
Ok(reloads) => reloads,
@@ -158,7 +186,11 @@ async fn responsible_until_done(state: &crate::server::AppState) {
tracing::error!(left, "some responsible people could not be set up, for good - until the content changes or portal starts again");
return;
}
tracing::warn!(left, attempt, "some responsible people could not be set up, trying again in a minute");
tracing::warn!(
left,
attempt,
"some responsible people could not be set up, trying again in a minute"
);
tokio::time::sleep(APART).await;
}
}
@@ -166,7 +198,14 @@ async fn responsible_until_done(state: &crate::server::AppState) {
async fn seed_bucket(state: &crate::server::AppState) -> Option<async_nats::jetstream::kv::Store> {
match state.jetstream.get_key_value(SEED_BUCKET).await {
Ok(store) => Some(store),
Err(_) => match state.jetstream.create_key_value(async_nats::jetstream::kv::Config { bucket: SEED_BUCKET.to_string(), ..Default::default() }).await {
Err(_) => match state
.jetstream
.create_key_value(async_nats::jetstream::kv::Config {
bucket: SEED_BUCKET.to_string(),
..Default::default()
})
.await
{
Ok(store) => Some(store),
Err(e) => {
tracing::error!("the seed bucket could not be made: {e} - making no accounts, so as not to make them twice");
@@ -177,7 +216,10 @@ async fn seed_bucket(state: &crate::server::AppState) -> Option<async_nats::jets
}
fn seed_admin_email() -> Option<String> {
std::env::var("SEED_ADMIN_EMAIL").ok().map(|e| e.trim().to_lowercase()).filter(|e| e.contains('@'))
std::env::var("SEED_ADMIN_EMAIL")
.ok()
.map(|e| e.trim().to_lowercase())
.filter(|e| e.contains('@'))
}
/// Who portal put in the responsible group and the content no longer
@@ -204,24 +246,37 @@ pub fn no_longer_responsible(
/// it could not finish with.
async fn responsible(state: &crate::server::AppState) -> usize {
let people = responsible_people(&state.questions.load());
let group = std::env::var("RESPONSIBLE_GROUP").ok().map(|g| g.trim().to_string()).filter(|g| !g.is_empty());
let group = std::env::var("RESPONSIBLE_GROUP")
.ok()
.map(|g| g.trim().to_string())
.filter(|g| !g.is_empty());
if people.is_empty() && group.is_none() {
return 0;
}
let Some(people_backend) = crate::people::People::from_env() else {
if !people.is_empty() {
tracing::info!(people = people.len(), "the content names responsible people, but {} - no accounts made", crate::people::People::NOT_CONFIGURED);
tracing::info!(
people = people.len(),
"the content names responsible people, but {} - no accounts made",
crate::people::People::NOT_CONFIGURED
);
}
return 0;
};
let Some(store) = seed_bucket(state).await else { return 0 };
let Some(store) = seed_bucket(state).await else {
return 0;
};
let mut left = 0;
for (email, (name, questions)) in &people {
let key = seed_key(email, group.as_deref().unwrap_or("responsible"));
if let Ok(Some(_)) = store.get(&key).await {
continue;
}
let name = if name.is_empty() { email.split('@').next().unwrap_or("").to_string() } else { name.clone() };
let name = if name.is_empty() {
email.split('@').next().unwrap_or("").to_string()
} else {
name.clone()
};
let who = crate::memberships::Identity::from_contact(email);
// With a group, one call makes the account if there is none and
// adds them either way; without one, only a missing account is
@@ -229,21 +284,30 @@ async fn responsible(state: &crate::server::AppState) -> usize {
// way a person is mailed only when an account was made for
// them: someone who had one already has their way in.
let made = match (&group, &people_backend) {
(Some(group), people_backend) => people_backend.invite(state, &name, &who, group, "responsible").await,
(None, crate::people::People::Kanidm(kanidm)) => match kanidm.has_person_with_mail(email).await {
(Some(group), people_backend) => {
people_backend
.invite(state, &name, &who, group, "responsible")
.await
}
(None, crate::people::People::Kanidm(kanidm)) => {
match kanidm.has_person_with_mail(email).await {
Ok(true) => {
let _ = store.put(&key, "existing".into()).await;
continue;
}
Ok(false) => kanidm.account(&name, email).await,
Err(e) => Err(e),
},
}
}
(None, crate::people::People::Memberships) => continue,
};
match made {
Ok(invited) => {
if let Ok(bytes) = serde_json::to_vec(&invited) {
let _ = state.nats.publish(crate::people::INVITED_SUBJECT, bytes.clone().into()).await;
let _ = state
.nats
.publish(crate::people::INVITED_SUBJECT, bytes.clone().into())
.await;
let _ = store.put(&key, bytes.into()).await;
}
if invited.created {
@@ -274,14 +338,21 @@ async fn responsible(state: &crate::server::AppState) -> usize {
};
let mut on_file = Vec::new();
for key in keys.into_iter().filter(|k| k.starts_with(&prefix)) {
let Ok(Some(bytes)) = store.get(&key).await else { continue };
let Ok(invited) = serde_json::from_slice::<crate::people::Invited>(&bytes) else { continue };
let Ok(Some(bytes)) = store.get(&key).await else {
continue;
};
let Ok(invited) = serde_json::from_slice::<crate::people::Invited>(&bytes) else {
continue;
};
on_file.push((key, invited));
}
for (key, invited) in no_longer_responsible(&on_file, &people, seed_admin_email().as_deref()) {
if !invited.already_member {
let who = crate::memberships::Identity::from_contact(&invited.email);
if let Err(e) = people_backend.remove(state, &who, &invited.username, &group, "responsible").await {
if let Err(e) = people_backend
.remove(state, &who, &invited.username, &group, "responsible")
.await
{
left += 1;
tracing::warn!(user = %invited.username, group, "could not take someone no longer responsible out of the group: {e}");
continue;
@@ -298,13 +369,24 @@ async fn responsible(state: &crate::server::AppState) -> usize {
/// or `SEED_ADMIN_PHONE`, for a site whose provider knows people by
/// their number.
async fn seed_admin(state: &crate::server::AppState) {
let phone = std::env::var("SEED_ADMIN_PHONE").ok().map(|p| p.trim().to_string()).filter(|p| !p.is_empty());
let email = std::env::var("SEED_ADMIN_EMAIL").ok().map(|e| e.trim().to_string()).filter(|e| e.contains('@'));
let phone = std::env::var("SEED_ADMIN_PHONE")
.ok()
.map(|p| p.trim().to_string())
.filter(|p| !p.is_empty());
let email = std::env::var("SEED_ADMIN_EMAIL")
.ok()
.map(|e| e.trim().to_string())
.filter(|e| e.contains('@'));
let (Some(contact), who) = (
phone.clone().or_else(|| email.clone()),
crate::memberships::Identity {
sub: None,
phone: phone.as_deref().and_then(|p| crate::memberships::normalize_phone(p, &std::env::var("PHONE_COUNTRY").unwrap_or_else(|_| "+47".into()))),
phone: phone.as_deref().and_then(|p| {
crate::memberships::normalize_phone(
p,
&std::env::var("PHONE_COUNTRY").unwrap_or_else(|_| "+47".into()),
)
}),
email: email.as_deref().map(|e| e.to_lowercase()),
},
) else {
@@ -316,7 +398,11 @@ async fn seed_admin(state: &crate::server::AppState) {
.map(|n| n.trim().to_string())
.filter(|n| !n.is_empty())
.unwrap_or_else(|| email.split('@').next().unwrap_or("admin").to_string());
let group = match std::env::var("SEED_GROUP").ok().map(|g| g.trim().to_string()).filter(|g| !g.is_empty()) {
let group = match std::env::var("SEED_GROUP")
.ok()
.map(|g| g.trim().to_string())
.filter(|g| !g.is_empty())
{
Some(g) => g,
None => match most_privileged_group(&state.questions.load()) {
Some(g) => g,
@@ -327,13 +413,22 @@ async fn seed_admin(state: &crate::server::AppState) {
},
};
let key = seed_key(&email, &group);
let Some(store) = seed_bucket(state).await else { return };
let Some(store) = seed_bucket(state).await else {
return;
};
if let Ok(Some(_)) = store.get(&key).await {
tracing::info!(group, "the first person was already invited into this group; not again");
tracing::info!(
group,
"the first person was already invited into this group; not again"
);
return;
}
let Some(people) = crate::people::People::from_env() else {
tracing::warn!(group, "SEED_ADMIN_EMAIL is set, but {} - nobody was seeded", crate::people::People::NOT_CONFIGURED);
tracing::warn!(
group,
"SEED_ADMIN_EMAIL is set, but {} - nobody was seeded",
crate::people::People::NOT_CONFIGURED
);
return;
};
@@ -343,7 +438,11 @@ async fn seed_admin(state: &crate::server::AppState) {
match people.invite(state, &name, &who, &group, "seed").await {
Ok(invited) => break invited,
Err(e) if attempt < ATTEMPTS => {
tracing::warn!(group, attempt, "seeding the first person failed, trying again in a minute: {e}");
tracing::warn!(
group,
attempt,
"seeding the first person failed, trying again in a minute: {e}"
);
tokio::time::sleep(APART).await;
}
Err(e) => {
@@ -353,7 +452,10 @@ async fn seed_admin(state: &crate::server::AppState) {
}
};
if let Ok(bytes) = serde_json::to_vec(&invited) {
let _ = state.nats.publish(crate::people::INVITED_SUBJECT, bytes.clone().into()).await;
let _ = state
.nats
.publish(crate::people::INVITED_SUBJECT, bytes.clone().into())
.await;
if let Err(e) = store.put(&key, bytes.into()).await {
tracing::error!(group, "the first person is invited, but it could not be remembered: {e} - a restart would invite them again");
}
@@ -374,7 +476,17 @@ mod tests {
.iter()
.map(|g| format!(" - name: Invite into {g}\n invite: {{ group: {g} }}\n"))
.collect();
let yaml = format!("name: Q?\n{}alternatives:\n{}", qualifies.map(|q| format!("qualifies: {q}\n")).unwrap_or_default(), if alternatives.is_empty() { " - name: A\n".to_string() } else { alternatives });
let yaml = format!(
"name: Q?\n{}alternatives:\n{}",
qualifies
.map(|q| format!("qualifies: {q}\n"))
.unwrap_or_default(),
if alternatives.is_empty() {
" - name: A\n".to_string()
} else {
alternatives
}
);
let mut q: Question = serde_yaml::from_str(&yaml).unwrap();
q.id = id.to_string();
(id.to_string(), q)
@@ -384,7 +496,11 @@ mod tests {
fn the_first_person_goes_where_the_most_invitations_can_be_made() {
let qs: HashMap<String, Question> = [
page("/", None, &[]),
page("/styret", Some("styret"), &["styret", "kasserer", "trafikk", "komiteer"]),
page(
"/styret",
Some("styret"),
&["styret", "kasserer", "trafikk", "komiteer"],
),
page("/kasserer", Some("kasserer"), &["kasserer"]),
page("/trafikk", Some("trafikk"), &["trafikk"]),
]
@@ -392,7 +508,10 @@ mod tests {
.collect();
assert_eq!(most_privileged_group(&qs).as_deref(), Some("styret"));
// A tie goes to the name that sorts first, every time.
let tie: HashMap<String, Question> = [page("/b", Some("b"), &["x"]), page("/a", Some("a"), &["y"])].into_iter().collect();
let tie: HashMap<String, Question> =
[page("/b", Some("b"), &["x"]), page("/a", Some("a"), &["y"])]
.into_iter()
.collect();
assert_eq!(most_privileged_group(&tie).as_deref(), Some("a"));
// No invitations anywhere: nobody to seed into without SEED_GROUP.
let none: HashMap<String, Question> = [page("/", None, &[])].into_iter().collect();
@@ -419,26 +538,54 @@ mod tests {
#[test]
fn whoever_the_content_stopped_naming_leaves_but_never_the_first_person() {
let people: BTreeMap<String, (String, Vec<String>)> = [("kari@vel.no".to_string(), ("Kari".to_string(), vec!["Hva?".to_string()]))].into_iter().collect();
let filed = vec![on_file("kari@vel.no", false), on_file("Ola@vel.no", false), on_file("leder@vel.no", false)];
let people: BTreeMap<String, (String, Vec<String>)> = [(
"kari@vel.no".to_string(),
("Kari".to_string(), vec!["Hva?".to_string()]),
)]
.into_iter()
.collect();
let filed = vec![
on_file("kari@vel.no", false),
on_file("Ola@vel.no", false),
on_file("leder@vel.no", false),
];
let gone = no_longer_responsible(&filed, &people, Some("leder@vel.no"));
assert_eq!(gone.iter().map(|(_, i)| i.username.as_str()).collect::<Vec<_>>(), vec!["Ola"]);
assert_eq!(
gone.iter()
.map(|(_, i)| i.username.as_str())
.collect::<Vec<_>>(),
vec!["Ola"]
);
assert_eq!(gone[0].0, "redaktor.ola_vel_no");
// Nobody named at all: everyone portal put there leaves.
assert_eq!(no_longer_responsible(&filed, &BTreeMap::new(), None).len(), 3);
assert_eq!(
no_longer_responsible(&filed, &BTreeMap::new(), None).len(),
3
);
}
#[test]
fn everyone_named_responsible_is_found_once_by_their_address() {
let mut qs: HashMap<String, Question> = HashMap::new();
for (id, name, contact) in [("/", "Hva gjør vi?", "mailto:Styret@Vel.no"), ("/melde-fra", "Hva vil du melde?", "styret@vel.no"), ("/b", "Ringe?", "+47 900 00 000"), ("/c", "Trafikk?", "trafikk@vel.no")] {
for (id, name, contact) in [
("/", "Hva gjør vi?", "mailto:Styret@Vel.no"),
("/melde-fra", "Hva vil du melde?", "styret@vel.no"),
("/b", "Ringe?", "+47 900 00 000"),
("/c", "Trafikk?", "trafikk@vel.no"),
] {
let mut q: Question = serde_yaml::from_str(&format!("name: \"{name}\"\nresponsible: {{ name: Styret, contact: \"{contact}\" }}\nalternatives:\n - name: A\n")).unwrap();
q.id = id.to_string();
qs.insert(id.to_string(), q);
}
let people = responsible_people(&qs);
assert_eq!(people.keys().collect::<Vec<_>>(), ["styret@vel.no", "trafikk@vel.no"]);
assert_eq!(people["styret@vel.no"].1, ["Hva gjør vi?", "Hva vil du melde?"]);
assert_eq!(
people.keys().collect::<Vec<_>>(),
["styret@vel.no", "trafikk@vel.no"]
);
assert_eq!(
people["styret@vel.no"].1,
["Hva gjør vi?", "Hva vil du melde?"]
);
assert_eq!(contact_email("+47 900 00 000"), None);
// A group's inbox is nobody: its members have the desk.
let mut held: Question = serde_yaml::from_str("name: \"Hva?\"\nresponsible: { name: Styret, contact: post@vel.no, group: styret }\nalternatives:\n - name: A\n").unwrap();
@@ -449,7 +596,13 @@ mod tests {
#[test]
fn a_seed_is_remembered_by_address_and_group() {
assert_eq!(seed_key(" Kari.Lien@Tomtervel.no ", "styret"), "styret.kari_lien_tomtervel_no");
assert_ne!(seed_key("kari@x.no", "styret"), seed_key("kari@x.no", "kasserer"));
assert_eq!(
seed_key(" Kari.Lien@Tomtervel.no ", "styret"),
"styret.kari_lien_tomtervel_no"
);
assert_ne!(
seed_key("kari@x.no", "styret"),
seed_key("kari@x.no", "kasserer")
);
}
}
+3 -1
View File
@@ -28,7 +28,9 @@ pub struct AppState {
/// hot-swapped alongside `questions` on the same reload. A bucket
/// with no entry here isn't event-sourced - plain KV mutate-in-place
/// still works (see `answers.rs`).
pub aggregates: Arc<ArcSwap<crate::content::Storylines<HashMap<String, crate::aggregates::AggregateSchema>>>>,
pub aggregates: Arc<
ArcSwap<crate::content::Storylines<HashMap<String, crate::aggregates::AggregateSchema>>>,
>,
/// Site branding from the content repo's optional `site.yaml`
/// (title, wordmark, hero kind) - hot-swapped with `questions`/
/// `aggregates` on the same reload. Default = the historical uhhm
+29 -11
View File
@@ -13,8 +13,8 @@ use axum::{
use openidconnect::{
core::{CoreAuthenticationFlow, CoreClient, CoreProviderMetadata, CoreUserInfoClaims},
AuthorizationCode, ClientId, ClientSecret, CsrfToken, EndpointMaybeSet, EndpointNotSet,
EndpointSet, IssuerUrl, Nonce, PkceCodeChallenge, PkceCodeVerifier, RedirectUrl, Scope,
OAuth2TokenResponse, TokenResponse,
EndpointSet, IssuerUrl, Nonce, OAuth2TokenResponse, PkceCodeChallenge, PkceCodeVerifier,
RedirectUrl, Scope, TokenResponse,
};
use serde::Deserialize;
use tower_sessions::Session;
@@ -148,7 +148,10 @@ pub async fn login(
Query(params): Query<LoginParams>,
) -> Result<Redirect, HandlerError> {
if let Some(redirect) = params.redirect.filter(|r| is_safe_redirect(r)) {
session.insert(REDIRECT_KEY, redirect).await.map_err(internal)?;
session
.insert(REDIRECT_KEY, redirect)
.await
.map_err(internal)?;
}
let oidc = state.oidc.configured()?;
@@ -168,7 +171,10 @@ pub async fn login(
// What else to ask the provider for: `phoneNumber` for one that
// verifies numbers (Vipps). Only when asked, because Kanidm denies
// a sign-in that requests a scope the person does not hold.
for scope in std::env::var("OIDC_EXTRA_SCOPES").unwrap_or_default().split_whitespace() {
for scope in std::env::var("OIDC_EXTRA_SCOPES")
.unwrap_or_default()
.split_whitespace()
{
request = request.add_scope(Scope::new(scope.to_string()));
}
let (auth_url, csrf_state, nonce) = request.url();
@@ -206,8 +212,7 @@ pub async fn callback(
let pkce_verifier: Option<String> = session.remove(PKCE_KEY).await.map_err(internal)?;
let nonce: Option<String> = session.remove(NONCE_KEY).await.map_err(internal)?;
let (Some(stored_csrf), Some(pkce_verifier), Some(nonce)) =
(stored_csrf, pkce_verifier, nonce)
let (Some(stored_csrf), Some(pkce_verifier), Some(nonce)) = (stored_csrf, pkce_verifier, nonce)
else {
return Err((
StatusCode::BAD_REQUEST,
@@ -262,12 +267,19 @@ pub async fn callback(
// The verified phone number lives at userinfo, not in the ID token,
// for providers that give one (Vipps). A provider that has none, or
// no userinfo endpoint, leaves it empty; nothing depends on it.
let phone = match oidc.client.user_info(token_response.access_token().clone(), None) {
Ok(request) => request
let phone = match oidc
.client
.user_info(token_response.access_token().clone(), None)
{
Ok(request) => {
request
.request_async(&oidc.http)
.await
.ok()
.and_then(|info: CoreUserInfoClaims| info.phone_number().map(|p| p.as_str().to_string())),
.and_then(|info: CoreUserInfoClaims| {
info.phone_number().map(|p| p.as_str().to_string())
})
}
Err(_) => None,
};
let mut user = User {
@@ -295,7 +307,9 @@ pub async fn callback(
.map_err(internal)?;
let redirect: Option<String> = session.remove(REDIRECT_KEY).await.map_err(internal)?;
let target = redirect.filter(|r| is_safe_redirect(r)).unwrap_or_else(|| "/".to_string());
let target = redirect
.filter(|r| is_safe_redirect(r))
.unwrap_or_else(|| "/".to_string());
// Being here is what an invite was for (people.rs): off the
// request, since nobody signing in should wait on a desk's list.
@@ -338,6 +352,10 @@ fn extract_groups_claim<T: serde::Serialize>(id_token: &T) -> Vec<String> {
payload
.get("groups")
.and_then(|g| g.as_array())
.map(|arr| arr.iter().filter_map(|v| v.as_str().map(String::from)).collect())
.map(|arr| {
arr.iter()
.filter_map(|v| v.as_str().map(String::from))
.collect()
})
.unwrap_or_default()
}
+20 -5
View File
@@ -21,7 +21,10 @@ fn from_cookies(header: Option<&str>, arity: usize) -> Option<usize> {
header?
.split(';')
.map(str::trim)
.find_map(|kv| kv.strip_prefix(COOKIE_NAME).and_then(|rest| rest.strip_prefix('=')))
.find_map(|kv| {
kv.strip_prefix(COOKIE_NAME)
.and_then(|rest| rest.strip_prefix('='))
})
.and_then(|v| v.trim().parse::<usize>().ok())
.filter(|n| *n < arity)
}
@@ -31,14 +34,23 @@ fn pick(arity: usize) -> usize {
(uuid::Uuid::new_v4().as_u128() % arity.max(1) as u128) as usize
}
pub async fn assign(State(state): State<super::AppState>, mut req: Request, next: Next) -> Response {
pub async fn assign(
State(state): State<super::AppState>,
mut req: Request,
next: Next,
) -> Response {
let arity = state.questions.load().arity();
let held = from_cookies(req.headers().get(COOKIE).and_then(|v| v.to_str().ok()), arity);
let held = from_cookies(
req.headers().get(COOKIE).and_then(|v| v.to_str().ok()),
arity,
);
let n = held.unwrap_or_else(|| pick(arity));
req.extensions_mut().insert(Storyline(n));
let mut res = next.run(req).await;
if held.is_none() && arity > 1 {
if let Ok(value) = format!("{COOKIE_NAME}={n}; Path=/; Max-Age=31536000; SameSite=Lax").parse() {
if let Ok(value) =
format!("{COOKIE_NAME}={n}; Path=/; Max-Age=31536000; SameSite=Lax").parse()
{
res.headers_mut().append(SET_COOKIE, value);
}
}
@@ -60,7 +72,10 @@ mod tests {
#[test]
fn a_cookie_names_a_storyline_the_site_tells() {
assert_eq!(from_cookies(Some("a=b; portal_storyline=1; c=d"), 3), Some(1));
assert_eq!(
from_cookies(Some("a=b; portal_storyline=1; c=d"), 3),
Some(1)
);
assert_eq!(from_cookies(Some("portal_storyline=7"), 3), None);
assert_eq!(from_cookies(Some("portal_storylines=1"), 3), None);
assert_eq!(from_cookies(None, 3), None);
+8 -2
View File
@@ -37,7 +37,9 @@ pub struct NatsStore {
impl std::fmt::Debug for NatsStore {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("NatsStore").field("bucket", &BUCKET).finish()
f.debug_struct("NatsStore")
.field("bucket", &BUCKET)
.finish()
}
}
@@ -73,7 +75,11 @@ impl tower_sessions::SessionStore for NatsStore {
async fn create(&self, record: &mut Record) -> Result<()> {
for _ in 0..ID_ATTEMPTS {
let bytes = Self::encode(record)?;
match self.store.create(&record.id.to_string(), bytes.into()).await {
match self
.store
.create(&record.id.to_string(), bytes.into())
.await
{
Ok(_) => return Ok(()),
Err(_) => record.id = Id::default(),
}
+1580 -355
View File
File diff suppressed because it is too large Load Diff
+34 -7
View File
@@ -27,23 +27,42 @@ pub fn alternative(alt: &Value, full: bool) -> String {
out.push_str(&format!(": {d}"));
}
out.push('\n');
if let Some(button) = alt["consequence"].as_array().and_then(|c| c.first()).and_then(Value::as_str) {
if let Some(button) = alt["consequence"]
.as_array()
.and_then(|c| c.first())
.and_then(Value::as_str)
{
out.push_str(&format!(" Button: {button}\n"));
}
for e in alt["encouragements"].as_array().into_iter().flatten().filter_map(Value::as_str) {
for e in alt["encouragements"]
.as_array()
.into_iter()
.flatten()
.filter_map(Value::as_str)
{
out.push_str(&format!(" {e}\n"));
}
for f in alt["features"].as_array().into_iter().flatten() {
let (name, desc, icon) = (s(&f["name"]).trim(), s(&f["description"]).trim(), s(&f["icon"]).trim());
let (name, desc, icon) = (
s(&f["name"]).trim(),
s(&f["description"]).trim(),
s(&f["icon"]).trim(),
);
let mut line = String::from(" ");
if !icon.is_empty() {
line.push_str(&format!("[icon {}] ", icon.rsplit(':').next().unwrap_or(icon)));
line.push_str(&format!(
"[icon {}] ",
icon.rsplit(':').next().unwrap_or(icon)
));
}
if !name.is_empty() {
line.push_str(name);
}
if !desc.is_empty() {
line.push_str(&format!("{}{desc}", if name.is_empty() { "" } else { ": " }));
line.push_str(&format!(
"{}{desc}",
if name.is_empty() { "" } else { ": " }
));
}
if !line.trim().is_empty() {
out.push_str(line.trim_end());
@@ -56,8 +75,16 @@ pub fn alternative(alt: &Value, full: bool) -> String {
.map(|r| {
let label = r["label"].as_str().unwrap_or_else(|| s(&r["name"]));
let hint = s(&r["placeholder"]).trim();
let opt = if r["optional"] == true { " (optional)" } else { "" };
if hint.is_empty() { format!("{label}{opt}") } else { format!("{label}{opt} e.g. {hint}") }
let opt = if r["optional"] == true {
" (optional)"
} else {
""
};
if hint.is_empty() {
format!("{label}{opt}")
} else {
format!("{label}{opt} e.g. {hint}")
}
})
.collect();
if !fields.is_empty() {
+228 -49
View File
@@ -74,7 +74,15 @@ pub fn run(argv: Vec<String>) -> anyhow::Result<()> {
// sampling, so a score comes with its spread. 0 is the run
// every earlier score used.
let seed: u64 = value("--seed").and_then(|s| s.parse().ok()).unwrap_or(0);
print!("{}", answer_blind(&cmd, &std::fs::read_to_string(&tasks).with_context(|| format!("reading {tasks}"))?, repo.as_deref(), seed)?);
print!(
"{}",
answer_blind(
&cmd,
&std::fs::read_to_string(&tasks).with_context(|| format!("reading {tasks}"))?,
repo.as_deref(),
seed
)?
);
return Ok(());
}
let opts = parse_args()?;
@@ -103,11 +111,16 @@ pub fn run(argv: Vec<String>) -> anyhow::Result<()> {
std::fs::write(opts.out.join("needs.yaml"), needs)?;
// A placeholder front door, so the lint gets as far as parsing
// the spec; every need fails, a parse error is what we look for.
std::fs::write(opts.out.join("questions/index.yaml"), "name: Placeholder?\n")?;
std::fs::write(
opts.out.join("questions/index.yaml"),
"name: Placeholder?\n",
)?;
let (_, report) = lint(&opts, &[])?;
if let Some(line) = report.lines().find(|l| l.contains("parsing needs.yaml")) {
eprintln!("needs round {round}: {line}");
feedback = format!("\n## Your last attempt failed to parse\n\n{line}\n\n```yaml\n{needs}\n```\n");
feedback = format!(
"\n## Your last attempt failed to parse\n\n{line}\n\n```yaml\n{needs}\n```\n"
);
continue;
}
eprintln!("needs round {round}: needs.yaml parses, frozen");
@@ -116,9 +129,14 @@ pub fn run(argv: Vec<String>) -> anyhow::Result<()> {
break;
}
if !feedback.is_empty() {
bail!("needs.yaml never parsed in {} round(s); the replies are in {}/llm/", opts.rounds, opts.out.display());
bail!(
"needs.yaml never parsed in {} round(s); the replies are in {}/llm/",
opts.rounds,
opts.out.display()
);
}
let needs = std::fs::read_to_string(opts.out.join("needs.yaml")).context("no needs.yaml was produced")?;
let needs = std::fs::read_to_string(opts.out.join("needs.yaml"))
.context("no needs.yaml was produced")?;
// Phase 2: structure. The page-writing model sees the needs and
// never the personas: pages written with the test in view route
@@ -165,12 +183,22 @@ pub fn run(argv: Vec<String>) -> anyhow::Result<()> {
Ok(files) => apply(&opts.out, files)?,
Err(_) => eprintln!("structure round {round}: the reply had no file blocks"),
}
let (ok, r, found) = lint_findings(&opts, &opts.out.join(format!("findings/round-{round}.jsonl")))?;
let (ok, r, found) = lint_findings(
&opts,
&opts.out.join(format!("findings/round-{round}.jsonl")),
)?;
report = r;
let Some(found) = found else {
tracked = false;
let fails = report.lines().filter(|l| l.starts_with("FAIL")).count();
eprintln!("structure round {round}: {}", if ok { "green".to_string() } else { format!("{fails} FAIL line(s)") });
eprintln!(
"structure round {round}: {}",
if ok {
"green".to_string()
} else {
format!("{fails} FAIL line(s)")
}
);
log.push(format!("structure round {round}: ok={ok} fails={fails}"));
if ok {
green = true;
@@ -179,8 +207,14 @@ pub fn run(argv: Vec<String>) -> anyhow::Result<()> {
continue;
};
let previous = ledger.clone();
let progress = ledger.update(round, found.into_iter().filter(|f| f.level == "fail").collect());
let line = format!("structure round {round}: {}", progress.summary(ledger.open.len()));
let progress = ledger.update(
round,
found.into_iter().filter(|f| f.level == "fail").collect(),
);
let line = format!(
"structure round {round}: {}",
progress.summary(ledger.open.len())
);
eprintln!("{line}");
log.push(line);
undone.clear();
@@ -201,8 +235,14 @@ pub fn run(argv: Vec<String>) -> anyhow::Result<()> {
"## Your last change was undone\n\nIt reopened findings an earlier round had closed:\n\n{}\nClose the open findings without touching what made these pass.\n\n",
progress.reopened.iter().map(|f| format!("- [{}] {}\n", f.id, f.message)).collect::<String>()
);
eprintln!("structure round {round}: reopened {} closed finding(s), undone", progress.reopened.len());
log.push(format!("structure round {round}: undone (reopened {})", progress.reopened.len()));
eprintln!(
"structure round {round}: reopened {} closed finding(s), undone",
progress.reopened.len()
);
log.push(format!(
"structure round {round}: undone (reopened {})",
progress.reopened.len()
));
continue;
}
// No progress, no more rounds: another pass at the same findings
@@ -214,8 +254,15 @@ pub fn run(argv: Vec<String>) -> anyhow::Result<()> {
}
}
if !green {
let open = if tracked { format!("\n\nStill open:\n{}", ledger.open_list(opts.rounds + 1)) } else { String::new() };
std::fs::write(opts.out.join("run.log"), format!("{}{open}\n\n{report}", log.join("\n")))?;
let open = if tracked {
format!("\n\nStill open:\n{}", ledger.open_list(opts.rounds + 1))
} else {
String::new()
};
std::fs::write(
opts.out.join("run.log"),
format!("{}{open}\n\n{report}", log.join("\n")),
)?;
bail!("structure did not go green; what is still open is in run.log");
}
@@ -224,22 +271,48 @@ pub fn run(argv: Vec<String>) -> anyhow::Result<()> {
let mut defects = String::new();
let mut summary = Vec::new();
for skim in [true, false] {
let tasks_path = opts.out.join(if skim { "tasks-skim.jsonl" } else { "tasks.jsonl" });
let mut args = vec!["--needs-tasks".to_string(), tasks_path.display().to_string()];
let tasks_path = opts.out.join(if skim {
"tasks-skim.jsonl"
} else {
"tasks.jsonl"
});
let mut args = vec![
"--needs-tasks".to_string(),
tasks_path.display().to_string(),
];
if skim {
args.push("--skim".to_string());
}
lint(&opts, &args)?;
let answers = answer_blind(opts.visitor.as_deref().unwrap_or(&opts.llm), &std::fs::read_to_string(&tasks_path)?, Some(&opts.out), 0)?;
let answers_path = opts.out.join(if skim { "answers-skim.jsonl" } else { "answers.jsonl" });
let answers = answer_blind(
opts.visitor.as_deref().unwrap_or(&opts.llm),
&std::fs::read_to_string(&tasks_path)?,
Some(&opts.out),
0,
)?;
let answers_path = opts.out.join(if skim {
"answers-skim.jsonl"
} else {
"answers.jsonl"
});
std::fs::write(&answers_path, answers)?;
let mut args = vec!["--needs-score".to_string(), answers_path.display().to_string()];
let mut args = vec![
"--needs-score".to_string(),
answers_path.display().to_string(),
];
if skim {
args.push("--skim".to_string());
}
let (_, scored) = lint(&opts, &args)?;
let mode = if skim { "headings only" } else { "headings and descriptions" };
for line in scored.lines().filter(|l| l.starts_with("MISROUTE") || l.starts_with("HESITANT")) {
let mode = if skim {
"headings only"
} else {
"headings and descriptions"
};
for line in scored
.lines()
.filter(|l| l.starts_with("MISROUTE") || l.starts_with("HESITANT"))
{
defects.push_str(&format!("[{mode}] {line}\n"));
}
if let Some(line) = scored.lines().find(|l| l.starts_with("wording:")) {
@@ -397,7 +470,9 @@ fn lint_findings(opts: &Opts, out: &Path) -> anyhow::Result<(bool, String, Optio
let found = raw
.lines()
.filter(|l| !l.trim().is_empty())
.map(|l| serde_json::from_str::<Finding>(l).with_context(|| format!("reading {}", out.display())))
.map(|l| {
serde_json::from_str::<Finding>(l).with_context(|| format!("reading {}", out.display()))
})
.collect::<anyhow::Result<Vec<_>>>()?;
Ok((ok, report, Some(found)))
}
@@ -438,7 +513,12 @@ impl Progress {
impl Ledger {
fn update(&mut self, round: usize, fails: Vec<Finding>) -> Progress {
let now: BTreeMap<String, Finding> = fails.into_iter().map(|f| (f.id.clone(), f)).collect();
let fixed: Vec<Finding> = self.open.values().filter(|f| !now.contains_key(&f.id)).cloned().collect();
let fixed: Vec<Finding> = self
.open
.values()
.filter(|f| !now.contains_key(&f.id))
.cloned()
.collect();
let mut new = Vec::new();
let mut reopened = Vec::new();
let mut still = 0;
@@ -457,7 +537,12 @@ impl Ledger {
self.since.remove(&f.id);
}
self.open = now;
Progress { fixed, new, still, reopened }
Progress {
fixed,
new,
still,
reopened,
}
}
/// The open findings for a prompt or a log: id, how long open, the
@@ -470,7 +555,11 @@ impl Ledger {
Some(&first) if first + 1 < round => format!(", open since round {first}"),
_ => String::new(),
};
let scope = if f.need.is_empty() { String::new() } else { format!(" {:?}", f.need) };
let scope = if f.need.is_empty() {
String::new()
} else {
format!(" {:?}", f.need)
};
format!("- [{}] {}{scope}{age}: {}\n", f.id, f.code, f.message)
})
.collect()
@@ -481,8 +570,20 @@ impl Ledger {
/// plus aggregates.yaml - when that is enough to act on. `None` when a
/// finding is about getting around the site (a path, an entry page, a
/// missing desk) or names no page: then the model needs to see it all.
fn named_files<'a>(out: &Path, findings: impl Iterator<Item = &'a Finding>) -> Option<Vec<PathBuf>> {
const WHOLE_SITE: &[&str] = &["schema", "entry-missing", "entry-gated", "path-missing", "path-closed", "path-over-budget", "desk-missing", "need-duplicate"];
fn named_files<'a>(
out: &Path,
findings: impl Iterator<Item = &'a Finding>,
) -> Option<Vec<PathBuf>> {
const WHOLE_SITE: &[&str] = &[
"schema",
"entry-missing",
"entry-gated",
"path-missing",
"path-closed",
"path-over-budget",
"desk-missing",
"need-duplicate",
];
let mut files = vec![out.join("aggregates.yaml")];
for f in findings {
if WHOLE_SITE.contains(&f.code.as_str()) {
@@ -490,12 +591,17 @@ fn named_files<'a>(out: &Path, findings: impl Iterator<Item = &'a Finding>) -> O
}
let text = format!("{} {}", f.message, f.fix.as_deref().unwrap_or_default());
for page in text.split_whitespace().filter(|w| w.starts_with('/')) {
let page = page.trim_end_matches(|c: char| !(c.is_alphanumeric() || c == '_' || c == '-' || c == '/'));
let page = page.trim_end_matches(|c: char| {
!(c.is_alphanumeric() || c == '_' || c == '-' || c == '/')
});
let rel = page.trim_start_matches('/');
let candidates = if rel.is_empty() {
vec![out.join("questions/index.yaml")]
} else {
vec![out.join(format!("questions/{rel}.yaml")), out.join(format!("questions/{rel}/index.yaml"))]
vec![
out.join(format!("questions/{rel}.yaml")),
out.join(format!("questions/{rel}/index.yaml")),
]
};
if let Some(path) = candidates.into_iter().find(|p| p.exists()) {
if !files.contains(&path) {
@@ -512,7 +618,10 @@ fn dump_files(out: &Path, files: &[PathBuf]) -> anyhow::Result<String> {
let mut text = String::new();
for path in files.iter().filter(|p| p.exists()) {
let rel = path.strip_prefix(out).unwrap_or(path).display().to_string();
text.push_str(&format!("<<<FILE {rel}>>>\n{}<<<END>>>\n\n", std::fs::read_to_string(path)?));
text.push_str(&format!(
"<<<FILE {rel}>>>\n{}<<<END>>>\n\n",
std::fs::read_to_string(path)?
));
}
Ok(text)
}
@@ -569,7 +678,9 @@ fn parse_files(text: &str) -> anyhow::Result<BTreeMap<String, String>> {
/// trusted: repo-relative, yaml, and only where content lives.
fn allowed(path: &str) -> bool {
let p = Path::new(path);
let plain = p.components().all(|c| matches!(c, std::path::Component::Normal(_)));
let plain = p
.components()
.all(|c| matches!(c, std::path::Component::Normal(_)));
plain
&& p.extension().and_then(|e| e.to_str()) == Some("yaml")
&& (matches!(path, "aggregates.yaml" | "site.yaml") || path.starts_with("questions/"))
@@ -640,8 +751,15 @@ fn restore(out: &Path, before: Vec<(PathBuf, String)>) -> anyhow::Result<()> {
fn dump(out: &Path) -> anyhow::Result<String> {
let mut text = String::new();
for path in content_files(out)? {
let rel = path.strip_prefix(out).unwrap_or(&path).display().to_string();
text.push_str(&format!("<<<FILE {rel}>>>\n{}<<<END>>>\n\n", std::fs::read_to_string(&path)?));
let rel = path
.strip_prefix(out)
.unwrap_or(&path)
.display()
.to_string();
text.push_str(&format!(
"<<<FILE {rel}>>>\n{}<<<END>>>\n\n",
std::fs::read_to_string(&path)?
));
}
Ok(text)
}
@@ -659,7 +777,12 @@ fn dump(out: &Path) -> anyhow::Result<String> {
/// card portal renders: button, encouragement, sections with icon and
/// guidance, fields with placeholders. Headings-only tasks (`--skim`,
/// where the task's text is the heading itself) stay headings only.
pub(crate) fn answer_blind(cmd: &str, tasks: &str, repo: Option<&Path>, seed_offset: u64) -> anyhow::Result<String> {
pub(crate) fn answer_blind(
cmd: &str,
tasks: &str,
repo: Option<&Path>,
seed_offset: u64,
) -> anyhow::Result<String> {
let mut answers = String::new();
for line in tasks.lines().filter(|l| !l.trim().is_empty()) {
let task: serde_json::Value = serde_json::from_str(line)?;
@@ -673,8 +796,14 @@ pub(crate) fn answer_blind(cmd: &str, tasks: &str, repo: Option<&Path>, seed_off
.flatten()
.map(|(k, v)| (k.as_str(), v.as_str().unwrap_or_default()))
.collect();
let skim = options.iter().all(|(name, text)| text.is_empty() || text == name);
let page = if skim { None } else { repo.and_then(|r| rate::load_page(r, task["page"].as_str().unwrap_or("/"))) };
let skim = options
.iter()
.all(|(name, text)| text.is_empty() || text == name);
let page = if skim {
None
} else {
repo.and_then(|r| rate::load_page(r, task["page"].as_str().unwrap_or("/")))
};
let seed = fnv(id).wrapping_add(seed_offset.wrapping_mul(0x9E37_79B9_7F4A_7C15) as _);
shuffle(&mut options, seed);
// Options are numbered and the answer is the number: with a
@@ -702,14 +831,25 @@ pub(crate) fn answer_blind(cmd: &str, tasks: &str, repo: Option<&Path>, seed_off
let mut head = page.as_ref().map(card::page_head).unwrap_or_default();
// What the page says before it offers anything (the opener, the
// look around): read, not picked.
if let Some(context) = task["state"]["page"].as_str().filter(|c| !c.trim().is_empty()) {
if let Some(context) = task["state"]["page"]
.as_str()
.filter(|c| !c.trim().is_empty())
{
head.push_str(&format!("The page first says:\n{context}\n\n"));
}
let prompt = format!(
"You are this person: {body}\n\n{instructions}\n{head}The page offers these options:\n\n{}\n\nWhich one do you pick? Judge only from your own situation and the wording in front of you. Give your honest confidence between 0 and 1; if two options fit you about equally, confidence is low.\nReply as JSON: {{\"choice\": <the option's number>, \"confidence\": 0.00}}",
listed.join("\n\n")
);
let reply = llm_env(cmd, &prompt, &[("FORMAT", "json"), ("SEED", &seed.to_string()), ("TEMPERATURE", "0.2")])?;
let reply = llm_env(
cmd,
&prompt,
&[
("FORMAT", "json"),
("SEED", &seed.to_string()),
("TEMPERATURE", "0.2"),
],
)?;
let parsed = simulate::first_object(&reply).unwrap_or(serde_json::Value::Null);
let names: Vec<&str> = options.iter().map(|(n, _)| *n).collect();
let raw = match &parsed["choice"] {
@@ -722,7 +862,9 @@ pub(crate) fn answer_blind(cmd: &str, tasks: &str, repo: Option<&Path>, seed_off
_ => simulate::resolve_choice(&names, &raw),
};
let confidence = parsed["confidence"].as_f64().unwrap_or(0.0);
answers.push_str(&serde_json::to_string(&serde_json::json!({ "id": id, "choice": choice, "confidence": confidence }))?);
answers.push_str(&serde_json::to_string(
&serde_json::json!({ "id": id, "choice": choice, "confidence": confidence }),
)?);
answers.push('\n');
}
Ok(answers)
@@ -730,7 +872,9 @@ pub(crate) fn answer_blind(cmd: &str, tasks: &str, repo: Option<&Path>, seed_off
/// FNV-1a over the task id: a seed that is the same on every run.
fn fnv(text: &str) -> u32 {
text.bytes().fold(0x811c_9dc5u32, |h, b| (h ^ u32::from(b)).wrapping_mul(0x0100_0193))
text.bytes().fold(0x811c_9dc5u32, |h, b| {
(h ^ u32::from(b)).wrapping_mul(0x0100_0193)
})
}
/// Fisher-Yates on a small xorshift, so the order is fixed by the seed.
@@ -803,21 +947,45 @@ mod ledger_tests {
use super::*;
fn f(id: &str, code: &str, message: &str) -> Finding {
Finding { id: id.into(), level: "fail".into(), code: code.into(), need: "hire".into(), message: message.into(), fix: None }
Finding {
id: id.into(),
level: "fail".into(),
code: code.into(),
need: "hire".into(),
message: message.into(),
fix: None,
}
}
#[test]
fn a_round_is_told_what_it_closed_kept_added_and_broke() {
let mut ledger = Ledger::default();
let p = ledger.update(1, vec![f("a", "strand", "x"), f("b", "strand", "y")]);
assert_eq!((p.fixed.len(), p.new.len(), p.still, p.reopened.len()), (0, 2, 0, 0));
let p = ledger.update(2, vec![f("b", "strand", "y"), f("c", "edge-unoffered", "z")]);
assert_eq!((p.fixed.len(), p.new.len(), p.still, p.reopened.len()), (1, 1, 1, 0));
assert_eq!(p.summary(ledger.open.len()), "2 open - fixed 1, still open 1, new 1, reopened 0");
assert_eq!(
(p.fixed.len(), p.new.len(), p.still, p.reopened.len()),
(0, 2, 0, 0)
);
let p = ledger.update(
2,
vec![f("b", "strand", "y"), f("c", "edge-unoffered", "z")],
);
assert_eq!(
(p.fixed.len(), p.new.len(), p.still, p.reopened.len()),
(1, 1, 1, 0)
);
assert_eq!(
p.summary(ledger.open.len()),
"2 open - fixed 1, still open 1, new 1, reopened 0"
);
// "a" was closed in round 2; its return is a regression.
let p = ledger.update(3, vec![f("a", "strand", "x"), f("b", "strand", "y")]);
assert_eq!(p.reopened.iter().map(|f| f.id.as_str()).collect::<Vec<_>>(), ["a"]);
assert!(ledger.open_list(4).contains("[b] strand \"hire\", open since round 1: y"));
assert_eq!(
p.reopened.iter().map(|f| f.id.as_str()).collect::<Vec<_>>(),
["a"]
);
assert!(ledger
.open_list(4)
.contains("[b] strand \"hire\", open since round 1: y"));
let p = ledger.update(4, vec![]);
assert_eq!(p.summary(ledger.open.len()), "green (fixed 2)");
}
@@ -830,11 +998,22 @@ mod ledger_tests {
std::fs::write(dir.join("questions/desk/index.yaml"), "x").unwrap();
std::fs::write(dir.join("questions/index.yaml"), "x").unwrap();
let mut strand = f("a", "strand", "records strand in \"quoted\"");
strand.fix = Some("add `{ from: quoted, to: won, label: ... }` to the transitions on /desk".into());
strand.fix =
Some("add `{ from: quoted, to: won, label: ... }` to the transitions on /desk".into());
let files = named_files(&dir, [&strand].into_iter()).unwrap();
assert_eq!(files, vec![dir.join("aggregates.yaml"), dir.join("questions/desk/index.yaml")]);
assert_eq!(
files,
vec![
dir.join("aggregates.yaml"),
dir.join("questions/desk/index.yaml")
]
);
// A finding about getting around the site needs all of it.
assert!(named_files(&dir, [&strand, &f("b", "path-missing", "no path")].into_iter()).is_none());
assert!(named_files(
&dir,
[&strand, &f("b", "path-missing", "no path")].into_iter()
)
.is_none());
std::fs::remove_dir_all(&dir).ok();
}
}
+34 -12
View File
@@ -284,7 +284,11 @@ pub fn run(argv: &[String]) -> anyhow::Result<()> {
let text = std::fs::read_to_string(&path).with_context(|| format!("reading {path}"))?;
let doc: serde_yaml::Value =
serde_yaml::from_str(&text).with_context(|| format!("parsing {path}"))?;
Some(doc.get("personas").cloned().ok_or_else(|| anyhow!("{path} has no `personas:`"))?)
Some(
doc.get("personas")
.cloned()
.ok_or_else(|| anyhow!("{path} has no `personas:`"))?,
)
}
None => None,
};
@@ -323,13 +327,14 @@ pub fn run(argv: &[String]) -> anyhow::Result<()> {
if let Some(list) = personas.as_mut() {
let mapped = map_personas(&cmd, &plan, list)?;
if mapped > 0 {
eprintln!("plan round {round}: {mapped} persona(s) mapped to this plan's needs");
eprintln!(
"plan round {round}: {mapped} persona(s) mapped to this plan's needs"
);
}
text.push_str("\n# Held out from every model that writes words.\n");
text.push_str(&serde_yaml::to_string(&serde_yaml::Mapping::from_iter([(
serde_yaml::Value::from("personas"),
list.clone(),
)]))?);
text.push_str(&serde_yaml::to_string(&serde_yaml::Mapping::from_iter([
(serde_yaml::Value::from("personas"), list.clone()),
]))?);
}
let whole: Plan =
serde_yaml::from_str(&text).context("the plan with its personas attached")?;
@@ -411,25 +416,42 @@ pub fn run(argv: &[String]) -> anyhow::Result<()> {
/// not have, or a persona the model leaves out, stays as it was.
fn map_personas(cmd: &str, plan: &Plan, personas: &mut serde_yaml::Value) -> anyhow::Result<usize> {
let ids: Vec<&str> = plan.needs.iter().map(|n| n.id.as_str()).collect();
let Some(list) = personas.as_sequence_mut() else { return Ok(0) };
let Some(list) = personas.as_sequence_mut() else {
return Ok(0);
};
let strangers: Vec<(String, String)> = list
.iter()
.filter(|p| !ids.contains(&p["need"].as_str().unwrap_or_default()))
.map(|p| (p["id"].as_str().unwrap_or_default().to_string(), p["text"].as_str().unwrap_or_default().to_string()))
.map(|p| {
(
p["id"].as_str().unwrap_or_default().to_string(),
p["text"].as_str().unwrap_or_default().to_string(),
)
})
.filter(|(id, _)| !id.is_empty())
.collect();
if strangers.is_empty() {
return Ok(0);
}
let needs: Vec<String> = plan.needs.iter().map(|n| format!("- {}: {} - wants {}", n.id, n.who, n.wants)).collect();
let people: Vec<String> = strangers.iter().map(|(id, text)| format!("- {id}: {text}")).collect();
let needs: Vec<String> = plan
.needs
.iter()
.map(|n| format!("- {}: {} - wants {}", n.id, n.who, n.wants))
.collect();
let people: Vec<String> = strangers
.iter()
.map(|(id, text)| format!("- {id}: {text}"))
.collect();
let prompt = format!(
"A site has these needs, one per kind of visitor:\n{}\n\nThese people arrive at it:\n{}\n\nFor each person, which need is theirs? Reply with a JSON object mapping each person's id to one need id from the list, and nothing else. Leave a person out if no need fits.",
needs.join("\n"),
people.join("\n")
);
let reply = crate::trainer::llm_with(cmd, &prompt, &[("FORMAT", "json"), ("TEMPERATURE", "0.0")])?;
let Some(mapping) = first_object(&reply) else { return Ok(0) };
let reply =
crate::trainer::llm_with(cmd, &prompt, &[("FORMAT", "json"), ("TEMPERATURE", "0.0")])?;
let Some(mapping) = first_object(&reply) else {
return Ok(0);
};
let mut mapped = 0;
for persona in list.iter_mut() {
let id = persona["id"].as_str().unwrap_or_default().to_string();
+364 -72
View File
@@ -34,11 +34,16 @@ use crate::trainer::card;
use crate::trainer::simulate::ask;
fn flag(argv: &[String], name: &str) -> Option<String> {
argv.iter().position(|a| a == name).and_then(|i| argv.get(i + 1)).cloned()
argv.iter()
.position(|a| a == name)
.and_then(|i| argv.get(i + 1))
.cloned()
}
fn load(path: &str) -> anyhow::Result<Value> {
Ok(serde_json::from_str(&std::fs::read_to_string(path).with_context(|| format!("reading {path}"))?)?)
Ok(serde_json::from_str(
&std::fs::read_to_string(path).with_context(|| format!("reading {path}"))?,
)?)
}
/// LIX = words per sentence + 100 * long words / words.
@@ -50,7 +55,11 @@ pub fn lix(text: &str) -> f64 {
if words.is_empty() {
return 0.0;
}
let sentences = text.split(['.', '!', '?', ':', '\n']).filter(|s| s.chars().any(char::is_alphabetic)).count().max(1);
let sentences = text
.split(['.', '!', '?', ':', '\n'])
.filter(|s| s.chars().any(char::is_alphabetic))
.count()
.max(1);
let long = words.iter().filter(|w| w.chars().count() > 6).count();
words.len() as f64 / sentences as f64 + 100.0 * long as f64 / words.len() as f64
}
@@ -74,7 +83,10 @@ fn page_text(step: &Value, page: Option<&Value>, followup: Option<&Value>, gutte
};
// The cards that lead nowhere (the opener, the look around) are
// read before the options; the gutted twin loses them too.
if let Some(context) = step["context"].as_str().filter(|c| !c.trim().is_empty() && !gutted) {
if let Some(context) = step["context"]
.as_str()
.filter(|c| !c.trim().is_empty() && !gutted)
{
text.push_str(&format!("\n{context}\n"));
}
for (i, option) in step["options"].as_array().into_iter().flatten().enumerate() {
@@ -105,7 +117,12 @@ fn page_text(step: &Value, page: Option<&Value>, followup: Option<&Value>, gutte
// tells a reader how many things they will be asked, and a model
// answers "can you tell what you will be asked" with yes to that
// (measured: 2 of 2 on every need).
let fields: Vec<&str> = step["fields"].as_array().into_iter().flatten().filter_map(|f| f["label"].as_str()).collect();
let fields: Vec<&str> = step["fields"]
.as_array()
.into_iter()
.flatten()
.filter_map(|f| f["label"].as_str())
.collect();
if !gutted && !fields.is_empty() {
text.push_str(&format!(
"\nThe form under \"{}\" asks for: {}.\n",
@@ -121,13 +138,24 @@ fn page_text(step: &Value, page: Option<&Value>, followup: Option<&Value>, gutte
fn finish_page_text(mut text: String, followup: Option<&Value>) -> String {
if let Some(next) = followup {
text.push_str(&format!("\nAfter you send it, the next page says:\n{}\n", next["name"].as_str().unwrap_or_default()));
if let Some(d) = next["description"].as_str().filter(|d| !d.trim().is_empty()) {
text.push_str(&format!(
"\nAfter you send it, the next page says:\n{}\n",
next["name"].as_str().unwrap_or_default()
));
if let Some(d) = next["description"]
.as_str()
.filter(|d| !d.trim().is_empty())
{
text.push_str(&format!("{}\n", d.trim()));
}
for alt in next["alternatives"].as_array().into_iter().flatten() {
let d = alt["description"].as_str().unwrap_or_default().trim();
text.push_str(&format!("- {}{}{}\n", alt["name"].as_str().unwrap_or_default(), if d.is_empty() { "" } else { ": " }, d));
text.push_str(&format!(
"- {}{}{}\n",
alt["name"].as_str().unwrap_or_default(),
if d.is_empty() { "" } else { ": " },
d
));
}
}
text
@@ -137,12 +165,30 @@ fn finish_page_text(mut text: String, followup: Option<&Value>) -> String {
/// description, and the followup. Headings and labels are left out.
fn prose_text(step: &Value, followup: Option<&Value>) -> String {
let mut prose = vec![step["question"].as_str().unwrap_or_default().to_string()];
prose.extend(step["options"].as_array().into_iter().flatten().filter_map(|o| o["description"].as_str()).map(str::to_string));
prose.extend(
step["options"]
.as_array()
.into_iter()
.flatten()
.filter_map(|o| o["description"].as_str())
.map(str::to_string),
);
if let Some(next) = followup {
prose.extend(next["description"].as_str().map(str::to_string));
prose.extend(next["alternatives"].as_array().into_iter().flatten().filter_map(|a| a["description"].as_str()).map(str::to_string));
prose.extend(
next["alternatives"]
.as_array()
.into_iter()
.flatten()
.filter_map(|a| a["description"].as_str())
.map(str::to_string),
);
}
prose.into_iter().filter(|p| !p.trim().is_empty()).collect::<Vec<_>>().join("\n")
prose
.into_iter()
.filter(|p| !p.trim().is_empty())
.collect::<Vec<_>>()
.join("\n")
}
/// The page a visitor lands on after the step's alternative: read from
@@ -152,7 +198,11 @@ fn followup_page(repo: Option<&Path>, step: &Value) -> Option<Value> {
let repo = repo?;
let page = load_page(repo, step["page"].as_str()?)?;
let alternative = step["alternative"].as_str()?;
let action = page["alternatives"].as_array()?.iter().find(|a| a["name"] == alternative)?["action"].as_str()?;
let action = page["alternatives"]
.as_array()?
.iter()
.find(|a| a["name"] == alternative)?["action"]
.as_str()?;
load_page(repo, action)
}
@@ -187,15 +237,33 @@ fn s(v: &Value) -> String {
/// the button for it. The outcome is put in plain words from the
/// state's own name, never from the label, and the buttons are listed
/// in a seeded shuffle.
fn button_pick(cmd: &str, group: &str, bucket: &str, from: &str, to: &str, labels: &[&str], seed: usize, words: &Value) -> anyhow::Result<String> {
fn button_pick(
cmd: &str,
group: &str,
bucket: &str,
from: &str,
to: &str,
labels: &[&str],
seed: usize,
words: &Value,
) -> anyhow::Result<String> {
// The state in the plan's own words when the spec carries them,
// else the id made readable.
let plain = |id: &str| words[id].as_str().map(str::to_string).unwrap_or_else(|| id.replace(['_', '-'], " "));
let plain = |id: &str| {
words[id]
.as_str()
.map(str::to_string)
.unwrap_or_else(|| id.replace(['_', '-'], " "))
};
let mut order: Vec<&str> = labels.to_vec();
// A fixed rotation per move: the right answer is not always first.
let by = seed % order.len().max(1);
order.rotate_left(by);
let listed: Vec<String> = order.iter().enumerate().map(|(i, l)| format!("{}. {l}", i + 1)).collect();
let listed: Vec<String> = order
.iter()
.enumerate()
.map(|(i, l)| format!("{}. {l}", i + 1))
.collect();
let prompt = format!(
"You work on the {} desk of a small organisation. A case in \"{}\" was \"{}\" and has just become \"{}\". The desk shows these buttons for it:\n\n{}\n\nWhich button records what just happened? Reply as JSON: {{\"choice\": <the button's number>}}",
plain(group),
@@ -206,7 +274,10 @@ fn button_pick(cmd: &str, group: &str, bucket: &str, from: &str, to: &str, label
);
let reply = ask(cmd, &prompt, &seed.to_string())?;
let n = reply["choice"].as_u64().unwrap_or(0) as usize;
Ok(order.get(n.wrapping_sub(1)).map(|l| l.to_string()).unwrap_or_default())
Ok(order
.get(n.wrapping_sub(1))
.map(|l| l.to_string())
.unwrap_or_default())
}
/// Every `mail:` on a state in the repo's aggregates.yaml, as
@@ -228,7 +299,12 @@ pub(crate) fn mail_texts(repo: &Path) -> Option<Vec<(String, String, String)>> {
}
out.split_whitespace().collect::<Vec<_>>().join(" ")
};
Some(raw_mails(repo, &doc).into_iter().map(|(name, subject, body)| (name, strip(&subject), strip(&body))).collect())
Some(
raw_mails(repo, &doc)
.into_iter()
.map(|(name, subject, body)| (name, strip(&subject), strip(&body)))
.collect(),
)
}
/// Every mail the site sends, as written: the state mails in
@@ -248,10 +324,17 @@ fn raw_mails(repo: &Path, aggregates: &Value) -> Vec<(String, String, String)> {
}
}
}
let site: Value = std::fs::read_to_string(repo.join("site.yaml")).ok().and_then(|t| serde_yaml::from_str(&t).ok()).unwrap_or(Value::Null);
let site: Value = std::fs::read_to_string(repo.join("site.yaml"))
.ok()
.and_then(|t| serde_yaml::from_str(&t).ok())
.unwrap_or(Value::Null);
let invite = &site["mail"]["invite"];
if invite.is_object() {
out.push(("invite".to_string(), invite["subject"].as_str().unwrap_or_default().to_string(), invite["body"].as_str().unwrap_or_default().to_string()));
out.push((
"invite".to_string(),
invite["subject"].as_str().unwrap_or_default().to_string(),
invite["body"].as_str().unwrap_or_default().to_string(),
));
}
out
}
@@ -265,7 +348,12 @@ fn as_received(template: &str, site_name: &str) -> String {
out.push_str(&rest[..i]);
let after = &rest[i + 1..];
match after.find('}') {
Some(j) if after[..j].chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.') && j > 0 => {
Some(j)
if after[..j]
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.')
&& j > 0 =>
{
out.push_str(&match &after[..j] {
"site" => "https://example.site".to_string(),
"chain" => "8f3a1c".to_string(),
@@ -287,19 +375,38 @@ fn as_received(template: &str, site_name: &str) -> String {
}
const MAIL_QUESTIONS: [(&str, &str); 3] = [
("happened", "Do you know, from this mail alone, what has happened and to what?"),
("now", "Do you know what, if anything, you can or must do now, and how?"),
("who", "Do you know who sent it, and how you would answer them?"),
(
"happened",
"Do you know, from this mail alone, what has happened and to what?",
),
(
"now",
"Do you know what, if anything, you can or must do now, and how?",
),
(
"who",
"Do you know who sent it, and how you would answer them?",
),
];
/// Three yes-or-no questions on one mail, from the person it is for.
fn mail_understanding(cmd: &str, reader: &str, subject: &str, body: &str, seed: usize) -> anyhow::Result<[bool; 3]> {
fn mail_understanding(
cmd: &str,
reader: &str,
subject: &str,
body: &str,
seed: usize,
) -> anyhow::Result<[bool; 3]> {
let prompt = format!(
"You are {reader}. This mail has just arrived:\n---\nSubject: {subject}\n\n{body}\n---\n\nAnswer three questions as yourself. Say yes only when the mail itself tells you; if you would be guessing, say no.\n{}\nReply as JSON: {{\"happened\": true or false, \"now\": true or false, \"who\": true or false}}",
MAIL_QUESTIONS.iter().map(|(k, q)| format!("- {k}: {q}")).collect::<Vec<_>>().join("\n")
);
let reply = ask(cmd, &prompt, &seed.to_string())?;
Ok([reply["happened"] == true, reply["now"] == true, reply["who"] == true])
Ok([
reply["happened"] == true,
reply["now"] == true,
reply["who"] == true,
])
}
/// `/` is `questions/index.yaml`; `/a/b` is `questions/a/b.yaml` or
@@ -309,9 +416,14 @@ pub(crate) fn load_page(repo: &Path, route: &str) -> Option<Value> {
let candidates = if rel.is_empty() {
vec![repo.join("questions/index.yaml")]
} else {
vec![repo.join(format!("questions/{rel}.yaml")), repo.join(format!("questions/{rel}/index.yaml"))]
vec![
repo.join(format!("questions/{rel}.yaml")),
repo.join(format!("questions/{rel}/index.yaml")),
]
};
let text = candidates.iter().find_map(|p| std::fs::read_to_string(p).ok())?;
let text = candidates
.iter()
.find_map(|p| std::fs::read_to_string(p).ok())?;
serde_yaml::from_str(&text).ok()
}
@@ -326,7 +438,12 @@ const MAX_REQUIRED: usize = 5;
fn page_load(step: &Value) -> (usize, usize) {
(
step["options"].as_array().map_or(0, Vec::len),
step["fields"].as_array().into_iter().flatten().filter(|f| f["optional"] != true).count(),
step["fields"]
.as_array()
.into_iter()
.flatten()
.filter(|f| f["optional"] != true)
.count(),
)
}
@@ -344,7 +461,12 @@ fn understanding(cmd: &str, who: &str, page: &str, seed: usize) -> anyhow::Resul
QUESTIONS.iter().map(|(k, q)| format!("- {k}: {q}")).collect::<Vec<_>>().join("\n")
);
let reply = ask(cmd, &prompt, &seed.to_string())?;
Ok([reply["which"] == true, reply["where"] == true, reply["asked"] == true, reply["next"] == true])
Ok([
reply["which"] == true,
reply["where"] == true,
reply["asked"] == true,
reply["next"] == true,
])
}
pub fn rate(argv: &[String]) -> anyhow::Result<()> {
@@ -365,7 +487,10 @@ pub fn rate(argv: &[String]) -> anyhow::Result<()> {
// that desk's buttons for the bucket.
let desks = desk_buttons(&model);
if !desks.is_empty() {
println!("{:<20} {:<24} {:>8} {}", "desk", "bucket", "buttons", "mistaken for");
println!(
"{:<20} {:<24} {:>8} {}",
"desk", "bucket", "buttons", "mistaken for"
);
for (page, group, bucket, moves) in &desks {
// A desk row shows only the buttons that start from the
// row's state, so those are the ones the worker chooses
@@ -374,12 +499,25 @@ pub fn rate(argv: &[String]) -> anyhow::Result<()> {
let mut asked = 0;
let mut mistaken = Vec::new();
for (i, (from, to, label)) in moves.iter().enumerate() {
let labels: Vec<&str> = moves.iter().filter(|(f, _, _)| f == from).map(|(_, _, l)| l.as_str()).collect();
let labels: Vec<&str> = moves
.iter()
.filter(|(f, _, _)| f == from)
.map(|(_, _, l)| l.as_str())
.collect();
if labels.len() < 2 {
continue;
}
asked += 1;
let picked = button_pick(&cmd, group, bucket, from, to, &labels, i, &model["buckets"][bucket]["words"])?;
let picked = button_pick(
&cmd,
group,
bucket,
from,
to,
&labels,
i,
&model["buckets"][bucket]["words"],
)?;
if picked == *label {
right += 1;
} else {
@@ -389,8 +527,18 @@ pub fn rate(argv: &[String]) -> anyhow::Result<()> {
if asked == 0 {
continue;
}
println!("{:<20} {:<24} {:>3} of {:<2} {}", page, bucket, right, asked, mistaken.join("; "));
report.insert(format!("desk {page} {bucket}"), json!({ "right": right, "of": asked, "mistaken": mistaken }));
println!(
"{:<20} {:<24} {:>3} of {:<2} {}",
page,
bucket,
right,
asked,
mistaken.join("; ")
);
report.insert(
format!("desk {page} {bucket}"),
json!({ "right": right, "of": asked, "mistaken": mistaken }),
);
}
println!();
}
@@ -404,12 +552,23 @@ pub fn rate(argv: &[String]) -> anyhow::Result<()> {
if let Some(mails) = repo.as_deref().and_then(mail_texts) {
if !mails.is_empty() {
let dir = repo.as_deref().unwrap_or(Path::new("."));
let site: Value = std::fs::read_to_string(dir.join("site.yaml")).ok().and_then(|t| serde_yaml::from_str(&t).ok()).unwrap_or(Value::Null);
let site: Value = std::fs::read_to_string(dir.join("site.yaml"))
.ok()
.and_then(|t| serde_yaml::from_str(&t).ok())
.unwrap_or(Value::Null);
let site_name = site["title"].as_str().unwrap_or("the site").to_string();
let aggregates: Value = std::fs::read_to_string(dir.join("aggregates.yaml")).ok().and_then(|t| serde_yaml::from_str(&t).ok()).unwrap_or(Value::Null);
let aggregates: Value = std::fs::read_to_string(dir.join("aggregates.yaml"))
.ok()
.and_then(|t| serde_yaml::from_str(&t).ok())
.unwrap_or(Value::Null);
let raw = raw_mails(dir, &aggregates);
println!("{:<34} {:>5} {:<14} {:>5} {:>8} {:>5} {:>5} {:<16}", "mail", "LIX", "", "words", "happened", "now", "who", "gutted twin says");
for (i, ((name, subject, body), (_, raw_subject, raw_body))) in mails.iter().zip(&raw).enumerate() {
println!(
"{:<34} {:>5} {:<14} {:>5} {:>8} {:>5} {:>5} {:<16}",
"mail", "LIX", "", "words", "happened", "now", "who", "gutted twin says"
);
for (i, ((name, subject, body), (_, raw_subject, raw_body))) in
mails.iter().zip(&raw).enumerate()
{
let text = format!("{subject}. {body}");
let score = lix(&text);
let reader = if name == "invite" {
@@ -417,30 +576,59 @@ pub fn rate(argv: &[String]) -> anyhow::Result<()> {
} else {
format!("someone who sent a form to {site_name} some time ago")
};
let (subject_in, body_in) = (as_received(raw_subject, &site_name), as_received(raw_body, &site_name));
let (subject_in, body_in) = (
as_received(raw_subject, &site_name),
as_received(raw_body, &site_name),
);
let got = mail_understanding(&cmd, &reader, &subject_in, &body_in, i)?;
let twin = mail_understanding(&cmd, &reader, "News", "There is news.", i)?;
let yn = |b: bool| if b { "yes" } else { "no" };
let failed: Vec<&str> = MAIL_QUESTIONS.iter().zip(twin).filter(|(_, t)| *t).map(|((k, _), _)| *k).collect();
let failed: Vec<&str> = MAIL_QUESTIONS
.iter()
.zip(twin)
.filter(|(_, t)| *t)
.map(|((k, _), _)| *k)
.collect();
println!(
"{:<34} {:>5.0} {:<14} {:>5} {:>8} {:>5} {:>5} {:<16}",
name, score, lix_band(score), text.split_whitespace().count(), yn(got[0]), yn(got[1]), yn(got[2]),
if failed.is_empty() { "ok".to_string() } else { format!("FAILED: {}", failed.join(", ")) }
name,
score,
lix_band(score),
text.split_whitespace().count(),
yn(got[0]),
yn(got[1]),
yn(got[2]),
if failed.is_empty() {
"ok".to_string()
} else {
format!("FAILED: {}", failed.join(", "))
}
);
report.insert(format!("mail {name}"), json!({
report.insert(
format!("mail {name}"),
json!({
"lix": score, "words": text.split_whitespace().count(),
"understanding": { "happened": got[0], "now": got[1], "who": got[2] },
"control_ok": failed.is_empty(),
}));
}),
);
}
println!();
}
}
println!("{:<28} {:>6} {:<14} {:>8} {:>8} {:>8} {:>8} {:<22} {:<22}", "need", "LIX", "", "which", "where", "asked", "next", "gutted twin says", "page load");
println!(
"{:<28} {:>6} {:<14} {:>8} {:>8} {:>8} {:>8} {:<22} {:<22}",
"need", "LIX", "", "which", "where", "asked", "next", "gutted twin says", "page load"
);
for need in model["needs"].as_array().into_iter().flatten() {
let id = need["id"].as_str().unwrap_or_default();
let steps = need["steps"].as_array().cloned().unwrap_or_default();
let personas: Vec<&Value> = model["personas"].as_array().into_iter().flatten().filter(|p| p["need"] == id).collect();
let personas: Vec<&Value> = model["personas"]
.as_array()
.into_iter()
.flatten()
.filter(|p| p["need"] == id)
.collect();
if personas.is_empty() {
continue;
}
@@ -450,15 +638,30 @@ pub fn rate(argv: &[String]) -> anyhow::Result<()> {
// after sending.
let mut pages = Vec::new();
for (i, step) in steps.iter().enumerate() {
let followup = if i + 1 == steps.len() { followup_page(repo.as_deref(), step) } else { None };
let page = repo.as_deref().and_then(|r| load_page(r, step["page"].as_str().unwrap_or("/")));
let row = if steps.len() > 1 { format!("{id} @{}", step["page"].as_str().unwrap_or_default()) } else { id.to_string() };
let followup = if i + 1 == steps.len() {
followup_page(repo.as_deref(), step)
} else {
None
};
let page = repo
.as_deref()
.and_then(|r| load_page(r, step["page"].as_str().unwrap_or("/")));
let row = if steps.len() > 1 {
format!("{id} @{}", step["page"].as_str().unwrap_or_default())
} else {
id.to_string()
};
let readability = lix(&prose_text(step, followup.as_ref()));
let (mut yes, mut control) = ([0usize; N], [0usize; N]);
for (i, persona) in personas.iter().enumerate() {
let who = persona["text"].as_str().unwrap_or_default();
for (tally, gutted) in [(&mut yes, false), (&mut control, true)] {
let answers = understanding(&cmd, who, &page_text(step, page.as_ref(), followup.as_ref(), gutted), i)?;
let answers = understanding(
&cmd,
who,
&page_text(step, page.as_ref(), followup.as_ref(), gutted),
i,
)?;
for (slot, answer) in tally.iter_mut().zip(answers) {
*slot += answer as usize;
}
@@ -466,12 +669,25 @@ pub fn rate(argv: &[String]) -> anyhow::Result<()> {
}
let n = personas.len();
let measured = |k: usize| k != 3 || followup.is_some();
let share = |k: usize| if measured(k) { format!("{}/{n}", yes[k]) } else { "-".to_string() };
let share = |k: usize| {
if measured(k) {
format!("{}/{n}", yes[k])
} else {
"-".to_string()
}
};
// The twin has no headings, no descriptions, no cards and no
// followup: every question must collapse on it, or that
// question is agreeing with anything.
let control_ok: Vec<bool> = (0..N).map(|k| !measured(k) || control[k] * 2 < n.max(1)).collect();
let failed: Vec<&str> = QUESTIONS.iter().zip(&control_ok).filter(|(_, ok)| !**ok).map(|((k, _), _)| *k).collect();
let control_ok: Vec<bool> = (0..N)
.map(|k| !measured(k) || control[k] * 2 < n.max(1))
.collect();
let failed: Vec<&str> = QUESTIONS
.iter()
.zip(&control_ok)
.filter(|(_, ok)| !**ok)
.map(|((k, _), _)| *k)
.collect();
let (options, required) = page_load(step);
println!(
"{:<28} {:>6.0} {:<14} {:>8} {:>8} {:>8} {:>8} {:<22} {:<22}",
@@ -487,12 +703,24 @@ pub fn rate(argv: &[String]) -> anyhow::Result<()> {
control[0],
control[1],
control[2],
if measured(3) { control[3].to_string() } else { "-".to_string() },
if failed.is_empty() { "ok".to_string() } else { format!("FAILED: {}", failed.join(", ")) }
if measured(3) {
control[3].to_string()
} else {
"-".to_string()
},
if failed.is_empty() {
"ok".to_string()
} else {
format!("FAILED: {}", failed.join(", "))
}
),
format!(
"{options} options, {required} required{}",
if options > MAX_OPTIONS || required > MAX_REQUIRED { " - SPLIT" } else { "" }
if options > MAX_OPTIONS || required > MAX_REQUIRED {
" - SPLIT"
} else {
""
}
),
);
pages.push(json!({
@@ -505,9 +733,15 @@ pub fn rate(argv: &[String]) -> anyhow::Result<()> {
"control_ok": failed.is_empty(),
}));
}
report.insert(id.to_string(), json!({ "steps": pages.len(), "pages": pages }));
report.insert(
id.to_string(),
json!({ "steps": pages.len(), "pages": pages }),
);
}
std::fs::write(out.join("rate.json"), serde_json::to_string_pretty(&report)?)?;
std::fs::write(
out.join("rate.json"),
serde_json::to_string_pretty(&report)?,
)?;
Ok(())
}
@@ -517,22 +751,49 @@ pub fn prefer(argv: &[String]) -> anyhow::Result<()> {
let b = load(&flag(argv, "--b").ok_or_else(|| anyhow!(usage))?)?;
let cmd = flag(argv, "--llm").unwrap_or_else(|| "claude -p".to_string());
println!("{:<28} {:>5} {:>5} {:>14} {:>5} {:>5} {:>14} {:>9}", "need", "A", "B", "no preference", "A", "B", "no preference", "LIX A->B");
println!(
"{:<28} {:>5} {:>5} {:>14} {:>5} {:>5} {:>14} {:>9}",
"need", "A", "B", "no preference", "A", "B", "no preference", "LIX A->B"
);
println!("{:<28} {:^26} {:^26}", "", "friendlier", "clearer");
for need_a in a["needs"].as_array().into_iter().flatten() {
let id = need_a["id"].as_str().unwrap_or_default();
let Some(need_b) = b["needs"].as_array().into_iter().flatten().find(|n| n["id"] == id) else { continue };
let (Some(step_a), Some(step_b)) = (need_a["steps"].as_array().and_then(|s| s.first()), need_b["steps"].as_array().and_then(|s| s.first())) else { continue };
let (text_a, text_b) = (page_text(step_a, None, None, false), page_text(step_b, None, None, false));
let Some(need_b) = b["needs"]
.as_array()
.into_iter()
.flatten()
.find(|n| n["id"] == id)
else {
continue;
};
let (Some(step_a), Some(step_b)) = (
need_a["steps"].as_array().and_then(|s| s.first()),
need_b["steps"].as_array().and_then(|s| s.first()),
) else {
continue;
};
let (text_a, text_b) = (
page_text(step_a, None, None, false),
page_text(step_b, None, None, false),
);
if text_a == text_b {
continue;
}
// [friendlier, clearer] x [A, B, none]
let mut wins = [[0usize; 3]; 2];
for (i, persona) in a["personas"].as_array().into_iter().flatten().filter(|p| p["need"] == id).enumerate() {
for (i, persona) in a["personas"]
.as_array()
.into_iter()
.flatten()
.filter(|p| p["need"] == id)
.enumerate()
{
let who = persona["text"].as_str().unwrap_or_default();
let mut picks: [[&str; 2]; 2] = [["", ""], ["", ""]];
for (order, (first, second)) in [(&text_a, &text_b), (&text_b, &text_a)].into_iter().enumerate() {
for (order, (first, second)) in [(&text_a, &text_b), (&text_b, &text_a)]
.into_iter()
.enumerate()
{
let prompt = format!(
"You are this person: {who}\n\nTwo versions of the same web page. They lead to the same thing; only the wording differs.\n\n=== VERSION 1 ===\n{first}\n=== VERSION 2 ===\n{second}\n\nAs yourself: which version makes you feel more welcome and at ease, and which one makes it clearer what to do?\nReply as JSON: {{\"friendlier\": 1 or 2, \"clearer\": 1 or 2}}"
);
@@ -554,7 +815,13 @@ pub fn prefer(argv: &[String]) -> anyhow::Result<()> {
}
println!(
"{:<28} {:>5} {:>5} {:>14} {:>5} {:>5} {:>14} {:>9}",
id, wins[0][0], wins[0][1], wins[0][2], wins[1][0], wins[1][1], wins[1][2],
id,
wins[0][0],
wins[0][1],
wins[0][2],
wins[1][0],
wins[1][1],
wins[1][2],
format!("{:.0}->{:.0}", lix(&text_a), lix(&text_b))
);
}
@@ -592,29 +859,46 @@ mod tests {
{"name": "Get a price", "description": "For owners.", "consequence": ["Send it"], "encouragements": ["We answer within two days."], "features": [{"name": "Your brief", "requirements": [{"name": "email", "label": "Email"}]}]},
{"name": "Work with us", "description": "For crews."}]});
let rich = page_text(&step, Some(&repo_page), Some(&next), false);
assert!(rich.contains("Button: Send it") && rich.contains("We answer within two days.") && rich.contains("Asks for: Email"));
assert!(
rich.contains("Button: Send it")
&& rich.contains("We answer within two days.")
&& rich.contains("Asks for: Email")
);
assert!(!rich.contains("asks for: Email, The build"));
let twin = page_text(&step, Some(&repo_page), Some(&next), true);
assert!(!twin.contains("Send it") && !twin.contains("Your brief"));
let twin = page_text(&step, None, Some(&next), true);
assert!(twin.contains("What brings you here?") && twin.contains("Option 2"));
assert!(!twin.contains("asks for"));
assert!(!twin.contains("Get a price") && !twin.contains("Email") && !twin.contains("project manager"));
assert!(
!twin.contains("Get a price")
&& !twin.contains("Email")
&& !twin.contains("project manager")
);
// Readability is measured on the sentences, not the headings.
let prose = prose_text(&step, Some(&next));
assert_eq!(prose, "What brings you here?\nFor owners.\nFor crews.\nA project manager reads it.");
assert_eq!(
prose,
"What brings you here?\nFor owners.\nFor crews.\nA project manager reads it."
);
}
#[test]
fn a_mail_is_read_as_it_arrives_and_the_invite_is_one_of_them() {
let got = as_received("Hei {name}, se {site}/decide/x?chain={chain}. {link} {ikke en nøkkel}", "Tomter Vel");
let got = as_received(
"Hei {name}, se {site}/decide/x?chain={chain}. {link} {ikke en nøkkel}",
"Tomter Vel",
);
assert_eq!(got, "Hei [name], se https://example.site/decide/x?chain=8f3a1c. https://id.example.site/reset/8f3a1c {ikke en nøkkel}");
let dir = std::env::temp_dir().join(format!("portal-trainer-mails-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
std::fs::write(dir.join("aggregates.yaml"), "aggregates:\n - bucket: b\n initial: open\n states:\n open: { event: received, mail: { to: submitter, subject: Fått, body: \"Vi har det, {name}.\" } }\n").unwrap();
std::fs::write(dir.join("site.yaml"), "title: T\nmail: { from: x@y.no, invite: { subject: Konto, body: \"Hei {name}\\n{link}\" } }\n").unwrap();
let mails = mail_texts(&dir).unwrap();
assert_eq!(mails.iter().map(|m| m.0.as_str()).collect::<Vec<_>>(), ["b:open", "invite"]);
assert_eq!(
mails.iter().map(|m| m.0.as_str()).collect::<Vec<_>>(),
["b:open", "invite"]
);
assert_eq!(mails[0].2, "Vi har det, .");
std::fs::remove_dir_all(dir).ok();
}
@@ -623,8 +907,16 @@ mod tests {
fn the_followup_page_is_read_from_the_repo_by_following_the_action() {
let dir = std::env::temp_dir().join(format!("portal-trainer-rate-{}", std::process::id()));
std::fs::create_dir_all(dir.join("questions/received")).unwrap();
std::fs::write(dir.join("questions/index.yaml"), "name: What?\nalternatives:\n - name: Get a price\n action: /received/price\n").unwrap();
std::fs::write(dir.join("questions/received/price.yaml"), "name: What happens now?\ndescription: We read it.\nalternatives:\n - name: Back\n").unwrap();
std::fs::write(
dir.join("questions/index.yaml"),
"name: What?\nalternatives:\n - name: Get a price\n action: /received/price\n",
)
.unwrap();
std::fs::write(
dir.join("questions/received/price.yaml"),
"name: What happens now?\ndescription: We read it.\nalternatives:\n - name: Back\n",
)
.unwrap();
let step = json!({"page": "/", "alternative": "Get a price"});
let next = followup_page(Some(&dir), &step).unwrap();
assert_eq!(next["name"], "What happens now?");
+84 -17
View File
@@ -47,19 +47,35 @@ struct Showing {
/// Per bucket, per storyline index, how many arrived and how many
/// ended in a state the need calls success (or, without `success`,
/// any finished state).
fn showings(counts: &BTreeMap<String, BTreeMap<String, BTreeMap<String, u64>>>, needs: &[Need]) -> BTreeMap<String, Vec<Showing>> {
fn showings(
counts: &BTreeMap<String, BTreeMap<String, BTreeMap<String, u64>>>,
needs: &[Need],
) -> BTreeMap<String, Vec<Showing>> {
let mut out = BTreeMap::new();
for (bucket, per_story) in counts {
let well: Vec<&str> = needs
.iter()
.filter(|n| &n.lands_in == bucket)
.flat_map(|n| if n.success.is_empty() { n.done_when.iter() } else { n.success.iter() })
.flat_map(|n| {
if n.success.is_empty() {
n.done_when.iter()
} else {
n.success.iter()
}
})
.map(String::as_str)
.collect();
let arity = per_story.keys().filter_map(|k| k.parse::<usize>().ok()).max().map(|m| m + 1).unwrap_or(1);
let arity = per_story
.keys()
.filter_map(|k| k.parse::<usize>().ok())
.max()
.map(|m| m + 1)
.unwrap_or(1);
let mut rows: Vec<Showing> = (0..arity).map(|_| Showing::default()).collect();
for (story, states) in per_story {
let Ok(n) = story.parse::<usize>() else { continue };
let Ok(n) = story.parse::<usize>() else {
continue;
};
for (state, count) in states {
rows[n].arrived += count;
if well.contains(&state.as_str()) {
@@ -73,16 +89,32 @@ fn showings(counts: &BTreeMap<String, BTreeMap<String, BTreeMap<String, u64>>>,
}
pub fn run(argv: &[String]) -> anyhow::Result<()> {
let value = |flag: &str| argv.iter().position(|a| a == flag).and_then(|i| argv.get(i + 1)).cloned();
let site = value("--site").ok_or_else(|| anyhow!("usage: portal-trainer report --site <https://the.site> [--needs <needs.yaml>]"))?;
let value = |flag: &str| {
argv.iter()
.position(|a| a == flag)
.and_then(|i| argv.get(i + 1))
.cloned()
};
let site = value("--site").ok_or_else(|| {
anyhow!("usage: portal-trainer report --site <https://the.site> [--needs <needs.yaml>]")
})?;
let url = format!("{}/report", site.trim_end_matches('/'));
let fetched = std::process::Command::new("curl").args(["-sSf", "--max-time", "30", &url]).output().context("running curl")?;
let fetched = std::process::Command::new("curl")
.args(["-sSf", "--max-time", "30", &url])
.output()
.context("running curl")?;
if !fetched.status.success() {
bail!("could not read {url}: {}", String::from_utf8_lossy(&fetched.stderr).trim());
bail!(
"could not read {url}: {}",
String::from_utf8_lossy(&fetched.stderr).trim()
);
}
let report: Report = serde_json::from_slice(&fetched.stdout).with_context(|| format!("reading {url}"))?;
let report: Report =
serde_json::from_slice(&fetched.stdout).with_context(|| format!("reading {url}"))?;
let needs: Needs = match value("--needs") {
Some(path) => serde_yaml::from_str(&std::fs::read_to_string(&path).with_context(|| format!("reading {path}"))?)?,
Some(path) => serde_yaml::from_str(
&std::fs::read_to_string(&path).with_context(|| format!("reading {path}"))?,
)?,
None => Needs::default(),
};
println!("{site}: {} storyline(s)", report.storylines);
@@ -97,11 +129,26 @@ pub fn run(argv: &[String]) -> anyhow::Result<()> {
for (bucket, per_story) in &rows {
println!("\n{bucket}");
for (n, showing) in per_story.iter().enumerate() {
let rate = if showing.arrived > 0 { format!("{:.0}%", 100.0 * showing.ended_well as f64 / showing.arrived as f64) } else { "-".to_string() };
println!(" storyline {n}: {} arrived, {} ended well ({rate})", showing.arrived, showing.ended_well);
let rate = if showing.arrived > 0 {
format!(
"{:.0}%",
100.0 * showing.ended_well as f64 / showing.arrived as f64
)
} else {
"-".to_string()
};
println!(
" storyline {n}: {} arrived, {} ended well ({rate})",
showing.arrived, showing.ended_well
);
}
if per_story.len() > 1 {
let best = per_story.iter().enumerate().max_by_key(|(_, s)| (s.ended_well, s.arrived)).map(|(n, _)| n).unwrap_or(0);
let best = per_story
.iter()
.enumerate()
.max_by_key(|(_, s)| (s.ended_well, s.arrived))
.map(|(n, _)| n)
.unwrap_or(0);
println!(" most so far: storyline {best}");
}
}
@@ -116,12 +163,32 @@ mod tests {
fn a_storyline_is_scored_on_what_arrived_and_what_ended_well() {
let counts: BTreeMap<String, BTreeMap<String, BTreeMap<String, u64>>> =
serde_json::from_str(r#"{"hazards": {"0": {"open": 3, "fixed": 4}, "1": {"open": 1, "fixed": 9, "dropped": 2}}}"#).unwrap();
let needs = vec![Need { lands_in: "hazards".into(), done_when: vec!["fixed".into(), "dropped".into()], success: vec!["fixed".into()] }];
let needs = vec![Need {
lands_in: "hazards".into(),
done_when: vec!["fixed".into(), "dropped".into()],
success: vec!["fixed".into()],
}];
let rows = showings(&counts, &needs);
assert_eq!(rows["hazards"][0], Showing { arrived: 7, ended_well: 4 });
assert_eq!(rows["hazards"][1], Showing { arrived: 12, ended_well: 9 });
assert_eq!(
rows["hazards"][0],
Showing {
arrived: 7,
ended_well: 4
}
);
assert_eq!(
rows["hazards"][1],
Showing {
arrived: 12,
ended_well: 9
}
);
// Without a success list, any finished state counts.
let any = vec![Need { lands_in: "hazards".into(), done_when: vec!["fixed".into(), "dropped".into()], success: vec![] }];
let any = vec![Need {
lands_in: "hazards".into(),
done_when: vec!["fixed".into(), "dropped".into()],
success: vec![],
}];
assert_eq!(showings(&counts, &any)["hazards"][1].ended_well, 11);
}
}
+77 -14
View File
@@ -11,7 +11,12 @@ use std::path::{Path, PathBuf};
use std::process::Command;
pub fn run(argv: &[String]) -> anyhow::Result<()> {
let flag = |name: &str| argv.iter().position(|a| a == name).and_then(|i| argv.get(i + 1)).cloned();
let flag = |name: &str| {
argv.iter()
.position(|a| a == name)
.and_then(|i| argv.get(i + 1))
.cloned()
};
let usage = "usage: portal-trainer score --site <content dir> --out <dir> [--iris <path>] [--llm 'cmd']";
let site = PathBuf::from(flag("--site").ok_or_else(|| anyhow!(usage))?);
let out = PathBuf::from(flag("--out").ok_or_else(|| anyhow!(usage))?);
@@ -27,52 +32,102 @@ pub fn run(argv: &[String]) -> anyhow::Result<()> {
let tasks_skim = out.join("tasks-skim.jsonl");
let model = out.join("model.json");
let mut log = String::new();
log.push_str(&lint(&iris, &questions, &["--needs-tasks", &s(&tasks_full), "--needs-sim", &s(&model)])?);
log.push_str(&lint(&iris, &questions, &["--needs-tasks", &s(&tasks_skim), "--skim"])?);
log.push_str(&lint(
&iris,
&questions,
&["--needs-tasks", &s(&tasks_full), "--needs-sim", &s(&model)],
)?);
log.push_str(&lint(
&iris,
&questions,
&["--needs-tasks", &s(&tasks_skim), "--skim"],
)?);
std::fs::write(out.join("iris.log"), &log)?;
let mut summary = serde_json::Map::new();
for (mode, tasks, skim) in [("full", &tasks_full, false), ("skim", &tasks_skim, true)] {
eprintln!("score: the visitor answers {mode}");
let answers = crate::trainer::answer_blind(&cmd, &std::fs::read_to_string(tasks)?, Some(&site), 0)?;
let answers =
crate::trainer::answer_blind(&cmd, &std::fs::read_to_string(tasks)?, Some(&site), 0)?;
let answers_path = out.join(format!("answers-{mode}.jsonl"));
std::fs::write(&answers_path, &answers)?;
let (tasks_arg, answers_arg) = (s(tasks), s(&answers_path));
let mut args = vec!["--needs-tasks", tasks_arg.as_str(), "--needs-score", answers_arg.as_str()];
let mut args = vec![
"--needs-tasks",
tasks_arg.as_str(),
"--needs-score",
answers_arg.as_str(),
];
if skim {
args.push("--skim");
}
let report = lint(&iris, &questions, &args)?;
println!("== {}", if skim { "headings only" } else { "with descriptions" });
println!(
"== {}",
if skim {
"headings only"
} else {
"with descriptions"
}
);
let mut right = None;
for line in report.lines() {
if line.starts_with("MISROUTE") || line.starts_with("HESITANT") || line.starts_with("UNANSWERED") {
if line.starts_with("MISROUTE")
|| line.starts_with("HESITANT")
|| line.starts_with("UNANSWERED")
{
println!("{line}");
}
if let Some(rest) = line.strip_prefix("wording: ") {
// "wording: 19/27 routed right (70%)"
let mut parts = rest.split_whitespace().next().unwrap_or_default().split('/');
let mut parts = rest
.split_whitespace()
.next()
.unwrap_or_default()
.split('/');
let a: usize = parts.next().and_then(|x| x.parse().ok()).unwrap_or(0);
let b: usize = parts.next().and_then(|x| x.parse().ok()).unwrap_or(0);
right = Some((a, b));
println!("{line}");
}
}
let (a, b) = right.ok_or_else(|| anyhow!("iris printed no wording score for {mode}; its output:\n{report}"))?;
let (a, b) = right.ok_or_else(|| {
anyhow!("iris printed no wording score for {mode}; its output:\n{report}")
})?;
summary.insert(mode.to_string(), serde_json::json!({ "right": a, "of": b }));
}
println!("== rate");
let rated = out.join("rated");
let rate_argv: Vec<String> = ["rate", "--model", &s(&model), "--out", &s(&rated), "--repo", &s(&site), "--llm", &cmd].iter().map(|x| x.to_string()).collect();
let rate_argv: Vec<String> = [
"rate",
"--model",
&s(&model),
"--out",
&s(&rated),
"--repo",
&s(&site),
"--llm",
&cmd,
]
.iter()
.map(|x| x.to_string())
.collect();
crate::trainer::rate::rate(&rate_argv)?;
summary.insert("site".into(), serde_json::json!(s(&site)));
summary.insert("visitor".into(), serde_json::json!(cmd));
std::fs::write(out.join("summary.json"), serde_json::to_string_pretty(&serde_json::Value::Object(summary.clone()))?)?;
std::fs::write(
out.join("summary.json"),
serde_json::to_string_pretty(&serde_json::Value::Object(summary.clone()))?,
)?;
println!(
"== summary: with descriptions {} of {}, headings only {} of {} (rate.json in {})",
summary["full"]["right"], summary["full"]["of"], summary["skim"]["right"], summary["skim"]["of"], rated.display()
summary["full"]["right"],
summary["full"]["of"],
summary["skim"]["right"],
summary["skim"]["of"],
rated.display()
);
Ok(())
}
@@ -92,9 +147,17 @@ fn lint(iris: &str, questions: &Path, args: &[&str]) -> anyhow::Result<String> {
.args(args)
.output()
.with_context(|| format!("running {iris} (pass --iris <path> if it is not on PATH)"))?;
let text = format!("{}{}", String::from_utf8_lossy(&output.stdout), String::from_utf8_lossy(&output.stderr));
let text = format!(
"{}{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
if !output.status.success() {
bail!("{iris} check --path {} {} failed:\n{text}", questions.display(), args.join(" "));
bail!(
"{iris} check --path {} {} failed:\n{text}",
questions.display(),
args.join(" ")
);
}
Ok(text)
}
+183 -37
View File
@@ -127,7 +127,12 @@ pub fn run(argv: &[String]) -> anyhow::Result<()> {
continue;
}
let bucket = &model["buckets"][need["lands_in"].as_str().unwrap_or_default()];
let done: Vec<&str> = need["done_when"].as_array().into_iter().flatten().filter_map(Value::as_str).collect();
let done: Vec<&str> = need["done_when"]
.as_array()
.into_iter()
.flatten()
.filter_map(Value::as_str)
.collect();
for (run, route) in all_routes(bucket, &done).into_iter().enumerate() {
let mut states = vec![bucket["initial"].as_str().unwrap_or("open").to_string()];
states.extend(route.iter().map(|m| m.to.clone()));
@@ -137,7 +142,10 @@ pub fn run(argv: &[String]) -> anyhow::Result<()> {
run,
target: states.last().cloned().unwrap_or_default(),
record: BTreeMap::from([("note".to_string(), json!("dry run"))]),
pressed: route.iter().map(|m| format!("{} -> {}", m.from, m.label)).collect(),
pressed: route
.iter()
.map(|m| format!("{} -> {}", m.from, m.label))
.collect(),
moved_by: route.iter().map(|m| m.by.clone()).collect(),
touches: route.len(),
ask_backs: Vec::new(),
@@ -148,9 +156,16 @@ pub fn run(argv: &[String]) -> anyhow::Result<()> {
writeln!(cases_file, "{}", serde_json::to_string(&case)?)?;
}
}
std::fs::write(out.join("score.json"), serde_json::to_string_pretty(&scores)?)?;
std::fs::write(
out.join("score.json"),
serde_json::to_string_pretty(&scores)?,
)?;
let relays: usize = scores.values().map(|s| s.cases).sum();
println!("dry run: {relays} route(s) across {} need(s) written to {}", scores.len(), out.join("cases.jsonl").display());
println!(
"dry run: {relays} route(s) across {} need(s) written to {}",
scores.len(),
out.join("cases.jsonl").display()
);
return Ok(());
}
@@ -180,7 +195,10 @@ pub fn run(argv: &[String]) -> anyhow::Result<()> {
writeln!(cases_file, "{}", serde_json::to_string(&case)?)?;
}
}
std::fs::write(out.join("score.json"), serde_json::to_string_pretty(&scores)?)?;
std::fs::write(
out.join("score.json"),
serde_json::to_string_pretty(&scores)?,
)?;
print_scores(&scores);
Ok(())
}
@@ -257,7 +275,8 @@ fn play(
);
let reply = ask(visitor_cmd, &prompt, &seed(i * 10 + 1))?;
if reply["abandon"] == true {
case.outcome = Outcome::Abandoned(reply["why"].as_str().unwrap_or_default().to_string());
case.outcome =
Outcome::Abandoned(reply["why"].as_str().unwrap_or_default().to_string());
return Ok(case);
}
if let Some(values) = reply["values"].as_object() {
@@ -277,9 +296,18 @@ fn play(
// collected, could this role really do what the button claims?
let bucket_name = need["lands_in"].as_str().unwrap_or_default();
let bucket = &model["buckets"][bucket_name];
let done: Vec<&str> = need["done_when"].as_array().into_iter().flatten().filter_map(Value::as_str).collect();
let done: Vec<&str> = need["done_when"]
.as_array()
.into_iter()
.flatten()
.filter_map(Value::as_str)
.collect();
let initial = bucket["initial"].as_str().unwrap_or("open").to_string();
let target = done.get(run % done.len().max(1)).copied().unwrap_or_default().to_string();
let target = done
.get(run % done.len().max(1))
.copied()
.unwrap_or_default()
.to_string();
case.target = target.clone();
case.states.push(initial.clone());
let Some(route) = route_to(bucket, &initial, &target) else {
@@ -304,7 +332,11 @@ fn play(
serde_json::to_string_pretty(&case.record)?,
);
let reply = ask(handler_cmd, &prompt, &seed(100 + step))?;
let question = reply["need_to_know"].as_str().unwrap_or_default().trim().to_string();
let question = reply["need_to_know"]
.as_str()
.unwrap_or_default()
.trim()
.to_string();
if reply["can_act"] == false && !question.is_empty() {
let answer = ask(
visitor_cmd,
@@ -314,7 +346,10 @@ fn play(
&seed(200 + step),
)?;
let n = case.ask_backs.len() + 1;
case.record.insert(format!("follow_up_{n}"), json!({ "asked": question, "answered": answer["answer"] }));
case.record.insert(
format!("follow_up_{n}"),
json!({ "asked": question, "answered": answer["answer"] }),
);
case.ask_backs.push(AskBack {
state: from.clone(),
button: label.clone(),
@@ -343,21 +378,47 @@ pub(crate) struct Move {
/// desk, and the submitter's own link. Roles overlap on one aggregate,
/// so a route is a relay, not one desk's work.
pub(crate) fn all_moves(bucket: &Value) -> Vec<Move> {
let legal = |from: &str, to: &str| bucket["transitions"][from].as_array().is_some_and(|a| a.iter().any(|x| x == to));
let legal = |from: &str, to: &str| {
bucket["transitions"][from]
.as_array()
.is_some_and(|a| a.iter().any(|x| x == to))
};
let mut moves: Vec<Move> = Vec::new();
for desk in bucket["desks"].as_array().into_iter().flatten() {
for m in desk["moves"].as_array().into_iter().flatten() {
let (Some(from), Some(label), Some(to)) = (m["from"].as_str(), m["label"].as_str(), m["to"].as_str()) else { continue };
let (Some(from), Some(label), Some(to)) =
(m["from"].as_str(), m["label"].as_str(), m["to"].as_str())
else {
continue;
};
if legal(from, to) && !moves.iter().any(|x| x.from == from && x.to == to) {
moves.push(Move { from: from.into(), label: label.into(), to: to.into(), by: desk["group"].as_str().unwrap_or("ungated").into() });
moves.push(Move {
from: from.into(),
label: label.into(),
to: to.into(),
by: desk["group"].as_str().unwrap_or("ungated").into(),
});
}
}
}
let states: Vec<String> = bucket["transitions"].as_object().map(|t| t.keys().cloned().collect()).unwrap_or_default();
for to in bucket["self_moves"].as_array().into_iter().flatten().filter_map(Value::as_str) {
let states: Vec<String> = bucket["transitions"]
.as_object()
.map(|t| t.keys().cloned().collect())
.unwrap_or_default();
for to in bucket["self_moves"]
.as_array()
.into_iter()
.flatten()
.filter_map(Value::as_str)
{
for from in &states {
if legal(from, to) && !moves.iter().any(|x| &x.from == from && x.to == to) {
moves.push(Move { from: from.clone(), label: "their own link".into(), to: to.into(), by: SUBMITTER.into() });
moves.push(Move {
from: from.clone(),
label: "their own link".into(),
to: to.into(),
by: SUBMITTER.into(),
});
}
}
}
@@ -388,7 +449,13 @@ fn route_to(bucket: &Value, from: &str, to: &str) -> Option<Vec<Move>> {
/// dry run walks, so that each button of each role is pressed at least
/// once against the real engine.
pub(crate) fn all_routes(bucket: &Value, done: &[&str]) -> Vec<Vec<Move>> {
fn walk(state: &str, done: &[&str], moves: &[Move], path: &mut Vec<Move>, out: &mut Vec<Vec<Move>>) {
fn walk(
state: &str,
done: &[&str],
moves: &[Move],
path: &mut Vec<Move>,
out: &mut Vec<Vec<Move>>,
) {
if done.contains(&state) {
out.push(path.clone());
return;
@@ -403,7 +470,13 @@ pub(crate) fn all_routes(bucket: &Value, done: &[&str]) -> Vec<Vec<Move>> {
}
}
let mut out = Vec::new();
walk(bucket["initial"].as_str().unwrap_or("open"), done, &all_moves(bucket), &mut Vec::new(), &mut out);
walk(
bucket["initial"].as_str().unwrap_or("open"),
done,
&all_moves(bucket),
&mut Vec::new(),
&mut out,
);
out
}
@@ -424,7 +497,11 @@ pub(crate) fn resolve_choice(names: &[&str], raw: &str) -> String {
/// be (`FORMAT=json`, honoured by engines/ollama.sh) and the first
/// object in the reply is taken either way.
pub(crate) fn ask(cmd: &str, prompt: &str, seed: &str) -> anyhow::Result<Value> {
let reply = llm_with(cmd, prompt, &[("FORMAT", "json"), ("SEED", seed), ("TEMPERATURE", "0.7")])?;
let reply = llm_with(
cmd,
prompt,
&[("FORMAT", "json"), ("SEED", seed), ("TEMPERATURE", "0.7")],
)?;
Ok(first_object(&reply).unwrap_or(Value::Null))
}
@@ -462,7 +539,11 @@ fn tally(score: &mut NeedScore, case: &Case) {
score.buttons.entry(button.clone()).or_default().0 += 1;
}
for a in &case.ask_backs {
score.buttons.entry(format!("{} -> {}", a.state, a.button)).or_default().1 += 1;
score
.buttons
.entry(format!("{} -> {}", a.state, a.button))
.or_default()
.1 += 1;
}
match &case.outcome {
Outcome::Finished(_) => score.finished += 1,
@@ -476,7 +557,9 @@ fn tally(score: &mut NeedScore, case: &Case) {
}
Outcome::Abandoned(why) => {
score.abandoned += 1;
score.abandon_reasons.push(format!("{}: {why}", case.persona));
score
.abandon_reasons
.push(format!("{}: {why}", case.persona));
}
}
}
@@ -501,7 +584,15 @@ fn per_case(n: usize, s: &NeedScore) -> String {
fn print_scores(scores: &BTreeMap<String, NeedScore>) {
println!(
"\n{:<28} {:>5} {:>9} {:>9} {:>10} {:>9} {:>8} {:>10} {:>13}",
"need", "cases", "finished", "misroute", "abandoned", "no route", "touches", "ask-backs", "unanswerable"
"need",
"cases",
"finished",
"misroute",
"abandoned",
"no route",
"touches",
"ask-backs",
"unanswerable"
);
for (need, s) in scores {
println!(
@@ -529,7 +620,11 @@ fn print_scores(scores: &BTreeMap<String, NeedScore>) {
" {need} [{} / {}]{} {}",
a.state,
a.button,
if a.answerable { "" } else { " (visitor did not know)" },
if a.answerable {
""
} else {
" (visitor did not know)"
},
a.question
);
}
@@ -550,10 +645,16 @@ fn print_scores(scores: &BTreeMap<String, NeedScore>) {
pub fn compare(argv: &[String]) -> anyhow::Result<()> {
let (a, b) = match (argv.get(1), argv.get(2)) {
(Some(a), Some(b)) => (a, b),
_ => return Err(anyhow!("usage: portal-trainer compare <a/score.json> <b/score.json>")),
_ => {
return Err(anyhow!(
"usage: portal-trainer compare <a/score.json> <b/score.json>"
))
}
};
let load = |p: &String| -> anyhow::Result<BTreeMap<String, NeedScore>> {
Ok(serde_json::from_str(&std::fs::read_to_string(p).with_context(|| format!("reading {p}"))?)?)
Ok(serde_json::from_str(
&std::fs::read_to_string(p).with_context(|| format!("reading {p}"))?,
)?)
};
let (a, b) = (load(a)?, load(b)?);
println!(
@@ -565,11 +666,27 @@ pub fn compare(argv: &[String]) -> anyhow::Result<()> {
println!(
"{:<28} {:>14} {:>14} {:>14} {:>16} {:>16}",
need,
format!("{} -> {}", pct(sa.misrouted, sa.cases), pct(sb.misrouted, sb.cases)),
format!("{} -> {}", pct(sa.abandoned, sa.cases), pct(sb.abandoned, sb.cases)),
format!(
"{} -> {}",
pct(sa.misrouted, sa.cases),
pct(sb.misrouted, sb.cases)
),
format!(
"{} -> {}",
pct(sa.abandoned, sa.cases),
pct(sb.abandoned, sb.cases)
),
format!("{} -> {}", pct(sa.stuck, sa.cases), pct(sb.stuck, sb.cases)),
format!("{} -> {}", per_case(sa.touches, sa), per_case(sb.touches, sb)),
format!("{} -> {}", per_case(sa.ask_backs, sa), per_case(sb.ask_backs, sb)),
format!(
"{} -> {}",
per_case(sa.touches, sa),
per_case(sb.touches, sb)
),
format!(
"{} -> {}",
per_case(sa.ask_backs, sa),
per_case(sb.ask_backs, sb)
),
);
}
Ok(())
@@ -589,8 +706,15 @@ mod tests {
#[test]
fn a_choice_is_matched_by_its_heading_however_the_model_dresses_it() {
let names = ["Ask about working", "Ask about working with us", "Get a price"];
assert_eq!(resolve_choice(&names, "Ask about working with us: For carpenters who..."), "Ask about working with us");
let names = [
"Ask about working",
"Ask about working with us",
"Get a price",
];
assert_eq!(
resolve_choice(&names, "Ask about working with us: For carpenters who..."),
"Ask about working with us"
);
assert_eq!(resolve_choice(&names, " get a PRICE "), "Get a price");
assert_eq!(resolve_choice(&names, "Something else"), "Something else");
}
@@ -620,7 +744,10 @@ mod tests {
// Losing a quoted job is the other group's button, so the way
// from open to that ending is a relay: two roles, one case.
let route = route_to(&bucket(), "surveyed", "lost").unwrap();
let relay: Vec<(&str, &str)> = route.iter().map(|m| (m.label.as_str(), m.by.as_str())).collect();
let relay: Vec<(&str, &str)> = route
.iter()
.map(|m| (m.label.as_str(), m.by.as_str()))
.collect();
assert_eq!(relay, [("Quote sent", "pm"), ("Theirs", "other")]);
assert_eq!(route_to(&bucket(), "open", "open").unwrap().len(), 0);
assert!(route_to(&bucket(), "won", "open").is_none());
@@ -630,7 +757,10 @@ mod tests {
fn the_submitters_own_link_is_a_move_like_any_other() {
let mut b = bucket();
b["self_moves"] = json!(["lost"]);
let theirs: Vec<Move> = all_moves(&b).into_iter().filter(|m| m.by == SUBMITTER).collect();
let theirs: Vec<Move> = all_moves(&b)
.into_iter()
.filter(|m| m.by == SUBMITTER)
.collect();
// Only where the machine allows it and no desk already holds it:
// open->lost is the project manager's and quoted->lost the other
// group's, which leaves surveyed->lost to the link.
@@ -641,7 +771,10 @@ mod tests {
#[test]
fn a_dry_run_walks_every_route_to_every_ending() {
let routes = all_routes(&bucket(), &["won", "lost"]);
let ends: Vec<String> = routes.iter().map(|r| format!("{}:{}", r.len(), r.last().unwrap().to)).collect();
let ends: Vec<String> = routes
.iter()
.map(|r| format!("{}:{}", r.len(), r.last().unwrap().to))
.collect();
// open->lost; open->surveyed->quoted->won; and the relay
// open->surveyed->quoted->lost, closed by the other group.
assert_eq!(ends.len(), 3, "{ends:?}");
@@ -664,15 +797,28 @@ mod tests {
moved_by: vec!["pm".into(), "pm".into()],
touches: 3,
ask_backs: vec![
AskBack { state: "open".into(), button: "Looked".into(), question: "Which floor?".into(), answerable: true },
AskBack { state: "surveyed".into(), button: "Quote sent".into(), question: "Load-bearing?".into(), answerable: false },
AskBack {
state: "open".into(),
button: "Looked".into(),
question: "Which floor?".into(),
answerable: true,
},
AskBack {
state: "surveyed".into(),
button: "Quote sent".into(),
question: "Load-bearing?".into(),
answerable: false,
},
],
outcome,
};
tally(&mut score, &base(Outcome::Finished("won".into())));
tally(&mut score, &base(Outcome::Stuck("no route".into())));
tally(&mut score, &base(Outcome::Misrouted("Wrong door".into())));
assert_eq!((score.cases, score.finished, score.stuck, score.misrouted), (3, 1, 1, 1));
assert_eq!(
(score.cases, score.finished, score.stuck, score.misrouted),
(3, 1, 1, 1)
);
assert_eq!((score.ask_backs, score.unanswerable), (6, 3));
}
}
+43 -11
View File
@@ -130,13 +130,18 @@ pub async fn upload(
}
}
let (filename, content_type, bytes) =
file.ok_or((StatusCode::BAD_REQUEST, "no file field in request".to_string()))?;
let (filename, content_type, bytes) = file.ok_or((
StatusCode::BAD_REQUEST,
"no file field in request".to_string(),
))?;
// Nothing has been written yet. Everything below decides whether
// anything will be.
if !may_upload(&state, &session, &question_id, &alternative).await {
return Err((StatusCode::FORBIDDEN, "not authorized to upload here".to_string()));
return Err((
StatusCode::FORBIDDEN,
"not authorized to upload here".to_string(),
));
}
let key = format!("{}-{filename}", uuid::Uuid::new_v4());
@@ -235,7 +240,10 @@ alternatives:
#[test]
fn an_upload_must_name_a_page_that_asked_for_a_file() {
let qs = questions(PAGE);
assert_eq!(page_taking_files(&qs, "/", "Send drawings"), Some("/".to_string()));
assert_eq!(
page_taking_files(&qs, "/", "Send drawings"),
Some("/".to_string())
);
// An alternative on the same page that takes no file.
assert_eq!(page_taking_files(&qs, "/", "Just ask"), None);
// Names that are not there at all.
@@ -246,7 +254,10 @@ alternatives:
#[test]
fn a_closed_alternative_takes_nothing() {
let qs = questions(&PAGE.replace(" record_as: enquiries\n", " record_as: enquiries\n disabled: true\n"));
let qs = questions(&PAGE.replace(
" record_as: enquiries\n",
" record_as: enquiries\n disabled: true\n",
));
assert_eq!(page_taking_files(&qs, "/", "Send drawings"), None);
}
}
@@ -277,15 +288,24 @@ pub async fn attachment(
// cannot be used to learn which buckets, records or fields exist.
let missing = || (StatusCode::NOT_FOUND, "not found".to_string());
let store = state.jetstream.get_key_value(&bucket).await.map_err(|_| missing())?;
let bytes = store.get(&record).await.map_err(|_| missing())?.ok_or_else(missing)?;
let store = state
.jetstream
.get_key_value(&bucket)
.await
.map_err(|_| missing())?;
let bytes = store
.get(&record)
.await
.map_err(|_| missing())?
.ok_or_else(missing)?;
let as_listed: serde_json::Value = serde_json::from_slice(&bytes).map_err(|_| missing())?;
// Through a list the caller may read, and one that shows this
// record: a list of "my cases" opens my files and nobody else's.
if !may_read_record(&state, &session, &bucket, &as_listed).await {
return Err(missing());
}
let answer: crate::answers::Answer = serde_json::from_value(as_listed).map_err(|_| missing())?;
let answer: crate::answers::Answer =
serde_json::from_value(as_listed).map_err(|_| missing())?;
let key = answer
.responses
.get(&field)
@@ -310,13 +330,22 @@ pub async fn attachment(
.body
.collect()
.await
.map_err(|e| (StatusCode::BAD_GATEWAY, format!("reading the file failed: {e}")))?
.map_err(|e| {
(
StatusCode::BAD_GATEWAY,
format!("reading the file failed: {e}"),
)
})?
.into_bytes();
// The stored key is `<uuid>-<original name>`; the person who gets
// the file should see the name they sent, not the uuid.
let filename = key.split_once('-').map(|(_, rest)| rest).unwrap_or(key);
let filename = filename.rsplit('/').next().unwrap_or(filename).replace(['"', '\\'], "");
let filename = filename
.rsplit('/')
.next()
.unwrap_or(filename)
.replace(['"', '\\'], "");
Ok((
[
(axum::http::header::CONTENT_TYPE, content_type),
@@ -349,7 +378,10 @@ async fn may_read_record(
question.alternatives.iter().any(|alternative| {
alternative.features.iter().any(|feature| {
feature.resource.as_ref().and_then(|r| r.bucket()) == Some(bucket)
&& feature.resource.as_ref().is_some_and(|r| crate::resource::matches_filter(r, record, &params))
&& feature
.resource
.as_ref()
.is_some_and(|r| crate::resource::matches_filter(r, record, &params))
&& access.allows(
&principal,
crate::access::Action::Read,
+219 -41
View File
@@ -34,27 +34,51 @@ async fn a_submission_is_created_moved_refused_and_read_back_through_the_real_en
let now = chrono::Utc::now().timestamp_millis();
// The submission: a KV record in the initial state, plus the first event.
store_answer(&js, &schemas, &bucket, "rec-1".into(), "/", "Hire us", &json!({"email": "a@b.example"}), now, 0)
store_answer(
&js,
&schemas,
&bucket,
"rec-1".into(),
"/",
"Hire us",
&json!({"email": "a@b.example"}),
now,
0,
)
.await
.expect("store");
let store = js.get_key_value(&bucket).await.expect("bucket exists");
let stored: Answer = serde_json::from_slice(&store.get("rec-1").await.unwrap().expect("record")).unwrap();
let stored: Answer =
serde_json::from_slice(&store.get("rec-1").await.unwrap().expect("record")).unwrap();
assert_eq!(stored.state, "open");
assert_eq!(stored.responses["email"], "a@b.example");
// A desk move the machine allows.
let moved = aggregates::transition(&js, schema, "rec-1", "quoted", json!({"by": "test"}), now + 1)
let moved = aggregates::transition(
&js,
schema,
"rec-1",
"quoted",
json!({"by": "test"}),
now + 1,
)
.await
.expect("open -> quoted is declared");
assert_eq!(moved.state, "quoted");
// Moves the machine does not allow are refused, whoever asks.
let back = aggregates::transition(&js, schema, "rec-1", "open", json!({}), now + 2).await;
assert!(matches!(back, Err(TransitionError::DisallowedTransition { .. })), "{back:?}");
assert!(
matches!(back, Err(TransitionError::DisallowedTransition { .. })),
"{back:?}"
);
let skip = aggregates::transition(&js, schema, "rec-1", "won", json!({}), now + 2).await;
assert!(skip.is_ok(), "quoted -> won is declared: {skip:?}");
let nobody = aggregates::transition(&js, schema, "rec-9", "quoted", json!({}), now + 3).await;
assert!(matches!(nobody, Err(TransitionError::UnknownAggregate)), "{nobody:?}");
assert!(
matches!(nobody, Err(TransitionError::UnknownAggregate)),
"{nobody:?}"
);
// The record reads back from its own event log, in the state the
// accepted moves left it, with a sequence that moved forward.
@@ -67,7 +91,8 @@ async fn a_submission_is_created_moved_refused_and_read_back_through_the_real_en
// A reply to the mail this record was sent becomes a note on it,
// through the same store the desks read - and moves nothing: the
// record is still where the desk left it.
let mut record: Answer = serde_json::from_slice(&store.get("rec-1").await.unwrap().expect("record")).unwrap();
let mut record: Answer =
serde_json::from_slice(&store.get("rec-1").await.unwrap().expect("record")).unwrap();
let state_before_the_note = record.state.clone();
let reply = portal::mail::Incoming {
v: portal::mail::CONTRACT,
@@ -81,13 +106,20 @@ async fn a_submission_is_created_moved_refused_and_read_back_through_the_real_en
};
let note = portal::mail::note_for(&reply, &record).expect("the address matches the record");
record.notes.push(note);
store.put("rec-1", serde_json::to_vec(&record).unwrap().into()).await.expect("note stored");
store
.put("rec-1", serde_json::to_vec(&record).unwrap().into())
.await
.expect("note stored");
let read_back: Answer = serde_json::from_slice(&store.get("rec-1").await.unwrap().expect("record")).unwrap();
let read_back: Answer =
serde_json::from_slice(&store.get("rec-1").await.unwrap().expect("record")).unwrap();
assert_eq!(read_back.notes.len(), 1);
assert_eq!(read_back.notes[0].text, "Thursday suits us.");
assert_eq!(read_back.notes[0].from, "a@b.example");
assert_eq!(read_back.state, state_before_the_note, "a reply is a note, never a decision");
assert_eq!(
read_back.state, state_before_the_note,
"a reply is a note, never a decision"
);
// The same reply read twice adds nothing.
assert!(portal::mail::note_for(&reply, &read_back).is_err());
// And the event log is untouched by it.
@@ -122,17 +154,39 @@ async fn a_session_survives_the_process_that_made_it() {
// A second store over the same JetStream is what the next boot of
// the binary has: nothing carried over in memory.
let next_boot = portal::sessions::NatsStore::open(&js).await.expect("reopen");
let found = next_boot.load(&record.id).await.expect("load").expect("still there");
let next_boot = portal::sessions::NatsStore::open(&js)
.await
.expect("reopen");
let found = next_boot
.load(&record.id)
.await
.expect("load")
.expect("still there");
assert_eq!(found.data.get("who"), Some(&json!("ingrid")));
// `create` never writes over somebody else's session: asked to
// create one whose id is taken, it takes a new id instead.
let mut collide = Record { id: record.id, data: Default::default(), expiry_date: record.expiry_date };
next_boot.create(&mut collide).await.expect("create beside it");
assert_ne!(collide.id, record.id, "an id already in use must not be reused");
let mut collide = Record {
id: record.id,
data: Default::default(),
expiry_date: record.expiry_date,
};
next_boot
.create(&mut collide)
.await
.expect("create beside it");
assert_ne!(
collide.id, record.id,
"an id already in use must not be reused"
);
assert_eq!(
next_boot.load(&record.id).await.expect("load").expect("original intact").data.get("who"),
next_boot
.load(&record.id)
.await
.expect("load")
.expect("original intact")
.data
.get("who"),
Some(&json!("ingrid")),
"the original session must be untouched"
);
@@ -146,11 +200,19 @@ async fn a_session_survives_the_process_that_made_it() {
- tower_sessions::cookie::time::Duration::minutes(1),
};
next_boot.save(&stale).await.expect("save stale");
assert!(next_boot.load(&stale.id).await.expect("load stale").is_none());
assert!(next_boot
.load(&stale.id)
.await
.expect("load stale")
.is_none());
// Signing out leaves nothing to read back.
next_boot.delete(&record.id).await.expect("delete");
assert!(next_boot.load(&record.id).await.expect("load after delete").is_none());
assert!(next_boot
.load(&record.id)
.await
.expect("load after delete")
.is_none());
next_boot.delete(&collide.id).await.ok();
}
@@ -175,12 +237,20 @@ async fn a_membership_is_invited_becomes_active_at_sign_in_and_can_be_taken_back
let questions = std::collections::HashMap::new();
let access = portal::access::Policy::from_content(&questions, &aggregates);
let state = portal::server::AppState {
leptos_options: leptos::config::LeptosOptions::builder().output_name("test").build(),
leptos_options: leptos::config::LeptosOptions::builder()
.output_name("test")
.build(),
nats,
jetstream: js,
questions: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(portal::content::Storylines::one(questions))),
aggregates: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(portal::content::Storylines::one(aggregates))),
site: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(portal::content::Storylines::one(Default::default()))),
questions: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(
portal::content::Storylines::one(questions),
)),
aggregates: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(
portal::content::Storylines::one(aggregates),
)),
site: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(
portal::content::Storylines::one(Default::default()),
)),
access: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(access)),
gitea_base: String::new(),
content_raw_base: String::new(),
@@ -198,17 +268,37 @@ async fn a_membership_is_invited_becomes_active_at_sign_in_and_can_be_taken_back
// A desk invites Kari by phone number; a decision grants her a
// second group by address. Both are invitations until she is seen.
let by_phone = Identity { sub: None, phone: Some("+4791234567".into()), email: None };
let by_mail = Identity { sub: None, phone: None, email: Some("kari@vel.no".into()) };
let first = memberships::grant(&state, styret, "Kari", &by_phone, "desk").await.expect("grant");
let by_phone = Identity {
sub: None,
phone: Some("+4791234567".into()),
email: None,
};
let by_mail = Identity {
sub: None,
phone: None,
email: Some("kari@vel.no".into()),
};
let first = memberships::grant(&state, styret, "Kari", &by_phone, "desk")
.await
.expect("grant");
assert!(first.created && first.reset_link.is_none());
let again = memberships::grant(&state, styret, "Kari", &by_phone, "desk").await.expect("grant again");
assert!(!again.created, "a second invite into the same group finds the first");
memberships::grant(&state, medlemmer, "Kari", &by_mail, "husstander:abc").await.expect("grant by mail");
let again = memberships::grant(&state, styret, "Kari", &by_phone, "desk")
.await
.expect("grant again");
assert!(
!again.created,
"a second invite into the same group finds the first"
);
memberships::grant(&state, medlemmer, "Kari", &by_mail, "husstander:abc")
.await
.expect("grant by mail");
let records = memberships::all(&state).await;
assert_eq!(records.len(), 2);
assert!(records.iter().all(|r| r.state == INVITED));
assert!(memberships::groups_of(&records, &by_phone).is_empty(), "an invitation is not yet a group");
assert!(
memberships::groups_of(&records, &by_phone).is_empty(),
"an invitation is not yet a group"
);
// She signs in: the provider knows her subject, phone and address.
let kari = portal::auth::User {
@@ -233,18 +323,47 @@ async fn a_membership_is_invited_becomes_active_at_sign_in_and_can_be_taken_back
}
// A desk takes her out of styret; the record stays, as removed.
assert!(memberships::revoke(&state, styret, &by_phone, "ola").await.expect("revoke"));
assert!(!memberships::revoke(&state, styret, &by_phone, "ola").await.expect("revoke twice"), "already out");
assert!(memberships::revoke(&state, styret, &by_phone, "ola")
.await
.expect("revoke"));
assert!(
!memberships::revoke(&state, styret, &by_phone, "ola")
.await
.expect("revoke twice"),
"already out"
);
let records = memberships::all(&state).await;
let styret_record = records.iter().find(|r| memberships::group_of(r) == styret).unwrap();
let styret_record = records
.iter()
.find(|r| memberships::group_of(r) == styret)
.unwrap();
assert_eq!(styret_record.state, REMOVED);
assert_eq!(memberships::groups_of(&records, &Identity { sub: Some("vipps-abc".into()), phone: None, email: None }), vec![medlemmer]);
assert_eq!(
memberships::groups_of(
&records,
&Identity {
sub: Some("vipps-abc".into()),
phone: None,
email: None
}
),
vec![medlemmer]
);
// Invited again: the same record, invited, and hers at the next sign-in.
let back = memberships::grant(&state, styret, "Kari", &by_mail, "desk").await.expect("re-grant");
let back = memberships::grant(&state, styret, "Kari", &by_mail, "desk")
.await
.expect("re-grant");
assert!(back.created);
let records = memberships::all(&state).await;
assert_eq!(records.iter().find(|r| memberships::group_of(r) == styret).unwrap().state, INVITED);
assert_eq!(
records
.iter()
.find(|r| memberships::group_of(r) == styret)
.unwrap()
.state,
INVITED
);
let groups = memberships::on_signed_in(&state, &kari).await;
assert!(groups.contains(&styret.to_string()));
}
@@ -265,15 +384,74 @@ async fn a_card_of_ones_own_is_replaced_not_added_to_even_on_a_double_click() {
// Kari's first card, then two sends of her second pressed at once,
// and Ola's card beside them.
store_answer(&js, &schemas, &bucket, "k1".into(), "/kontor/ansatt", "a", &card("Kari", "kari"), now, 0).await.unwrap();
store_answer(&js, &schemas, &bucket, "o1".into(), "/kontor/ansatt", "a", &card("Ola", "ola"), now + 1, 0).await.unwrap();
store_answer(&js, &schemas, &bucket, "k2".into(), "/kontor/ansatt", "a", &card("Kari N.", "kari"), now + 2, 0).await.unwrap();
store_answer(&js, &schemas, &bucket, "k3".into(), "/kontor/ansatt", "a", &card("Kari N.", "kari"), now + 2, 0).await.unwrap();
store_answer(
&js,
&schemas,
&bucket,
"k1".into(),
"/kontor/ansatt",
"a",
&card("Kari", "kari"),
now,
0,
)
.await
.unwrap();
store_answer(
&js,
&schemas,
&bucket,
"o1".into(),
"/kontor/ansatt",
"a",
&card("Ola", "ola"),
now + 1,
0,
)
.await
.unwrap();
store_answer(
&js,
&schemas,
&bucket,
"k2".into(),
"/kontor/ansatt",
"a",
&card("Kari N.", "kari"),
now + 2,
0,
)
.await
.unwrap();
store_answer(
&js,
&schemas,
&bucket,
"k3".into(),
"/kontor/ansatt",
"a",
&card("Kari N.", "kari"),
now + 2,
0,
)
.await
.unwrap();
// Both racing sends settle after storing; both must keep the same one.
let (a, b) = tokio::join!(keep_newest_own(&js, &bucket, "open", "kari"), keep_newest_own(&js, &bucket, "open", "kari"));
let (a, b) = tokio::join!(
keep_newest_own(&js, &bucket, "open", "kari"),
keep_newest_own(&js, &bucket, "open", "kari")
);
let _ = (a, b);
let kari = own_records(&js, &bucket, "kari").await;
assert_eq!(kari.iter().map(|a| a.id.as_str()).collect::<Vec<_>>(), vec!["k3"], "one card, the newest");
assert_eq!(own_records(&js, &bucket, "ola").await.len(), 1, "nobody else's card is touched");
assert_eq!(
kari.iter().map(|a| a.id.as_str()).collect::<Vec<_>>(),
vec!["k3"],
"one card, the newest"
);
assert_eq!(
own_records(&js, &bucket, "ola").await.len(),
1,
"nobody else's card is touched"
);
}