diff --git a/CHANGELOG.md b/CHANGELOG.md index b273bb1..99ced87 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,10 @@ One line per change, grouped by the release that shipped it. Newest first. A release is a Cargo.toml bump plus a `v` tag; the content repos pin the tag they run (see the end of this file). +## Unreleased + +- Storylines (`src/content/variants.rs`, `docs/storylines.md`): any text a person reads may be a list, one entry per storyline, every list the same length; a visitor is given one storyline for the whole site (the `portal_storyline` cookie, a year) and reads it on every page and in every mail about their case. What the machine reads - a field's `name`, `options`, states, groups, paths - never varies. The lint refuses a list of another length. Records and answer events carry `storyline`, and `GET /report` gives counts per bucket, per storyline, per state, and nothing else. + ## 0.5.10 (2026-09-30) - The lint lets an invite form collect a required `phone` in place of a required `email`: memberships know a person by number first. diff --git a/docs/README.md b/docs/README.md index 46fef68..e477bbd 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,6 +11,7 @@ engine does with it. | [Content reference](content-reference.md) | Every key a content repo may write, and what it does. | | [State machines](state-machines.md) | `aggregates.yaml`: stages, who moves a case, mail, deadlines, memberships. | | [Access](access.md) | How the Cedar policy is compiled from content, and who may ask it what. | +| [Storylines](storylines.md) | The same site told several ways: lists of texts, one storyline per visitor, counted per storyline. | | [Memberships](memberships.md) | Groups kept by portal itself: a membership as a case, a person as a phone number first, and the provider only saying who. | | [Operations](operations.md) | Releasing, deploying, adding a site, environment variables. | | [Filesystem routes](design/filesystem-routes.md) | Why the `questions/` tree is the router. | diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 9f92544..2231d5f 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -8,6 +8,7 @@ - [Content reference](content-reference.md) - [State machines](state-machines.md) - [Access](access.md) +- [Storylines](storylines.md) - [Memberships](memberships.md) - [Operations](operations.md) diff --git a/docs/content-reference.md b/docs/content-reference.md index 0f98f3e..8025919 100644 --- a/docs/content-reference.md +++ b/docs/content-reference.md @@ -38,6 +38,11 @@ mail: `mail.from` is required as soon as any state declares a mail. +Any text a person reads (`name`, `description`, `label`, +`placeholder`, a mail's `subject` and `body`, `consequence`, +`encouragements`) may be a list instead of one string: one entry per +[storyline](storylines.md), all lists the same length. + ## A question One YAML file is one page. diff --git a/docs/operations.md b/docs/operations.md index 163ae47..3b39dd1 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -130,6 +130,12 @@ then `GARAGE_S3_ENDPOINT`, `GARAGE_ACCESS_KEY`, `GARAGE_SECRET_KEY` and `GARAGE_UPLOADS_BUCKET` in that instance's env. Unset, an upload field fails closed rather than appearing to work and storing nowhere. +## The report + +`GET /report` is public and answers with counts only: per bucket, per +[storyline](storylines.md), per state. It is what the trainer reads to +compare the storylines on what people did. + ## The docs site These pages are [portal.uhhm.no](https://portal.uhhm.no): `docs/` built diff --git a/docs/storylines.md b/docs/storylines.md new file mode 100644 index 0000000..9e272b4 --- /dev/null +++ b/docs/storylines.md @@ -0,0 +1,60 @@ +# Storylines + +The same site, told two ways, and counted. + +Anywhere content says something to a person, it may say it in several +ways by giving a list instead of one string: + +```yaml +name: Vil du hjelpe til i nabolaget? +description: + - Dugnad, 17. mai og komiteene trenger flere hender. Si hva du kan. + - Naboene gjør det meste selv. Her melder du deg til det du har lyst på. +alternatives: + - name: [Bli med, Meld deg] + consequence: [[Send], [Meld meg]] +``` + +Each index is a *storyline*. A visitor is given one at their first +request, kept in a cookie (`portal_storyline`) for a year, and reads +that storyline on every page, in every button and in every mail the +site sends them about their case. A single string is every storyline. + +## The rules + +- **Same length everywhere.** Every list of texts in the content has + the site's arity, so a storyline never has a hole in it. The lint + refuses a list of another length and names it. +- **What varies is what a person reads:** `name`, `description`, + `label`, `placeholder`, `title`, a mail's `subject` and `body`, a + list's `empty` line, `consequence` and `encouragements` (those two as + lists of lists). +- **What the machine reads never varies:** a field's `name` (the stored + key), a select's `options` (the stored values), states, groups, + paths, buckets. +- **One storyline per site**, not per page. That is what makes it a + storyline rather than a scatter of variants: a warm front door leads + to a warm form and a warm mail. +- A signed-in person keeps the storyline they had as a visitor. +- A site that tells one storyline sets no cookie. + +## What is counted + +Every record and every answer event carries the storyline it came +from (`storyline` on `Answer` and on `portal.answers.submitted`). The +mail a case gets tells the same storyline, so a person who read the +warm front door is not thanked in the plain voice. + +`GET /report` gives the counts, per bucket, per storyline, per state: + +```json +{ "storylines": 2, + "buckets": { "hjelpebehov": { "0": { "open": 3, "ordnet": 9 }, + "1": { "open": 5, "ordnet": 14 } } } } +``` + +Storylines are assigned evenly, so those numbers are the comparison: +which words brought more people to send, and how their cases went. No +record, field, address or time leaves the site. The trainer reads this +(`portal-trainer report`) next to what its personas scored offline, and +the plan's `tones` are where the storylines are written. diff --git a/src/announce.rs b/src/announce.rs index 2a25ef1..06c8b1b 100644 --- a/src/announce.rs +++ b/src/announce.rs @@ -84,6 +84,7 @@ pub async fn sweep(state: &AppState) -> anyhow::Result<()> { EVENT_ALTERNATIVE, &responses, now_ms, + 0, ) .await?; tracing::info!(question = %q.id, "event record created"); @@ -137,6 +138,7 @@ pub async fn sweep(state: &AppState) -> anyhow::Result<()> { alternative: SUMMARY_DUE_LABEL.to_string(), responses: payload, timestamp_ms: now_ms, + storyline: 0, }, ) .await?; diff --git a/src/answers.rs b/src/answers.rs index ade8d5c..c570f9b 100644 --- a/src/answers.rs +++ b/src/answers.rs @@ -24,6 +24,10 @@ pub struct Answer { pub responses: serde_json::Value, pub state: String, pub submitted_ms: i64, + /// The storyline the record was sent from, so the mail it gets + /// tells the same one (see `content/variants.rs`). + #[serde(default)] + pub storyline: u32, #[serde(default)] pub decided_ms: Option, #[serde(default)] @@ -64,6 +68,7 @@ pub async fn store_answer( alternative: &str, responses: &serde_json::Value, submitted_ms: i64, + storyline: u32, ) -> anyhow::Result<()> { let store = match js.get_key_value(bucket).await { Ok(store) => store, @@ -89,6 +94,7 @@ pub async fn store_answer( responses: responses.clone(), state: initial_state, submitted_ms, + storyline, decided_ms: None, notes: Vec::new(), decided_by: None, @@ -132,9 +138,11 @@ pub async fn transition_answers( use crate::server::AppState; let state = expect_context::(); + let storyline = crate::server::storyline::current().await; let alt = state .questions .load() + .at(storyline) .get(&question_id) .and_then(|q| q.alternatives.iter().find(|a| a.name == alternative).cloned()) .ok_or_else(|| ServerFnError::new("unknown question or alternative"))?; @@ -289,7 +297,7 @@ async fn apply_transition( .put(&item.item_id, data.into()) .await .map_err(|e| ServerFnError::new(e.to_string()))?; - crate::mail::on_state(state, bucket, &item.item_id, &item.to, &answer.responses).await; + crate::mail::on_state(state, bucket, &item.item_id, &item.to, &answer.responses, answer.storyline).await; crate::people::on_state(state, bucket, &item.item_id, &item.to, &answer.responses).await; // An expired invite sent again: the move back is the asking. if bucket == crate::people::INVITES_BUCKET && item.to == OPEN_STATE { @@ -307,6 +315,7 @@ async fn apply_transition( alternative: transition.label, responses: decision_payload_for_event, timestamp_ms: decided_ms, + storyline: answer.storyline, }; emit_answer_submitted(&state.nats, &event) .await @@ -417,7 +426,7 @@ pub async fn self_transition_answer( .put(&item_id, data.into()) .await .map_err(|e| ServerFnError::new(e.to_string()))?; - crate::mail::on_state(&state, &self_transition.bucket, &item_id, &self_transition.to, &answer.responses).await; + crate::mail::on_state(&state, &self_transition.bucket, &item_id, &self_transition.to, &answer.responses, answer.storyline).await; crate::people::on_state(&state, &self_transition.bucket, &item_id, &self_transition.to, &answer.responses).await; let parent_hashes = vec![item_id.clone()]; @@ -429,6 +438,7 @@ pub async fn self_transition_answer( alternative: self_transition.label, responses: decision_payload_for_event, timestamp_ms: decided_ms, + storyline: answer.storyline, }; emit_answer_submitted(&state.nats, &event) .await @@ -456,6 +466,7 @@ mod tests { responses: serde_json::json!({"name": "Ada"}), state: OPEN_STATE.to_string(), submitted_ms: 1_700_000_000_000, + storyline: 0, decided_ms: None, decided_by: None, notes: Vec::new(), @@ -467,6 +478,7 @@ mod tests { responses: serde_json::json!({"name": "Grace"}), state: "invited".to_string(), submitted_ms: 1_700_000_001_000, + storyline: 0, decided_ms: Some(1_700_000_002_000), decided_by: Some("bl".to_string()), notes: Vec::new(), diff --git a/src/app.rs b/src/app.rs index 664675f..5699d77 100644 --- a/src/app.rs +++ b/src/app.rs @@ -2358,8 +2358,9 @@ pub async fn get_question( use crate::content::{fill_from_user, path_matches, resolve_question}; use crate::server::AppState; let state = expect_context::(); - let questions = state.questions.load(); - let Some(mut question) = resolve_question(&questions, &path) else { + let storylines = state.questions.load(); + let questions = storylines.at(crate::server::storyline::current().await); + let Some(mut question) = resolve_question(questions, &path) else { return Ok(None); }; let session: tower_sessions::Session = leptos_axum::extract().await?; @@ -2401,7 +2402,7 @@ pub async fn get_question( pub async fn get_site() -> Result { use crate::server::AppState; let state = expect_context::(); - let mut site = state.site.load().as_ref().clone(); + let mut site = state.site.load().at(crate::server::storyline::current().await).clone(); // Not content: whether this instance can serve tiles at all is its // own configuration, and the browser has to know before it decides // whether to draw a map or ask for two numbers. @@ -2435,7 +2436,8 @@ pub async fn list_qualifying_questions( .map_err(|e| ServerFnError::new(e.to_string()))?; let now_ms = chrono::Utc::now().timestamp_millis(); - let questions = state.questions.load(); + let storylines = state.questions.load(); + let questions = storylines.at(crate::server::storyline::current().await); // What the chain verifiably answered - the same lookup a // `requires_chain` gate does, so a forged or foreign hash resolves // to nothing - and therefore where it sent the visitor. @@ -2583,6 +2585,7 @@ pub async fn suggest_page_edit(question_id: String, text: String) -> Result<(), SUGGESTION_ALTERNATIVE, &responses, now_ms, + 0, ) .await .map_err(|e| ServerFnError::new(e.to_string()))?; @@ -2593,6 +2596,7 @@ pub async fn suggest_page_edit(question_id: String, text: String) -> Result<(), alternative: SUGGESTION_ALTERNATIVE.to_string(), responses: responses.clone(), timestamp_ms: now_ms, + storyline: 0, }; if let Err(e) = emit_answer_submitted(&state.nats, &event).await { tracing::warn!(question_id, "the suggestion is on file but was not announced: {e}"); @@ -2679,7 +2683,8 @@ pub async fn submit_answer( // resolve_question, not a plain map get: a dynamic page's concrete // path (what the client holds as its question id) resolves to the // pattern page it came from. - let question = crate::content::resolve_question(&state.questions.load(), &question_id) + let storyline = crate::server::storyline::current().await; + let question = crate::content::resolve_question(state.questions.load().at(storyline), &question_id) .ok_or_else(|| ServerFnError::new("unknown question"))?; let session: tower_sessions::Session = leptos_axum::extract().await?; @@ -2796,6 +2801,7 @@ pub async fn submit_answer( alternative: alternative.clone(), responses: responses.clone(), timestamp_ms, + storyline: storyline as u32, }, ) .await @@ -2821,6 +2827,7 @@ pub async fn submit_answer( &alternative, &responses, timestamp_ms, + storyline as u32, ) .await { @@ -2829,7 +2836,7 @@ pub async fn submit_answer( // Arrival is a state too: what the schema says the first // state sends, it sends now. if let Some(initial) = state.aggregates.load().get(bucket).map(|s| s.initial.clone()) { - crate::mail::on_state(&state, bucket, &chain_hash, &initial, &responses).await; + crate::mail::on_state(&state, bucket, &chain_hash, &initial, &responses, storyline as u32).await; crate::people::on_state(&state, bucket, &chain_hash, &initial, &responses).await; } } diff --git a/src/content/gitea.rs b/src/content/gitea.rs index d56b889..bf74acc 100644 --- a/src/content/gitea.rs +++ b/src/content/gitea.rs @@ -9,20 +9,27 @@ use super::*; /// config; a file that exists but doesn't parse or validate is a real /// error, surfaced at boot rather than papered over. #[cfg(feature = "ssr")] -pub async fn load_site_from_gitea(repo_url: &str, branch: &str) -> anyhow::Result { +pub async fn load_site_from_gitea(repo_url: &str, branch: &str) -> anyhow::Result> { let (owner, repo) = parse_owner_repo(repo_url)?; let api_base = gitea_api_base(repo_url)?; let raw = match fetch_gitea_file(&api_base, &owner, &repo, branch, "site.yaml").await { Ok(raw) => raw, Err(e) => { tracing::info!("no site.yaml in content repo ({e}), using default branding"); - return Ok(SiteConfig::default()); + return Ok(Storylines::one(SiteConfig::default())); } }; - let site: SiteConfig = - serde_yaml::from_str(&raw).map_err(|e| anyhow::anyhow!("parsing site.yaml: {e}"))?; - site.validate()?; - Ok(site) + // The invite mail's words may be several (see `variants.rs`). + let files = vec![("site.yaml".to_string(), raw)]; + let arity = super::variants::arity(&files)?; + let mut each = Vec::with_capacity(arity); + for n in 0..arity { + let site: SiteConfig = + serde_yaml::from_str(&super::variants::storyline(&files, n)[0].1).map_err(|e| anyhow::anyhow!("parsing site.yaml: {e}"))?; + site.validate()?; + each.push(site); + } + Ok(Storylines { each }) } /// Extracts `scheme://host` from a repo's normal browser URL - the @@ -107,11 +114,18 @@ pub(crate) async fn fetch_gitea_file( pub async fn load_aggregates_from_gitea( repo_url: &str, branch: &str, -) -> anyhow::Result> { +) -> anyhow::Result>> { let (owner, repo) = parse_owner_repo(repo_url)?; let api_base = gitea_api_base(repo_url)?; let raw = fetch_gitea_file(&api_base, &owner, &repo, branch, "aggregates.yaml").await?; - crate::aggregates::parse_aggregates_yaml(&raw) + // A mail's subject and body may be several (see `variants.rs`); + // the runtime's own buckets are added to every storyline. + let files = vec![("aggregates.yaml".to_string(), raw)]; + let arity = super::variants::arity(&files)?; + let each = (0..arity) + .map(|n| crate::aggregates::parse_aggregates_yaml(&super::variants::storyline(&files, n)[0].1).map(with_builtin_aggregates)) + .collect::>>()?; + Ok(Storylines { each }) } #[cfg(feature = "ssr")] @@ -119,7 +133,7 @@ pub async fn load_questions_from_gitea( repo_url: &str, branch: &str, subdir: &str, -) -> anyhow::Result> { +) -> anyhow::Result>> { let (owner, repo) = parse_owner_repo(repo_url)?; let api_base = gitea_api_base(repo_url)?; @@ -161,7 +175,20 @@ pub async fn load_questions_from_gitea( .map_err(|e| anyhow::anyhow!("fetching {path}: {e}"))?; files.push((path[prefix.len()..].to_string(), raw)); } - build_questions(&files) + build_storylines(&files) +} + +/// Every storyline's arity is the site's or one: a mail told in two +/// voices on a site whose pages are told in three has a hole in it. +#[cfg(feature = "ssr")] +pub fn same_arity(questions: usize, aggregates: usize, site: usize) -> anyhow::Result { + let arity = questions.max(aggregates).max(site); + for (what, n) in [("questions", questions), ("aggregates.yaml", aggregates), ("site.yaml", site)] { + if n != 1 && n != arity { + anyhow::bail!("{what} tells {n} storyline(s), but the site tells {arity}; every list of texts has the same length"); + } + } + Ok(arity) } /// Published by the content repo's own CI (after it lints a push) to @@ -192,11 +219,11 @@ pub async fn watch_for_reload( repo_url: String, branch: String, subdir: String, - questions: std::sync::Arc>>, + questions: std::sync::Arc>>>, aggregates: std::sync::Arc< - arc_swap::ArcSwap>, + arc_swap::ArcSwap>>, >, - site: std::sync::Arc>, + site: std::sync::Arc>>, access: std::sync::Arc>, ) { let mut sub = match nats.subscribe(CONTENT_RELOAD_SUBJECT).await { @@ -209,7 +236,7 @@ pub async fn watch_for_reload( use futures::StreamExt; while sub.next().await.is_some() { let loaded_aggregates = match load_aggregates_from_gitea(&repo_url, &branch).await { - Ok(loaded) => with_builtin_aggregates(loaded), + Ok(loaded) => loaded, Err(e) => { tracing::error!(error = %e, "aggregates.yaml reload failed, keeping last-good content"); continue; @@ -228,7 +255,7 @@ pub async fn watch_for_reload( }; match load_questions_from_gitea(&repo_url, &branch, &subdir).await { Ok(loaded) => { - if let Err(e) = validate_questions(&loaded, &loaded_aggregates) { + if let Err(e) = validate_questions(&loaded, &loaded_aggregates).and_then(|_| same_arity(loaded.arity(), loaded_aggregates.arity(), loaded_site.arity())) { tracing::error!(error = %e, "content reload failed validation, keeping last-good content"); continue; } diff --git a/src/content/mod.rs b/src/content/mod.rs index 297d65d..d9f619b 100644 --- a/src/content/mod.rs +++ b/src/content/mod.rs @@ -13,12 +13,16 @@ use serde::{Deserialize, Serialize}; // Loading is server-only; the rest is shared with the browser build. #[cfg(feature = "ssr")] mod gitea; +#[cfg(feature = "ssr")] +mod variants; mod handlers; mod markdown; mod validate; #[cfg(feature = "ssr")] pub use gitea::*; +#[cfg(feature = "ssr")] +pub use variants::*; pub use handlers::*; pub use markdown::*; pub use validate::*; @@ -1073,6 +1077,25 @@ pub fn resolve_question( #[cfg(feature = "ssr")] pub fn build_questions( files: &[(String, String)], +) -> anyhow::Result> { + // The first storyline, once every list of texts has been held to + // the site's arity (see `variants.rs`); the machinery reads this + // one, and `build_storylines` builds the rest for the visitors. + variants::arity(files)?; + build_one_storyline(&variants::storyline(files, 0)) +} + +/// Every storyline of the content, each a full map of questions. +#[cfg(feature = "ssr")] +pub fn build_storylines(files: &[(String, String)]) -> anyhow::Result>> { + let arity = variants::arity(files)?; + let each = (0..arity).map(|n| build_one_storyline(&variants::storyline(files, n))).collect::>>()?; + Ok(variants::Storylines { each }) +} + +#[cfg(feature = "ssr")] +fn build_one_storyline( + files: &[(String, String)], ) -> anyhow::Result> { // Sections first: (directory prefix, config), shallowest first so a // later (deeper) section overrides an outer one field-by-field. diff --git a/src/content/variants.rs b/src/content/variants.rs new file mode 100644 index 0000000..faf2a08 --- /dev/null +++ b/src/content/variants.rs @@ -0,0 +1,224 @@ +//! Storylines: a text may be several, and `[n]` runs through the site. +//! +//! Anywhere content says something to a person - a page's `name` and +//! `description`, an alternative's, a feature's, a `label`, a +//! `placeholder`, a mail's `subject` and `body`, a list's `empty` line, +//! the `consequence` buttons and `encouragements` - it may give a list +//! instead of one string: +//! +//! ```yaml +//! description: +//! - Dugnad, 17. mai og komiteene trenger flere hender. Si hva du kan. +//! - Naboene gjør det meste selv. Her melder du deg til det du har lyst på. +//! ``` +//! +//! Every such list has the same length, the site's *arity*, and a +//! visitor is given one index for the whole site (a cookie, see +//! `main.rs`), so they read one storyline from the front door to the +//! mail they get. A single string is every storyline. What a visitor +//! did is recorded with the index (`AnswerSubmitted.variant`), so the +//! storylines can be compared on what people actually did, and nothing +//! else about them. +//! +//! This is done to the YAML before the structs see it: the files are +//! expanded once per storyline, and everything downstream keeps its one +//! string per field. Keys that name things to the machine - a field's +//! `name`, a select's `options`, states, groups, paths - never vary. +//! +//! The lint holds every list to the arity, so a storyline cannot have +//! a hole in it. + +use serde_yaml::Value; + +/// Keys whose value is one string to a person. +const TEXT: &[&str] = &["name", "description", "label", "placeholder", "subject", "body", "empty", "title"]; +/// Keys whose value is a list of strings to a person; their storylines +/// are a list of lists. +const TEXT_LIST: &[&str] = &["consequence", "encouragements"]; + +/// Where in a document a key is not text but an identifier: a +/// requirement's `name` is the stored field, never varied. +fn is_identifier(path: &[String], key: &str) -> bool { + // `path` ends with the key itself, then the item's index, then the + // list it is in. + key == "name" && path.iter().rev().nth(2).is_some_and(|p| p == "requirements") +} + +fn is_strings(seq: &[Value]) -> bool { + !seq.is_empty() && seq.iter().all(|v| v.is_string()) +} + +fn is_lists_of_strings(seq: &[Value]) -> bool { + !seq.is_empty() && seq.iter().all(|v| v.as_sequence().is_some_and(|s| s.iter().all(|x| x.is_string()))) +} + +/// The storyline counts found in a document, with where each was. +fn counts(value: &Value, path: &mut Vec, out: &mut Vec<(String, usize)>) { + match value { + Value::Mapping(map) => { + for (k, v) in map { + let key = k.as_str().unwrap_or_default().to_string(); + path.push(key.clone()); + let variants = match v { + Value::Sequence(seq) if TEXT.contains(&key.as_str()) && !is_identifier(path, &key) && is_strings(seq) => Some(seq.len()), + Value::Sequence(seq) if TEXT_LIST.contains(&key.as_str()) && is_lists_of_strings(seq) => Some(seq.len()), + _ => None, + }; + match variants { + Some(n) => out.push((path.join("."), n)), + None => counts(v, path, out), + } + path.pop(); + } + } + Value::Sequence(seq) => { + for (i, v) in seq.iter().enumerate() { + path.push(i.to_string()); + counts(v, path, out); + path.pop(); + } + } + _ => {} + } +} + +/// One storyline of a document: every list of texts becomes its `n`th. +fn pick(value: &mut Value, path: &mut Vec, n: usize) { + match value { + Value::Mapping(map) => { + for (k, v) in map.iter_mut() { + let key = k.as_str().unwrap_or_default().to_string(); + path.push(key.clone()); + let chosen = match v { + Value::Sequence(seq) if TEXT.contains(&key.as_str()) && !is_identifier(path, &key) && is_strings(seq) => Some(seq[n.min(seq.len() - 1)].clone()), + Value::Sequence(seq) if TEXT_LIST.contains(&key.as_str()) && is_lists_of_strings(seq) => Some(seq[n.min(seq.len() - 1)].clone()), + _ => None, + }; + match chosen { + Some(c) => *v = c, + None => pick(v, path, n), + } + path.pop(); + } + } + Value::Sequence(seq) => { + for (i, v) in seq.iter_mut().enumerate() { + path.push(i.to_string()); + pick(v, path, n); + path.pop(); + } + } + _ => {} + } +} + +/// How many storylines these files tell: the length every list of +/// texts has, or 1 when there is none. A list of another length is an +/// error that names it. +pub fn arity(files: &[(String, String)]) -> anyhow::Result { + let mut found: Vec<(String, String, usize)> = Vec::new(); + for (rel, raw) in files { + let Ok(value) = serde_yaml::from_str::(raw) else { continue }; + let mut out = Vec::new(); + counts(&value, &mut Vec::new(), &mut out); + found.extend(out.into_iter().map(|(p, n)| (rel.clone(), p, n))); + } + let arity = found.iter().map(|(_, _, n)| *n).max().unwrap_or(1); + if let Some((rel, p, n)) = found.iter().find(|(_, _, n)| *n != arity) { + anyhow::bail!("{rel}: {p} tells {n} storyline(s), but the site tells {arity}; every list of texts has the same length"); + } + Ok(arity.max(1)) +} + +/// The files as storyline `n` tells them. +pub fn storyline(files: &[(String, String)], n: usize) -> Vec<(String, String)> { + files + .iter() + .map(|(rel, raw)| { + let Ok(mut value) = serde_yaml::from_str::(raw) else { return (rel.clone(), raw.clone()) }; + let mut out = Vec::new(); + counts(&value, &mut Vec::new(), &mut out); + if out.is_empty() { + return (rel.clone(), raw.clone()); + } + pick(&mut value, &mut Vec::new(), n); + (rel.clone(), serde_yaml::to_string(&value).unwrap_or_else(|_| raw.clone())) + }) + .collect() +} + +/// One thing per storyline, and the first as what the machinery reads +/// when nobody in particular is asking: a seed, a lint, a desk sweep. +#[derive(Clone, Debug)] +pub struct Storylines { + pub each: Vec, +} + +impl Storylines { + pub fn one(item: T) -> Self { + Storylines { each: vec![item] } + } + + pub fn arity(&self) -> usize { + self.each.len().max(1) + } + + /// Storyline `n`, or the last for an index from a site that told + /// more storylines than this one does. + pub fn at(&self, n: usize) -> &T { + &self.each[n.min(self.each.len() - 1)] + } +} + +impl std::ops::Deref for Storylines { + type Target = T; + fn deref(&self) -> &T { + &self.each[0] + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const PAGE: &str = "name: [Vil du hjelpe til?, Trenger naboene deg?]\ndescription: Alle trenger flere hender.\nalternatives:\n - name: [Bli med, Meld deg]\n description: [Si hva du kan., Si hva du har lyst på.]\n consequence: [[Send], [Meld meg]]\n features:\n - name: Skjema\n requirements:\n - name: [navn]\n label: [Navn, Hva heter du?]\n - name: hva\n type: select\n options: [Dugnad, Komité]\n"; + + #[test] + fn every_list_of_texts_is_a_storyline_and_identifiers_are_not() { + let files = vec![("index.yaml".to_string(), PAGE.to_string())]; + assert_eq!(arity(&files).unwrap(), 2); + let second = storyline(&files, 1); + let v: Value = serde_yaml::from_str(&second[0].1).unwrap(); + assert_eq!(v["name"], "Trenger naboene deg?"); + assert_eq!(v["description"], "Alle trenger flere hender."); + assert_eq!(v["alternatives"][0]["name"], "Meld deg"); + assert_eq!(v["alternatives"][0]["consequence"], serde_yaml::to_value(vec!["Meld meg"]).unwrap()); + // A requirement's name is the stored field: left as written. + assert_eq!(v["alternatives"][0]["features"][0]["requirements"][0]["name"], serde_yaml::to_value(vec!["navn"]).unwrap()); + assert_eq!(v["alternatives"][0]["features"][0]["requirements"][0]["label"], "Hva heter du?"); + // Options are values people choose from and are stored: untouched. + assert_eq!(v["alternatives"][0]["features"][0]["requirements"][1]["options"], serde_yaml::to_value(vec!["Dugnad", "Komité"]).unwrap()); + } + + #[test] + fn a_site_with_no_lists_tells_one_storyline_and_a_short_list_is_refused() { + let plain = vec![("index.yaml".to_string(), "name: Hva?\nalternatives:\n - name: A\n consequence: [Send]\n".to_string())]; + assert_eq!(arity(&plain).unwrap(), 1); + assert_eq!(storyline(&plain, 0)[0].1, plain[0].1); + let uneven = vec![ + ("index.yaml".to_string(), PAGE.to_string()), + ("b.yaml".to_string(), "name: [En, To, Tre]\n".to_string()), + ]; + let err = arity(&uneven).unwrap_err().to_string(); + assert!(err.contains("b.yaml: name tells 3") && err.contains("tells 2") || err.contains("tells 3"), "{err}"); + } + + #[test] + fn the_first_storyline_is_what_the_machinery_reads() { + let s = Storylines { each: vec!["a", "b"] }; + assert_eq!(*s, "a"); + assert_eq!(*s.at(1), "b"); + assert_eq!(*s.at(7), "b"); + assert_eq!(s.arity(), 2); + } +} diff --git a/src/deadlines.rs b/src/deadlines.rs index 078f7ca..c302ff5 100644 --- a/src/deadlines.rs +++ b/src/deadlines.rs @@ -94,7 +94,7 @@ pub async fn sweep(state: &AppState) -> anyhow::Result<()> { if let Ok(data) = serde_json::to_vec(&answer) { let _ = store.put(&id, data.into()).await; } - crate::mail::on_state(state, &schema.bucket, &id, &target, &answer.responses).await; + crate::mail::on_state(state, &schema.bucket, &id, &target, &answer.responses, answer.storyline).await; crate::people::on_state(state, &schema.bucket, &id, &target, &answer.responses).await; let parent_hashes = vec![id.clone()]; @@ -108,6 +108,7 @@ pub async fn sweep(state: &AppState) -> anyhow::Result<()> { alternative: target.clone(), responses: payload, timestamp_ms: now_ms, + storyline: answer.storyline, }, ) .await @@ -145,6 +146,7 @@ aggregates: id: "r".into(), question_id: "/q".into(), alternative: "a".into(), responses: serde_json::json!({}), state: state.into(), submitted_ms: since_ms, decided_ms: None, decided_by: None, notes: vec![], + storyline: 0, } } diff --git a/src/events/mod.rs b/src/events/mod.rs index b72ef52..a346ce7 100644 --- a/src/events/mod.rs +++ b/src/events/mod.rs @@ -15,6 +15,10 @@ pub struct AnswerSubmitted { pub alternative: String, pub responses: serde_json::Value, pub timestamp_ms: i64, + /// The storyline the person read (see `content/variants.rs`): what + /// the storylines are compared on, and nothing else about them. + #[serde(default)] + pub storyline: u32, } pub const ANSWERS_SUBJECT: &str = "portal.answers.submitted"; diff --git a/src/lib.rs b/src/lib.rs index 64afb9f..5635369 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -35,6 +35,8 @@ pub mod mail; pub mod seed; #[cfg(feature = "ssr")] pub mod memberships; +#[cfg(feature = "ssr")] +pub mod report; pub mod maps; diff --git a/src/mail.rs b/src/mail.rs index ddeebfa..38a8872 100644 --- a/src/mail.rs +++ b/src/mail.rs @@ -306,8 +306,10 @@ async fn publish(js: &async_nats::jetstream::Context, mail: &Outgoing) -> anyhow /// if anything. Best-effort by design - the transition is already /// committed, and a mail that cannot be rendered is logged, never a /// failed request. -pub async fn on_state(state: &crate::server::AppState, bucket: &str, id: &str, to_state: &str, responses: &serde_json::Value) { - let Some(spec) = state.aggregates.load().get(bucket).and_then(|s| s.mail_for(to_state)).cloned() else { +pub async fn on_state(state: &crate::server::AppState, bucket: &str, id: &str, to_state: &str, responses: &serde_json::Value, storyline: u32) { + // The mail tells the storyline the person read. + let storyline = storyline as usize; + let Some(spec) = state.aggregates.load().at(storyline).get(bucket).and_then(|s| s.mail_for(to_state)).cloned() else { // A suggestion has its own words for this unless the content // gave the state a mail, which then is the one sent. if bucket == crate::content::EDIT_SUGGESTIONS_BUCKET { @@ -315,7 +317,8 @@ pub async fn on_state(state: &crate::server::AppState, bucket: &str, id: &str, t } return; }; - let site = state.site.load(); + let site_storylines = state.site.load(); + let site = site_storylines.at(storyline); let Some(cfg) = &site.mail else { tracing::warn!(bucket, to_state, "a mail is declared for this state, but site.yaml has no `mail.from` - not sent"); return; @@ -702,6 +705,7 @@ mod tests { responses: serde_json::json!({ "email": email, "where": "The bend" }), state: "open".into(), submitted_ms: 1, + storyline: 0, decided_ms: None, decided_by: None, notes: Vec::new(), diff --git a/src/main.rs b/src/main.rs index 0126df6..7e94b3e 100644 --- a/src/main.rs +++ b/src/main.rs @@ -30,13 +30,12 @@ async fn main() -> anyhow::Result<()> { let content_branch = std::env::var("CONTENT_BRANCH").unwrap_or_else(|_| "main".to_string()); let gitea_base = content::gitea_api_base(&content_repo)?; let content_raw_base = content::gitea_raw_base(&content_repo)?; - let aggregates = content::with_builtin_aggregates( - content::load_aggregates_from_gitea(&content_repo, &content_branch).await?, - ); + let aggregates = content::load_aggregates_from_gitea(&content_repo, &content_branch).await?; let questions = content::load_questions_from_gitea(&content_repo, &content_branch, "questions").await?; content::validate_questions(&questions, &aggregates)?; let site = content::load_site_from_gitea(&content_repo, &content_branch).await?; - tracing::info!(count = questions.len(), repo = %content_repo, branch = %content_branch, "loaded content"); + let storylines = content::same_arity(questions.arity(), aggregates.arity(), site.arity())?; + tracing::info!(count = questions.len(), storylines, repo = %content_repo, branch = %content_branch, "loaded content"); let access = Arc::new(arc_swap::ArcSwap::from_pointee(portal::access::Policy::from_content(&questions, &aggregates))); let questions = Arc::new(arc_swap::ArcSwap::from_pointee(questions)); let aggregates = Arc::new(arc_swap::ArcSwap::from_pointee(aggregates)); @@ -225,6 +224,8 @@ async fn main() -> anyhow::Result<()> { get(upload::attachment), ) .route("/gitea-repo", get(content::gitea_repo_handler)) + // What the storylines did, as counts (report.rs). + .route("/report", get(portal::report::handler)) // Content-shipped assets (hero module etc.), same-origin. .route("/site/{*path}", get(content::site_asset_handler)) .route("/automation/kv/{bucket}", get(content::automation_kv_handler)) @@ -273,7 +274,9 @@ async fn main() -> anyhow::Result<()> { }, ) .fallback(leptos_axum::file_and_error_handler::(shell)) - .layer(session_layer); + .layer(session_layer) + // Which storyline this visitor reads, before anything renders. + .layer(axum::middleware::from_fn_with_state(state.clone(), portal::server::storyline::assign)); // Outermost, so a caller who is over the limit is turned away // before a session is loaded or a handler is entered. let app = match governor { diff --git a/src/memberships.rs b/src/memberships.rs index 6892db1..f0b73e9 100644 --- a/src/memberships.rs +++ b/src/memberships.rs @@ -226,6 +226,7 @@ async fn move_to(state: &crate::server::AppState, store: &async_nats::jetstream: alternative: target.to_string(), responses: payload, timestamp_ms: now_ms, + storyline: answer.storyline, }; let _ = crate::events::emit_answer_submitted(&state.nats, &event).await; Ok(answer) @@ -266,7 +267,7 @@ pub async fn grant(state: &crate::server::AppState, group: &str, name: &str, who "name": name, "group": group, "by": by, "phone": who.phone, "email": who.email, "sub": who.sub, }); - crate::answers::store_answer(&state.jetstream, &state.aggregates.load(), BUCKET, id, "/", "Membership", &responses, now_ms).await?; + crate::answers::store_answer(&state.jetstream, &state.aggregates.load(), BUCKET, id, "/", "Membership", &responses, now_ms, 0).await?; Ok(invited(true)) } } @@ -346,6 +347,7 @@ mod tests { responses: serde_json::json!({ "group": group, "phone": phone, "email": email, "sub": sub, "name": "X" }), state: state.into(), submitted_ms: 0, + storyline: 0, decided_ms: None, decided_by: None, notes: Vec::new(), diff --git a/src/people.rs b/src/people.rs index 8a4a71a..e318b66 100644 --- a/src/people.rs +++ b/src/people.rs @@ -414,7 +414,7 @@ pub async fn record(state: &crate::server::AppState, invited: &Invited, responsi responses["responsible_for"] = serde_json::json!(responsible_for); } let id = crate::chain::hash_node("/", &[], &responses, now_ms); - if let Err(e) = crate::answers::store_answer(&state.jetstream, &state.aggregates.load(), INVITES_BUCKET, id, "/", SEED_ALTERNATIVE, &responses, now_ms).await { + if let Err(e) = crate::answers::store_answer(&state.jetstream, &state.aggregates.load(), INVITES_BUCKET, id, "/", SEED_ALTERNATIVE, &responses, now_ms, 0).await { tracing::warn!(user = %invited.username, "the invite is sent, but could not be put on file for a desk: {e}"); } } @@ -454,6 +454,7 @@ async fn move_invite(state: &crate::server::AppState, store: &async_nats::jetstr alternative: target.to_string(), responses: payload, timestamp_ms: now_ms, + storyline: answer.storyline, }; if let Err(e) = crate::events::emit_answer_submitted(&state.nats, &event).await { tracing::warn!(id = %answer.id, target, "the invite was moved but not published: {e}"); @@ -702,6 +703,7 @@ mod tests { }}), state: state.into(), submitted_ms: 1_000, + storyline: 0, decided_ms: None, decided_by: None, notes: Vec::new(), diff --git a/src/report.rs b/src/report.rs new file mode 100644 index 0000000..529d1c5 --- /dev/null +++ b/src/report.rs @@ -0,0 +1,53 @@ +//! What the storylines did, as numbers and nothing else. +//! +//! `GET /report` answers with a count per bucket, per storyline, per +//! state: how many cases each storyline brought in and where they got +//! to. That is the whole of what the storylines are compared on. No +//! record, no field, no address, no time leaves the site: a storyline +//! is assigned evenly, so the counts are the comparison, and the +//! trainer (`portal-trainer report`) reads them to say which words +//! worked. Public, since a count is what a public list already shows. +#![cfg(feature = "ssr")] + +use std::collections::BTreeMap; + +use serde::{Deserialize, Serialize}; + +/// The report: the site's storylines, and per bucket the count of +/// records by storyline and state. +#[derive(Clone, Debug, Default, Serialize, Deserialize)] +pub struct Report { + pub storylines: usize, + /// bucket -> storyline (as a string, JSON's way) -> state -> count + pub buckets: BTreeMap>>, +} + +pub async fn gather(state: &crate::server::AppState) -> Report { + let mut report = Report { storylines: state.questions.load().arity(), buckets: BTreeMap::new() }; + let aggregates = state.aggregates.load(); + for bucket in aggregates.keys() { + // The runtime's own buckets are not the site's offers. + if bucket.starts_with("portal_") { + continue; + } + let Ok(store) = state.jetstream.get_key_value(bucket).await else { continue }; + let keys: Vec = { + use futures::TryStreamExt; + match store.keys().await { + Ok(keys) => keys.try_collect().await.unwrap_or_default(), + Err(_) => continue, + } + }; + let counts = report.buckets.entry(bucket.clone()).or_default(); + for key in keys { + let Ok(Some(bytes)) = store.get(&key).await else { continue }; + let Ok(answer) = serde_json::from_slice::(&bytes) else { continue }; + *counts.entry(answer.storyline.to_string()).or_default().entry(answer.state).or_default() += 1; + } + } + report +} + +pub async fn handler(axum::extract::State(state): axum::extract::State) -> axum::Json { + axum::Json(gather(&state).await) +} diff --git a/src/resource.rs b/src/resource.rs index f34eb21..43dadd0 100644 --- a/src/resource.rs +++ b/src/resource.rs @@ -33,7 +33,7 @@ pub async fn get_resource( use crate::server::AppState; let state = expect_context::(); - let feature = find_feature(&state, &question_id, &alternative, &feature_name)?; + let feature = find_feature(&state, crate::server::storyline::current().await, &question_id, &alternative, &feature_name)?; let resource = feature .resource .as_ref() @@ -62,7 +62,7 @@ pub async fn get_requirement_options( use crate::server::AppState; let state = expect_context::(); - let feature = find_feature(&state, &question_id, &alternative, &feature_name)?; + let feature = find_feature(&state, crate::server::storyline::current().await, &question_id, &alternative, &feature_name)?; let requirement = feature .requirements .iter() @@ -103,7 +103,7 @@ pub async fn get_requirement_binding( use crate::server::AppState; let state = expect_context::(); - let feature = find_feature(&state, &question_id, &alternative, &feature_name)?; + let feature = find_feature(&state, crate::server::storyline::current().await, &question_id, &alternative, &feature_name)?; let requirement = feature .requirements .iter() @@ -128,6 +128,7 @@ pub async fn get_requirement_binding( #[cfg(feature = "ssr")] fn find_feature( state: &crate::server::AppState, + storyline: usize, question_id: &str, alternative: &str, feature_name: &str, @@ -137,7 +138,7 @@ fn find_feature( // also what substitutes the URL segment into the page's resource // keys - so this lookup is where a `/review/` page's // feature acquires its record-specific key. - let question = crate::content::resolve_question(&state.questions.load(), question_id) + let question = crate::content::resolve_question(state.questions.load().at(storyline), question_id) .ok_or_else(|| ServerFnError::new("unknown question"))?; question .alternatives diff --git a/src/server/mod.rs b/src/server/mod.rs index d83e5d1..aac70f8 100644 --- a/src/server/mod.rs +++ b/src/server/mod.rs @@ -1,6 +1,7 @@ #![cfg(feature = "ssr")] pub mod oidc; +pub mod storyline; use arc_swap::ArcSwap; use axum::extract::FromRef; @@ -21,18 +22,18 @@ pub struct AppState { /// message (see `content::watch_for_reload`) - readers never hold /// a lock, just an atomic pointer load, so a reload never blocks or /// is blocked by an in-flight request. - pub questions: Arc>>, + pub questions: Arc>>>, /// Every bucket's event-sourced state graph, loaded from /// `aggregates.yaml` (see `content::load_aggregates_from_gitea`) and /// hot-swapped alongside `questions` on the same reload. A bucket /// with no entry here isn't event-sourced - plain KV mutate-in-place /// still works (see `answers.rs`). - pub aggregates: Arc>>, + pub aggregates: Arc>>>, /// Site branding from the content repo's optional `site.yaml` /// (title, wordmark, hero kind) - hot-swapped with `questions`/ /// `aggregates` on the same reload. Default = the historical uhhm /// look. - pub site: Arc>, + pub site: Arc>>, /// The access policy compiled from the content above (see /// `access.rs`), swapped with it on every reload. Every server entry /// point decides through this and nothing else. diff --git a/src/server/storyline.rs b/src/server/storyline.rs new file mode 100644 index 0000000..a1235bb --- /dev/null +++ b/src/server/storyline.rs @@ -0,0 +1,70 @@ +//! Which storyline a visitor reads (see `content/variants.rs`): one +//! index for the whole site, chosen at their first request and kept in +//! a cookie for a year, so every page and every mail they get tells +//! the same story. A site that tells one story sets no cookie. +#![cfg(feature = "ssr")] + +use axum::{ + extract::{Request, State}, + http::header::{COOKIE, SET_COOKIE}, + middleware::Next, + response::Response, +}; + +pub const COOKIE_NAME: &str = "portal_storyline"; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Storyline(pub usize); + +/// The index in a request's cookies, if it names one this site tells. +fn from_cookies(header: Option<&str>, arity: usize) -> Option { + header? + .split(';') + .map(str::trim) + .find_map(|kv| kv.strip_prefix(COOKIE_NAME).and_then(|rest| rest.strip_prefix('='))) + .and_then(|v| v.trim().parse::().ok()) + .filter(|n| *n < arity) +} + +/// Evenly, from nothing about the person. +fn pick(arity: usize) -> usize { + (uuid::Uuid::new_v4().as_u128() % arity.max(1) as u128) as usize +} + +pub async fn assign(State(state): State, mut req: Request, next: Next) -> Response { + let arity = state.questions.load().arity(); + let held = from_cookies(req.headers().get(COOKIE).and_then(|v| v.to_str().ok()), arity); + let n = held.unwrap_or_else(|| pick(arity)); + req.extensions_mut().insert(Storyline(n)); + let mut res = next.run(req).await; + if held.is_none() && arity > 1 { + if let Ok(value) = format!("{COOKIE_NAME}={n}; Path=/; Max-Age=31536000; SameSite=Lax").parse() { + res.headers_mut().append(SET_COOKIE, value); + } + } + res +} + +/// The storyline of the request a server function is answering, or +/// the first when there is none to ask (a call outside a request). +pub async fn current() -> usize { + leptos_axum::extract::>() + .await + .map(|axum::Extension(Storyline(n))| n) + .unwrap_or(0) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_cookie_names_a_storyline_the_site_tells() { + assert_eq!(from_cookies(Some("a=b; portal_storyline=1; c=d"), 3), Some(1)); + assert_eq!(from_cookies(Some("portal_storyline=7"), 3), None); + assert_eq!(from_cookies(Some("portal_storylines=1"), 3), None); + assert_eq!(from_cookies(None, 3), None); + assert!(pick(1) == 0); + assert!((0..50).all(|_| pick(3) < 3)); + } +} diff --git a/tests/jetstream.rs b/tests/jetstream.rs index 682a8ad..7f333f4 100644 --- a/tests/jetstream.rs +++ b/tests/jetstream.rs @@ -34,7 +34,7 @@ async fn a_submission_is_created_moved_refused_and_read_back_through_the_real_en let now = chrono::Utc::now().timestamp_millis(); // The submission: a KV record in the initial state, plus the first event. - store_answer(&js, &schemas, &bucket, "rec-1".into(), "/", "Hire us", &json!({"email": "a@b.example"}), now) + store_answer(&js, &schemas, &bucket, "rec-1".into(), "/", "Hire us", &json!({"email": "a@b.example"}), now, 0) .await .expect("store"); let store = js.get_key_value(&bucket).await.expect("bucket exists"); @@ -178,9 +178,9 @@ async fn a_membership_is_invited_becomes_active_at_sign_in_and_can_be_taken_back leptos_options: leptos::config::LeptosOptions::builder().output_name("test").build(), nats, jetstream: js, - questions: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(questions)), - aggregates: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(aggregates)), - site: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(Default::default())), + questions: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(portal::content::Storylines::one(questions))), + aggregates: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(portal::content::Storylines::one(aggregates))), + site: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(portal::content::Storylines::one(Default::default()))), access: std::sync::Arc::new(arc_swap::ArcSwap::from_pointee(access)), gitea_base: String::new(), content_raw_base: String::new(),