Event-sourced applicant/subscriber/project aggregates, generalized resources
Deploy / deploy (push) Failing after 3s

Replaces the free-string, direct-KV-mutate state model in answers.rs
with a proper event log (events/store.rs, JetStream-backed, CAS via
expected_last_subject_sequence) and three pure state machines
(aggregates/{applicant,subscriber,project}.rs). Closes a real
lost-update race in the old transition_answer (concurrent decisions on
the same item could both win, publishing contradictory events). KV
buckets become best-effort read-model projections, not the source of
truth. Content-declared transition targets are now validated at
load/reload time against the real compiled transition tables, not
accepted as arbitrary strings.

Buckets renamed to describe their content, not their relation to the
app (portal_applicants -> applicants, etc); "inquiry" folded into a
richer "project" concept.

ResourceSpec generalized beyond a single KV bucket: Kv | GiteaStarred |
GiteaOrgRepos | Url sources, with an optional jq filter (via the jaq
crate) to shape live data for the frontend. Url source is SSRF-guarded
(https-only, rejects loopback/private/link-local, real DNS resolve).

New headless question_lint binary (validates content against compiled
transition tables with no NATS/OIDC/server involved) and a one-time
backfill_events binary (dry-run by default) for migrating existing KV
data onto the new event log.

Questions get an optional `responsible` contact plus a lightweight
"report this question" action.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Bendik Aagaard Lynghaug
2026-08-06 08:52:29 +02:00
co-authored by Claude Sonnet 5
parent 213b1130bb
commit ebf4bf91b3
16 changed files with 1811 additions and 67 deletions
+40
View File
@@ -0,0 +1,40 @@
//! Collaborator applicants (`/review`'s "Applicants" alternative,
//! `record_as: applicants`). `Open -> Invited | Declined`, straight
//! from the alternatives already declared in `questions/review.yaml`.
use super::AggregateKind;
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
pub enum State {
Open,
Invited,
Declined,
}
impl AggregateKind for State {
const AGGREGATE_TYPE: &'static str = "applicant";
const INITIAL_STATE: Self = State::Open;
fn from_event_type(event_type: &str) -> Option<Self> {
match event_type {
"applied" => Some(State::Open),
"invited" => Some(State::Invited),
"declined" => Some(State::Declined),
_ => None,
}
}
fn event_type(self) -> &'static str {
match self {
State::Open => "applied",
State::Invited => "invited",
State::Declined => "declined",
}
}
fn allowed(self) -> &'static [Self] {
match self {
State::Open => &[State::Invited, State::Declined],
State::Invited | State::Declined => &[],
}
}
}
+294
View File
@@ -0,0 +1,294 @@
//! Generic pure-replay + CAS-write machinery shared by every aggregate
//! type (`applicant`, `subscriber`, `project`) - one small `AggregateKind`
//! impl per type supplies its own states/transition table/event-type
//! names, this module supplies the mechanics once. Mirrors dodrenett's
//! `lib/machines/*.ts` shape (a `Record<State, State[]>` table + a pure
//! `replay()`), but generalized across types instead of copy-pasted per
//! type, and with real optimistic concurrency on write (dodrenett had
//! none - see `crate::events::store`).
#![cfg(feature = "ssr")]
pub mod applicant;
pub mod project;
pub mod subscriber;
use crate::events::store::{append_event, load_events, Event};
/// A state machine's shape: its states, its transition table, and how
/// states map to/from the event-log's `event_type` strings. Kept as a
/// trait (not a generic enum param alone) so each aggregate module can
/// own its own `State` enum and transition table as plain, readable
/// Rust, not a shared data structure everyone has to thread through.
pub trait AggregateKind: Copy + Clone + PartialEq + Eq + std::fmt::Debug + 'static {
/// The bucket-name -> aggregate-type key used both for `events.rs`'s
/// subject scheme and for `content.rs`'s load-time transition
/// validation (see `content::aggregate_type_for_bucket`).
const AGGREGATE_TYPE: &'static str;
/// The state a brand-new aggregate starts in - also the state a
/// creation event (see `create`) asserts.
const INITIAL_STATE: Self;
/// Maps an event-log `event_type` string to the state it represents
/// - `None` for anything this aggregate type doesn't recognize
/// (replay skips it, never errors).
fn from_event_type(event_type: &str) -> Option<Self>;
/// The event-log `event_type` string that represents transitioning
/// *into* this state.
fn event_type(self) -> &'static str;
/// States reachable directly from this one. Checked at write time
/// (`transition`); replay is deliberately more lenient (see below).
fn allowed(self) -> &'static [Self];
}
/// One aggregate's current view, replayed from its event history.
#[derive(Clone, Debug)]
pub struct Aggregate<K> {
pub id: String,
pub state: K,
pub payload: serde_json::Value,
pub sequence: u64,
}
/// Pure, no IO, never panics. An event that doesn't fit the transition
/// table from wherever replay has gotten to so far is silently skipped
/// - not an error, not even the *first* recognized event has anything
/// to be "from" yet, so it's always accepted. This lenience (not
/// enforcing the transition table during replay, only at write time -
/// see `transition` below) is deliberate and matches dodrenett's proven
/// "self-healing against malformed history" policy: a corrupted or
/// out-of-order event never crashes a read, it's just ignored.
pub fn replay<K: AggregateKind>(id: &str, events: &[Event]) -> Option<Aggregate<K>> {
let mut current: Option<Aggregate<K>> = None;
for event in events {
let Some(next) = K::from_event_type(&event.event_type) else {
continue;
};
let accept = match &current {
None => true,
Some(agg) => next == agg.state || agg.state.allowed().contains(&next),
};
if !accept {
continue;
}
current = Some(Aggregate {
id: id.to_string(),
state: next,
payload: event.payload.clone(),
sequence: event.sequence,
});
}
current
}
#[derive(Debug, Clone)]
pub enum TransitionError {
/// No creation event found for this id - nothing to transition.
UnknownAggregate,
DisallowedTransition { from: String, to: String },
/// A concurrent writer already advanced this aggregate past the
/// sequence this call read - the fix for the lost-update race
/// `answers::transition_answer` used to have (see
/// `crate::events::store::append_event`'s CAS).
Conflict,
Store(String),
}
impl std::fmt::Display for TransitionError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::UnknownAggregate => write!(f, "unknown item"),
Self::DisallowedTransition { from, to } => {
write!(f, "cannot go from {from} to {to}")
}
Self::Conflict => write!(f, "item was just changed by someone else, try again"),
Self::Store(e) => write!(f, "storage error: {e}"),
}
}
}
fn classify_publish_err(e: async_nats::jetstream::context::PublishError) -> TransitionError {
use async_nats::jetstream::context::PublishErrorKind;
match e.kind() {
PublishErrorKind::WrongLastSequence | PublishErrorKind::WrongLastMessageId => {
TransitionError::Conflict
}
_ => TransitionError::Store(e.to_string()),
}
}
/// Appends the creation event for a brand-new aggregate. `expected_seq:
/// Some(0)` (no message yet on this id's subject) doubles as a
/// create-only-once guard - a double-submit race on the same id fails
/// the same way a disallowed transition does, rather than silently
/// overwriting.
pub async fn create<K: AggregateKind>(
js: &async_nats::jetstream::Context,
id: &str,
payload: serde_json::Value,
occurred_at_ms: i64,
) -> Result<Aggregate<K>, TransitionError> {
let ack = append_event(
js,
K::AGGREGATE_TYPE,
id,
K::INITIAL_STATE.event_type(),
payload.clone(),
occurred_at_ms,
Some(0),
)
.await
.map_err(classify_publish_err)?;
Ok(Aggregate {
id: id.to_string(),
state: K::INITIAL_STATE,
payload,
sequence: ack.sequence,
})
}
/// Loads and replays `id`'s current state, then - if `target` is
/// actually reachable from it - appends the corresponding event with
/// the aggregate's just-replayed sequence as the CAS guard.
pub async fn transition<K: AggregateKind>(
js: &async_nats::jetstream::Context,
id: &str,
target: K,
payload: serde_json::Value,
occurred_at_ms: i64,
) -> Result<Aggregate<K>, TransitionError> {
let events = load_events(js, K::AGGREGATE_TYPE, id)
.await
.map_err(|e| TransitionError::Store(e.to_string()))?;
let current = replay::<K>(id, &events).ok_or(TransitionError::UnknownAggregate)?;
if !current.state.allowed().contains(&target) {
return Err(TransitionError::DisallowedTransition {
from: format!("{:?}", current.state),
to: format!("{target:?}"),
});
}
let ack = append_event(
js,
K::AGGREGATE_TYPE,
id,
target.event_type(),
payload.clone(),
occurred_at_ms,
Some(current.sequence),
)
.await
.map_err(classify_publish_err)?;
Ok(Aggregate {
id: id.to_string(),
state: target,
payload,
sequence: ack.sequence,
})
}
#[cfg(test)]
mod tests {
use super::*;
use crate::events::store::Event;
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
enum TestState {
Open,
A,
B,
}
impl AggregateKind for TestState {
const AGGREGATE_TYPE: &'static str = "test";
const INITIAL_STATE: Self = TestState::Open;
fn from_event_type(event_type: &str) -> Option<Self> {
match event_type {
"opened" => Some(TestState::Open),
"a" => Some(TestState::A),
"b" => Some(TestState::B),
_ => None,
}
}
fn event_type(self) -> &'static str {
match self {
TestState::Open => "opened",
TestState::A => "a",
TestState::B => "b",
}
}
fn allowed(self) -> &'static [Self] {
match self {
TestState::Open => &[TestState::A, TestState::B],
TestState::A | TestState::B => &[],
}
}
}
fn event(seq: u64, event_type: &str) -> Event {
Event {
id: uuid::Uuid::new_v4(),
aggregate_type: "test".to_string(),
aggregate_id: "x".to_string(),
event_type: event_type.to_string(),
payload: serde_json::json!({}),
occurred_at_ms: 0,
sequence: seq,
}
}
#[test]
fn empty_history_is_no_aggregate() {
assert!(replay::<TestState>("x", &[]).is_none());
}
#[test]
fn valid_path_replays() {
let events = vec![event(1, "opened"), event(2, "a")];
let agg = replay::<TestState>("x", &events).unwrap();
assert_eq!(agg.state, TestState::A);
assert_eq!(agg.sequence, 2);
}
#[test]
fn disallowed_edge_is_skipped_not_fatal() {
// "a" then "b" - B isn't reachable from A, so the second event
// is silently ignored, not an error.
let events = vec![event(1, "opened"), event(2, "a"), event(3, "b")];
let agg = replay::<TestState>("x", &events).unwrap();
assert_eq!(agg.state, TestState::A);
}
#[test]
fn unrecognized_event_type_is_skipped() {
let events = vec![event(1, "opened"), event(2, "bogus"), event(3, "a")];
let agg = replay::<TestState>("x", &events).unwrap();
assert_eq!(agg.state, TestState::A);
}
proptest::proptest! {
#[test]
fn replay_never_panics_and_state_always_valid(
types in proptest::collection::vec(
proptest::sample::select(vec!["opened", "a", "b", "bogus"]),
0..30,
)
) {
let events: Vec<Event> = types
.iter()
.enumerate()
.map(|(i, t)| event(i as u64, t))
.collect();
let result = replay::<TestState>("x", &events);
if let Some(agg) = result {
// Whatever state we landed in must itself be a real
// state this machine declares - trivially true here
// since TestState::allowed() only ever returns TestState
// values, but this is the shape the real aggregates'
// proptests repeat: replaying arbitrary event-type noise
// must never produce anything but a legitimate state.
let _: TestState = agg.state;
assert!(agg.sequence <= events.len() as u64);
}
}
}
}
+43
View File
@@ -0,0 +1,43 @@
//! Project submissions (`record_as: projects` - absorbs what used to be
//! called "inquiries"). A visitor submits a project idea plus who it's
//! for (org/contact details, captured as plain payload fields - no
//! separate Organization aggregate yet, see the event-sourcing plan).
//! `Open -> Accepted | Declined` - a real b2b-pipeline pair of terminal
//! states rather than the old generic `Handled`.
use super::AggregateKind;
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
pub enum State {
Open,
Accepted,
Declined,
}
impl AggregateKind for State {
const AGGREGATE_TYPE: &'static str = "project";
const INITIAL_STATE: Self = State::Open;
fn from_event_type(event_type: &str) -> Option<Self> {
match event_type {
"submitted" => Some(State::Open),
"accepted" => Some(State::Accepted),
"declined" => Some(State::Declined),
_ => None,
}
}
fn event_type(self) -> &'static str {
match self {
State::Open => "submitted",
State::Accepted => "accepted",
State::Declined => "declined",
}
}
fn allowed(self) -> &'static [Self] {
match self {
State::Open => &[State::Accepted, State::Declined],
State::Accepted | State::Declined => &[],
}
}
}
+38
View File
@@ -0,0 +1,38 @@
//! Newsletter subscribers (`record_as: subscribers`). `Open ->
//! Unsubscribed`, fired via `self_transition` (see `answers.rs`) rather
//! than an owner-gated `transition`, so a subscriber never needs a
//! signed-in session to leave.
use super::AggregateKind;
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
pub enum State {
Open,
Unsubscribed,
}
impl AggregateKind for State {
const AGGREGATE_TYPE: &'static str = "subscriber";
const INITIAL_STATE: Self = State::Open;
fn from_event_type(event_type: &str) -> Option<Self> {
match event_type {
"subscribed" => Some(State::Open),
"unsubscribed" => Some(State::Unsubscribed),
_ => None,
}
}
fn event_type(self) -> &'static str {
match self {
State::Open => "subscribed",
State::Unsubscribed => "unsubscribed",
}
}
fn allowed(self) -> &'static [Self] {
match self {
State::Open => &[State::Unsubscribed],
State::Unsubscribed => &[],
}
}
}
+157 -20
View File
@@ -1,15 +1,15 @@
//! Durable, queryable storage for submitted answers - the KV-backed
//! counterpart to the ephemeral `events::AnswerSubmitted` every
//! submission already publishes. The runtime here has no notion of
//! "applicant" or any other specific workflow: an `Alternative.record_as`
//! names a NATS KV bucket (nothing more); on submit, the response gets
//! wrapped in the same envelope (`Answer`) regardless of what content
//! produced it. `resource::get_resource` reads it back generically, and
//! `transition_answer` below moves an answer between states named in
//! content (`ResourceSpec.transitions`), never states hardcoded here.
//! Whatever workflow this ends up powering - applicant review, order
//! triage, comment moderation - is entirely a property of some
//! `portal-content` YAML, not of this file.
//! Read-model storage for submitted answers - the KV-backed projection
//! of the durable event log now underneath it (`events::store`,
//! `aggregates`) for the three buckets migrated onto that machinery
//! this pass (`content::aggregate_type_for_bucket`). An
//! `Alternative.record_as`/`ResourceSpec`'s bucket is still just a
//! bucket name as far as `resource::get_resource` is concerned - this
//! file is where that name additionally gets checked against a real,
//! Rust-defined transition table, for the buckets that have one.
//! A bucket outside that three-entry map still works exactly as
//! before (direct KV mutate-in-place, no event log, no CAS) - this is
//! deliberately not a hard cutover, so content isn't forced to migrate
//! before it needs to.
use leptos::prelude::*;
use serde::{Deserialize, Serialize};
@@ -71,7 +71,30 @@ pub async fn store_answer(
decided_by: None,
};
let data = serde_json::to_vec(&answer)?;
store.put(id, data.into()).await?;
store.put(id.clone(), data.into()).await?;
// Also seed the event log for the three aggregate types migrated
// onto it this pass - best-effort, same "a hiccup here shouldn't
// fail a submission the NATS notification event has already
// recorded" policy as the KV write above.
if let Some(agg_type) = crate::content::aggregate_type_for_bucket(bucket) {
use crate::aggregates::{applicant, create, project, subscriber};
let seed = match agg_type {
"applicant" => create::<applicant::State>(js, &id, responses.clone(), submitted_ms)
.await
.map(|_| ()),
"subscriber" => create::<subscriber::State>(js, &id, responses.clone(), submitted_ms)
.await
.map(|_| ()),
"project" => create::<project::State>(js, &id, responses.clone(), submitted_ms)
.await
.map(|_| ()),
_ => Ok(()),
};
if let Err(e) = seed {
tracing::error!(bucket, id, error = %e, "failed to seed event log for submission");
}
}
Ok(())
}
@@ -138,9 +161,12 @@ pub async fn transition_answer(
return Err(ServerFnError::new("not authorized"));
}
let bucket = resource
.bucket()
.ok_or_else(|| ServerFnError::new("resource is not kv-backed, has no transitions"))?;
let store = state
.jetstream
.get_key_value(&resource.bucket)
.get_key_value(bucket)
.await
.map_err(|e| ServerFnError::new(format!("resource bucket unavailable: {e}")))?;
let bytes = store
@@ -155,6 +181,22 @@ pub async fn transition_answer(
}
let decided_ms = chrono::Utc::now().timestamp_millis();
let decision_payload_for_event =
serde_json::json!({ "to": to, "item": item_id, "by": user.username });
// For the three buckets migrated onto the event log this pass, the
// real fix for the lost-update race two concurrent decisions on the
// same item used to hit: append with CAS on the aggregate's
// just-replayed sequence, so a second racing caller's write is
// rejected instead of silently overwriting the first. A bucket
// outside that map falls back to the direct KV mutate this always
// did - not every resource has to be migrated to keep working.
if let Some(agg_type) = crate::content::aggregate_type_for_bucket(bucket) {
transition_by_aggregate(&state.jetstream, agg_type, &item_id, &to, decision_payload_for_event.clone(), decided_ms)
.await
.map_err(|e| ServerFnError::new(e.to_string()))?;
}
answer.state = to.clone();
answer.decided_ms = Some(decided_ms);
answer.decided_by = Some(user.username.clone());
@@ -169,15 +211,14 @@ pub async fn transition_answer(
// submission hash, published the same way any other answer is -
// question_id/alternative come from this call's own arguments and
// the content-declared transition label, never a hardcoded value.
let decision_payload = serde_json::json!({ "to": to, "item": item_id, "by": user.username });
let parent_hashes = vec![item_id.clone()];
let chain_hash = hash_node(&question_id, &parent_hashes, &decision_payload, decided_ms);
let chain_hash = hash_node(&question_id, &parent_hashes, &decision_payload_for_event, decided_ms);
let event = AnswerSubmitted {
chain_hash,
parent_hashes,
question_id,
alternative: transition.label,
responses: decision_payload,
responses: decision_payload_for_event,
timestamp_ms: decided_ms,
};
emit_answer_submitted(&state.nats, &event)
@@ -187,6 +228,43 @@ pub async fn transition_answer(
Ok(())
}
/// Dispatches to the right aggregate type's `transition()` by name -
/// the one place `answers.rs` has to know the fixed, three-entry set of
/// migrated aggregate types (`content::aggregate_type_for_bucket`'s
/// counterpart), since `crate::aggregates` itself stays generic over
/// any `AggregateKind` and has no list of its own to dispatch through.
#[cfg(feature = "ssr")]
async fn transition_by_aggregate(
js: &async_nats::jetstream::Context,
aggregate_type: &str,
item_id: &str,
to: &str,
payload: serde_json::Value,
occurred_at_ms: i64,
) -> Result<(), crate::aggregates::TransitionError> {
use crate::aggregates::{applicant, project, subscriber, transition, AggregateKind, TransitionError};
macro_rules! dispatch {
($state:ty) => {{
let target = <$state>::from_event_type(to).ok_or_else(|| {
TransitionError::DisallowedTransition {
from: "?".to_string(),
to: to.to_string(),
}
})?;
transition::<$state>(js, item_id, target, payload, occurred_at_ms).await?;
}};
}
match aggregate_type {
"applicant" => dispatch!(applicant::State),
"subscriber" => dispatch!(subscriber::State),
"project" => dispatch!(project::State),
_ => return Err(TransitionError::UnknownAggregate),
}
Ok(())
}
/// The self-service counterpart to `transition_answer`: no signed-in
/// session, no group check - authorized instead by already holding
/// `item_id` (a chain hash, opaque and unguessable) plus a matching
@@ -244,6 +322,22 @@ pub async fn self_transition_answer(
}
let decided_ms = chrono::Utc::now().timestamp_millis();
let decision_payload_for_event =
serde_json::json!({ "to": self_transition.to, "item": item_id, "by": email });
if let Some(agg_type) = crate::content::aggregate_type_for_bucket(&self_transition.bucket) {
transition_by_aggregate(
&state.jetstream,
agg_type,
&item_id,
&self_transition.to,
decision_payload_for_event.clone(),
decided_ms,
)
.await
.map_err(|e| ServerFnError::new(e.to_string()))?;
}
answer.state = self_transition.to.clone();
answer.decided_ms = Some(decided_ms);
answer.decided_by = Some(email.clone());
@@ -254,15 +348,14 @@ pub async fn self_transition_answer(
.await
.map_err(|e| ServerFnError::new(e.to_string()))?;
let decision_payload = serde_json::json!({ "to": self_transition.to, "item": item_id, "by": email });
let parent_hashes = vec![item_id.clone()];
let chain_hash = hash_node(&question_id, &parent_hashes, &decision_payload, decided_ms);
let chain_hash = hash_node(&question_id, &parent_hashes, &decision_payload_for_event, decided_ms);
let event = AnswerSubmitted {
chain_hash,
parent_hashes,
question_id,
alternative: self_transition.label,
responses: decision_payload,
responses: decision_payload_for_event,
timestamp_ms: decided_ms,
};
emit_answer_submitted(&state.nats, &event)
@@ -271,3 +364,47 @@ pub async fn self_transition_answer(
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
/// `app.rs`'s list-rendering path only shows the nice
/// button-per-transition UI if a resource's raw JSON value
/// deserializes as `Vec<Answer>` - otherwise it silently falls
/// back to a plain read-only dump with no error. Guards that shape
/// contract directly, since nothing else would catch a drift here.
#[test]
fn answer_list_round_trips_through_the_shape_app_rs_expects() {
let answers = vec![
Answer {
id: "abc123".to_string(),
question_id: "/review".to_string(),
alternative: "Applicants".to_string(),
responses: serde_json::json!({"name": "Ada"}),
state: OPEN_STATE.to_string(),
submitted_ms: 1_700_000_000_000,
decided_ms: None,
decided_by: None,
},
Answer {
id: "def456".to_string(),
question_id: "/review".to_string(),
alternative: "Applicants".to_string(),
responses: serde_json::json!({"name": "Grace"}),
state: "invited".to_string(),
submitted_ms: 1_700_000_001_000,
decided_ms: Some(1_700_000_002_000),
decided_by: Some("bl".to_string()),
},
];
let value = serde_json::Value::Array(
answers.iter().map(|a| serde_json::to_value(a).unwrap()).collect(),
);
let round_tripped: Vec<Answer> =
serde_json::from_value(value).expect("Answer list round-trips");
assert_eq!(round_tripped.len(), 2);
assert_eq!(round_tripped[1].state, "invited");
assert_eq!(round_tripped[1].decided_by.as_deref(), Some("bl"));
}
}
+91 -2
View File
@@ -9,7 +9,7 @@ use serde::{Deserialize, Serialize};
use crate::answers::{Answer, SelfTransitionAnswer, TransitionAnswer};
use crate::auth::{current_user, User};
use crate::content::{is_qualified, Alternative, Question, Transition};
use crate::content::{is_qualified, Alternative, Question, Responsible, Transition};
use crate::resource::get_resource;
/// The visible site name/wordmark - "portal" is just this codebase's
@@ -157,6 +157,7 @@ fn QuestionView(
.into_any();
}
let question_id_for_note = question_id.clone();
view! {
<Hero
title=question.name.clone()
@@ -197,10 +198,64 @@ fn QuestionView(
}
/>
</div>
{question
.responsible
.clone()
.map(|r| {
view! { <ResponsibleNote question_id=question_id_for_note.clone() responsible=r/> }
})}
}
.into_any()
}
/// "Asked by X — contact them if you get stuck", plus a one-click
/// "report this question" action (see `report_question`) - mainly
/// meant for a question that reads as unhelpfully LLM-generated, so
/// whoever's responsible for it hears about it.
#[component]
fn ResponsibleNote(question_id: String, responsible: Responsible) -> impl IntoView {
let report = ServerAction::<ReportQuestion>::new();
let reported = RwSignal::new(false);
Effect::new(move |_| {
if report.value().get().is_some_and(|r| r.is_ok()) {
reported.set(true);
}
});
view! {
<p class="question-responsible">
"Asked by " {responsible.name.clone()} ""
<a href=format!("mailto:{}", responsible.contact)>"contact them"</a>
" if you get stuck."
{move || {
if reported.get() {
view! { <span class="question-reported">" Reported — thanks."</span> }
.into_any()
} else {
let question_id = question_id.clone();
view! {
<button
type="button"
class="question-report"
disabled=move || report.pending().get()
on:click=move |_| {
report
.dispatch(ReportQuestion {
question_id: question_id.clone(),
reason: None,
});
}
>
" Not helpful? Report this question."
</button>
}
.into_any()
}
}}
</p>
}
}
// Binds the actual "YES - Rasterized Lines" piece live at uhhm.no
// (ported to `public/yes.js`, an ES module now instead of a page-owning
// script) as a typed JS class - the idiomatic wasm-bindgen way to drive
@@ -691,7 +746,11 @@ fn ResourceFeature(
let feature_name = feature_name.clone();
move || (question_id.clone(), alternative.clone(), feature_name.clone())
},
|(q, a, f)| get_resource(q, a, f),
// Empty params for now - forwarding sibling requirement values
// as query params (for a resource parameterized by other form
// fields) is deferred, see the resource-backed multi-select
// requirement UI follow-up.
|(q, a, f)| get_resource(q, a, f, std::collections::HashMap::new()),
);
view! {
@@ -996,6 +1055,36 @@ pub async fn submit_answer(
Ok(SubmitResult { next, chain_hash })
}
/// Fire-and-forget "this question wasn't helpful" report - a plain log
/// entry in the event store (`aggregate_type: "question_report"`, no
/// state machine, nothing to transition), reviewable via
/// `events::store::load_events(js, "question_report", question_id)` the
/// same way any other aggregate's history is. No auth, same as reading
/// a public question - a report is cheap, low-stakes signal, not
/// something worth gating behind a session.
#[server]
pub async fn report_question(
question_id: String,
reason: Option<String>,
) -> Result<(), ServerFnError> {
use crate::server::AppState;
let state = expect_context::<AppState>();
let occurred_at_ms = chrono::Utc::now().timestamp_millis();
crate::events::store::append_event(
&state.jetstream,
"question_report",
&question_id,
"reported",
serde_json::json!({ "reason": reason }),
occurred_at_ms,
None,
)
.await
.map_err(|e| ServerFnError::new(e.to_string()))?;
Ok(())
}
/// Uploads one file to `POST /upload` and returns its stored object key
/// - the value a `type: file` requirement contributes to
/// `responses_json`, same as any other field. A plain `fetch`, not a
+148
View File
@@ -0,0 +1,148 @@
//! One-time migration: for each of the three renamed buckets
//! (`portal_applicants` -> `applicants`, `portal_subscribers` ->
//! `subscribers`, `portal_project_inquiries` -> `projects`), seeds the
//! new event log with a creation event (plus a decision event for any
//! record already decided) and re-puts the record into the new-named
//! KV bucket - then immediately replays what it just wrote and checks
//! it reproduces the original record's state exactly, per the
//! event-sourcing plan's verification step.
//!
//! Defaults to a dry run (lists what it *would* do, writes nothing) -
//! pass `--apply` to actually write. The old-named buckets are never
//! touched or deleted by this tool; remove them by hand once every
//! record's replay has been confirmed to match.
#![cfg(feature = "ssr")]
use portal::aggregates::{self, applicant, project, subscriber, AggregateKind};
use portal::answers::{Answer, OPEN_STATE};
use portal::events::store as event_store;
const MIGRATIONS: &[(&str, &str, &str)] = &[
("portal_applicants", "applicants", "applicant"),
("portal_subscribers", "subscribers", "subscriber"),
("portal_project_inquiries", "projects", "project"),
];
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let apply = std::env::args().any(|a| a == "--apply");
let nats_url =
std::env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".to_string());
let parsed = url::Url::parse(&nats_url)?;
let mut opts = async_nats::ConnectOptions::new();
if !parsed.username().is_empty() {
opts = opts.user_and_password(
parsed.username().to_string(),
parsed.password().unwrap_or_default().to_string(),
);
}
let nats = opts.connect(&nats_url).await?;
let js = async_nats::jetstream::new(nats);
event_store::ensure_stream(&js).await?;
println!("mode: {}\n", if apply { "APPLY (writing)" } else { "DRY RUN (no writes)" });
for (old_bucket, new_bucket, agg_type) in MIGRATIONS {
println!("=== {old_bucket} -> {new_bucket} ({agg_type}) ===");
let store = match js.get_key_value(*old_bucket).await {
Ok(s) => s,
Err(e) => {
println!(" skip: bucket unavailable: {e}");
continue;
}
};
use futures::TryStreamExt;
let keys: Vec<String> = store.keys().await?.try_collect().await?;
if keys.is_empty() {
println!(" (empty)");
}
for key in keys {
let Some(bytes) = store.get(&key).await? else { continue };
let answer: Answer = serde_json::from_slice(&bytes)?;
println!(" {key}: state={:?} submitted_ms={}", answer.state, answer.submitted_ms);
if apply {
if let Err(e) = seed_and_verify(&js, agg_type, &key, &answer, new_bucket).await {
println!(" ! {e}");
}
}
}
}
if !apply {
println!("\nDry run only - no writes made. Re-run with --apply to actually backfill.");
}
Ok(())
}
async fn seed_and_verify(
js: &async_nats::jetstream::Context,
agg_type: &str,
id: &str,
answer: &Answer,
new_bucket: &str,
) -> anyhow::Result<()> {
match agg_type {
"applicant" => seed::<applicant::State>(js, id, answer).await?,
"subscriber" => seed::<subscriber::State>(js, id, answer).await?,
"project" => seed::<project::State>(js, id, answer).await?,
other => anyhow::bail!("unknown aggregate type {other}"),
}
let new_store = match js.get_key_value(new_bucket).await {
Ok(s) => s,
Err(_) => {
js.create_key_value(async_nats::jetstream::kv::Config {
bucket: new_bucket.to_string(),
..Default::default()
})
.await?
}
};
new_store.put(id, serde_json::to_vec(answer)?.into()).await?;
Ok(())
}
/// Appends the initial event, then - if the original record was
/// already decided - a second event carrying the real decision, before
/// replaying and checking the result matches. Two events for a decided
/// record, not one: a single "imported" event would show a decided
/// record as forever open on replay.
async fn seed<K: AggregateKind>(
js: &async_nats::jetstream::Context,
id: &str,
answer: &Answer,
) -> anyhow::Result<()> {
aggregates::create::<K>(js, id, answer.responses.clone(), answer.submitted_ms)
.await
.map_err(|e| anyhow::anyhow!("seeding initial event: {e}"))?;
if answer.state != OPEN_STATE {
match K::from_event_type(&answer.state) {
Some(target) => {
let payload = serde_json::json!({
"to": answer.state,
"item": id,
"by": answer.decided_by,
});
let occurred = answer.decided_ms.unwrap_or(answer.submitted_ms);
aggregates::transition::<K>(js, id, target, payload, occurred)
.await
.map_err(|e| anyhow::anyhow!("seeding decision event: {e}"))?;
}
None => println!(" ! unrecognized decided state {:?}, only seeded initial event", answer.state),
}
}
let events = event_store::load_events(js, K::AGGREGATE_TYPE, id).await?;
match aggregates::replay::<K>(id, &events) {
Some(agg) if agg.state.event_type() == answer.state => {
println!(" OK: replay matches original state {:?}", answer.state);
}
Some(agg) => {
println!(" MISMATCH: replayed {:?}, original was {:?}", agg.state, answer.state);
}
None => println!(" MISMATCH: no aggregate replayed at all"),
}
Ok(())
}
+76
View File
@@ -0,0 +1,76 @@
//! Headless schema-check binary - loads content (from a Gitea repo URL
//! or a local directory) and validates it exactly the way
//! `content::watch_for_reload`/`main.rs`'s boot path do, with no NATS,
//! OIDC, web server, or JetStream connection involved. Built once by
//! portal's own deploy workflow and downloaded as a release artifact by
//! `questions`' own CI, rather than compiled there - keeps that repo's
//! CI coupling to "run a static binary," not "build a Rust workspace."
#![cfg(feature = "ssr")]
use portal::content;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let mut args = std::env::args().skip(1);
let mut repo: Option<String> = None;
let mut branch = "main".to_string();
let mut subdir = "questions".to_string();
let mut path: Option<String> = None;
while let Some(arg) = args.next() {
match arg.as_str() {
"--repo" => repo = args.next(),
"--branch" => branch = args.next().unwrap_or(branch),
"--subdir" => subdir = args.next().unwrap_or(subdir),
"--path" => path = args.next(),
other => {
eprintln!("unknown argument: {other}");
std::process::exit(2);
}
}
}
let questions = if let Some(dir) = path {
load_from_dir(&dir)?
} else if let Some(repo_url) = repo {
content::load_questions_from_gitea(&repo_url, &branch, &subdir).await?
} else {
eprintln!(
"usage: question-lint --repo <gitea-url> [--branch main] [--subdir questions] | --path <local-dir>"
);
std::process::exit(2);
};
match content::validate_questions(&questions) {
Ok(()) => {
println!("OK: {} question(s) valid", questions.len());
Ok(())
}
Err(e) => {
eprintln!("FAIL: {e}");
std::process::exit(1);
}
}
}
/// The offline counterpart to `content::load_questions_from_gitea` -
/// same "every `*.yaml` file becomes a `Question` keyed by its own
/// `id`" shape, just reading a local checkout instead of Gitea's API,
/// for linting a branch that hasn't been pushed yet.
fn load_from_dir(
dir: &str,
) -> anyhow::Result<std::collections::HashMap<String, content::Question>> {
let mut out = std::collections::HashMap::new();
for entry in std::fs::read_dir(dir)? {
let entry = entry?;
let path = entry.path();
if path.extension().and_then(|e| e.to_str()) != Some("yaml") {
continue;
}
let raw = std::fs::read_to_string(&path)?;
let question: content::Question = serde_yaml::from_str(&raw)
.map_err(|e| anyhow::anyhow!("parsing {}: {e}", path.display()))?;
out.insert(question.id.clone(), question);
}
Ok(out)
}
+151 -3
View File
@@ -21,6 +21,20 @@ pub struct Question {
pub qualifies: Option<String>,
#[serde(default)]
pub alternatives: Vec<Alternative>,
/// Who to contact if a visitor gets stuck or finds this question
/// unhelpful - rendered as a small line on the page, and the person
/// a "report this question" action names in its own event payload
/// (see `events::store`, `question_report` aggregate-less events).
#[serde(default)]
pub responsible: Option<Responsible>,
}
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct Responsible {
pub name: String,
/// A mailto address or plain email - deliberately just a string,
/// same as everywhere else content keeps contact info simple.
pub contact: String,
}
/// Whether `user` may view/submit `question`. `true` when the question
@@ -105,8 +119,9 @@ pub struct Feature {
/// item.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct ResourceSpec {
pub bucket: String,
/// A single item; omit to list the whole bucket.
pub source: ResourceSource,
/// A single item; omit to list the whole bucket. Only meaningful
/// for a `Kv` source.
#[serde(default)]
pub key: Option<String>,
/// Kanidm group required to read this resource.
@@ -122,9 +137,59 @@ pub struct ResourceSpec {
/// States a listed answer may move to, and the button label for
/// each - empty means the resource is read-only. The allow-list a
/// `transition_answer` call is checked against, so a client can
/// never move an answer to a state content didn't declare.
/// never move an answer to a state content didn't declare. Only
/// meaningful for a `Kv` source - a live external pull is always
/// read-only.
#[serde(default)]
pub transitions: Vec<Transition>,
/// A jq filter reshaping the fetched value before it reaches the
/// frontend - e.g. `.[] | {name, url: .html_url}` to pick just the
/// fields a showcase card needs out of a raw Gitea API response.
/// Evaluated via the `jaq` crate (`resource::apply_jq`), no shell
/// -out. `None` (or a `Kv` source) returns the fetched value as-is.
#[serde(default)]
pub jq: Option<String>,
}
/// Where a resource's live data actually comes from. `Kv` is the
/// original (and still only mutable) behavior - a NATS KV bucket this
/// server itself owns. The `Gitea*` variants are live pulls from the
/// same Gitea instance content is loaded from (`AppState.gitea_base`),
/// for surfacing things like a portfolio of starred repos without
/// hand-curating a KV bucket for it. Not mutually exclusive with
/// owner-curated resources - a hand-picked feature list is just a `Kv`
/// resource an owner writes into; a `GiteaStarred`/`GiteaOrgRepos` pull
/// is the automatic catalog to choose from. Both can appear as
/// different features on the same page.
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum ResourceSource {
Kv { bucket: String },
GiteaStarred { username: String },
GiteaOrgRepos { org: String },
/// Any other HTTPS JSON endpoint - the general escape hatch for a
/// live resource that isn't this Gitea instance. Deliberately
/// scheme-restricted and checked against loopback/private/link
/// -local addresses at fetch time (`resource::fetch_url_resource`)
/// - content is trusted today (same org owns both repos), but a
/// server-side fetch of a content-supplied URL is real SSRF surface
/// the moment that stops being true, so it fails closed rather than
/// trusting every host by default.
Url { url: String },
}
impl ResourceSpec {
/// The KV bucket this resource reads/writes, if it's `Kv`-sourced -
/// `None` for a live external pull, which has no bucket and (see
/// `transitions` above) can't be mutated through this mechanism.
pub fn bucket(&self) -> Option<&str> {
match &self.source {
ResourceSource::Kv { bucket } => Some(bucket),
ResourceSource::GiteaStarred { .. }
| ResourceSource::GiteaOrgRepos { .. }
| ResourceSource::Url { .. } => None,
}
}
}
#[derive(Clone, Debug, Serialize, Deserialize)]
@@ -251,6 +316,85 @@ pub async fn load_questions_from_gitea(
Ok(out)
}
/// Maps a content-declared KV bucket name to the aggregate type
/// (`crate::aggregates::*::State::AGGREGATE_TYPE`) it's event-sourced
/// under - the fixed, three-entry table for this pass (see the
/// event-sourcing plan). A bucket name not listed here is left alone
/// entirely (no validation applied) - deliberate: only buckets actually
/// migrated onto the generic aggregate machinery get their transitions
/// checked.
#[cfg(feature = "ssr")]
pub fn aggregate_type_for_bucket(bucket: &str) -> Option<&'static str> {
match bucket {
"applicants" => Some("applicant"),
"subscribers" => Some("subscriber"),
"projects" => Some("project"),
_ => None,
}
}
#[cfg(feature = "ssr")]
fn is_valid_transition_target(aggregate_type: &str, to: &str) -> bool {
use crate::aggregates::AggregateKind;
fn check<K: AggregateKind>(to: &str) -> bool {
K::INITIAL_STATE.allowed().iter().any(|s| s.event_type() == to)
}
match aggregate_type {
"applicant" => check::<crate::aggregates::applicant::State>(to),
"subscriber" => check::<crate::aggregates::subscriber::State>(to),
"project" => check::<crate::aggregates::project::State>(to),
_ => false,
}
}
/// Validates every declared transition target (`SelfTransition.to`,
/// `ResourceSpec.transitions[].to`) against the real compiled
/// transition table for its bucket's aggregate type, for every bucket
/// this pass actually migrated (see `aggregate_type_for_bucket`).
/// Called on every content load/reload (`watch_for_reload`, `main.rs`'s
/// boot path) and by the standalone `question-lint` binary, so a YAML
/// typo becomes a caught, logged rejection instead of a silently
/// -accepted, later-broken string.
#[cfg(feature = "ssr")]
pub fn validate_questions(
questions: &std::collections::HashMap<String, Question>,
) -> anyhow::Result<()> {
for question in questions.values() {
for alternative in &question.alternatives {
if let Some(st) = &alternative.self_transition {
if let Some(agg) = aggregate_type_for_bucket(&st.bucket) {
if !is_valid_transition_target(agg, &st.to) {
anyhow::bail!(
"question {:?} alternative {:?}: self_transition.to {:?} is not a valid transition for bucket {:?} ({agg})",
question.id, alternative.name, st.to, st.bucket
);
}
}
}
for feature in &alternative.features {
let Some(resource) = &feature.resource else {
continue;
};
let Some(bucket) = resource.bucket() else {
continue;
};
let Some(agg) = aggregate_type_for_bucket(bucket) else {
continue;
};
for transition in &resource.transitions {
if !is_valid_transition_target(agg, &transition.to) {
anyhow::bail!(
"question {:?} alternative {:?} feature {:?}: transition.to {:?} is not a valid transition for bucket {:?} ({agg})",
question.id, alternative.name, feature.name, transition.to, bucket
);
}
}
}
}
}
Ok(())
}
/// Published by the content repo's own CI (after it lints a push) to
/// tell every running instance to pick up the change - a plain fire
/// and forget NATS publish, no payload, matching `events.rs`'s
@@ -283,6 +427,10 @@ pub async fn watch_for_reload(
while sub.next().await.is_some() {
match load_questions_from_gitea(&repo_url, &branch, &subdir).await {
Ok(loaded) => {
if let Err(e) = validate_questions(&loaded) {
tracing::error!(error = %e, "content reload failed validation, keeping last-good content");
continue;
}
let count = loaded.len();
questions.store(std::sync::Arc::new(loaded));
tracing::info!(count, "reloaded content");
+2
View File
@@ -6,6 +6,8 @@
//! change shape when that shows up, only gain a second subscriber.
#![cfg(feature = "ssr")]
pub mod store;
use serde::{Deserialize, Serialize};
#[derive(Clone, Debug, Serialize, Deserialize)]
+150
View File
@@ -0,0 +1,150 @@
//! Durable, replayable event log for the aggregate state machines in
//! `crate::aggregates` - the source of truth `answers::Answer` KV
//! records now only *project*. Ported from dodrenett's `lib/eventStore.ts`
//! with two gaps closed rather than copied: dodrenett's `sequence`
//! field was always hardcoded `0` (ordering relied entirely on
//! JetStream's own append order anyway - here the real per-message
//! sequence is used directly, nothing fake); dodrenett never used
//! `Nats-Msg-Id`/CAS at all. This module uses both, via `async-nats`
//! 0.38's native support - no new dependency.
//!
//! Subject scheme is `events.<aggregate_type>.<aggregate_id>`,
//! deliberately *not* `events.<type>.<id>.<eventType>` like dodrenett -
//! dropping the event type from the subject means
//! `expected_last_subject_sequence` gives real optimistic concurrency
//! across an aggregate's *entire* history, not just same-typed events.
//! The cost: no cheap subject-wildcard query for "every `invited` event
//! across all applicants" - nobody needs that today, and it's a filter
//! over `load_events`'s output if it's ever needed.
use serde::{Deserialize, Serialize};
pub const STREAM_NAME: &str = "EVENTS";
/// One durably-stored fact about an aggregate. `sequence` is filled in
/// from JetStream's own per-message sequence number on read
/// (`load_events`) - never trust a value baked into the payload itself.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct Event {
pub id: uuid::Uuid,
pub aggregate_type: String,
pub aggregate_id: String,
pub event_type: String,
pub payload: serde_json::Value,
pub occurred_at_ms: i64,
#[serde(default)]
pub sequence: u64,
}
fn subject(aggregate_type: &str, aggregate_id: &str) -> String {
format!("events.{aggregate_type}.{aggregate_id}")
}
/// Idempotent - safe to call on every boot. `max_age` deliberately left
/// at the default (unlimited) - dodrenett's 7-day retention was an
/// explicit dev default, not something worth inheriting for what's
/// meant to be the durable source of truth.
pub async fn ensure_stream(
js: &async_nats::jetstream::Context,
) -> anyhow::Result<async_nats::jetstream::stream::Stream> {
let stream = js
.get_or_create_stream(async_nats::jetstream::stream::Config {
name: STREAM_NAME.to_string(),
subjects: vec!["events.>".to_string()],
..Default::default()
})
.await?;
Ok(stream)
}
/// Appends one event. `expected_seq` set to the aggregate's last-known
/// sequence turns this into a compare-and-swap: a concurrent caller
/// racing against the same aggregate with a stale `expected_seq` gets
/// `PublishError` back instead of silently winning a lost-update race
/// (see `crate::aggregates`, `transition_answer`'s old direct
/// KV-mutate-in-place bug this replaces). `event.id` is set as the
/// JetStream `Nats-Msg-Id` for the stream's dedup window - only
/// catches a genuine retry with the *same* id, not a fresh submission;
/// callers that want real retry-dedup must pass a stable id themselves.
/// Returns the raw `PublishError` (not wrapped in `anyhow`) so callers
/// like `crate::aggregates::classify_publish_err` can distinguish a CAS
/// conflict (`WrongLastSequence`) from every other failure mode.
pub async fn append_event(
js: &async_nats::jetstream::Context,
aggregate_type: &str,
aggregate_id: &str,
event_type: &str,
payload: serde_json::Value,
occurred_at_ms: i64,
expected_seq: Option<u64>,
) -> Result<async_nats::jetstream::publish::PublishAck, async_nats::jetstream::context::PublishError>
{
let event = Event {
id: uuid::Uuid::new_v4(),
aggregate_type: aggregate_type.to_string(),
aggregate_id: aggregate_id.to_string(),
event_type: event_type.to_string(),
payload,
occurred_at_ms,
sequence: 0, // ignored on write - the server assigns the real one
};
// Every field here is a plain, always-serializable type (strings,
// an i64, a Uuid, a serde_json::Value) - matches `chain.rs::hash_node`'s
// own "json values always serialize" precedent.
let data = serde_json::to_vec(&event).expect("Event always serializes");
let mut publish = async_nats::jetstream::context::Publish::build()
.payload(data.into())
.message_id(event.id.to_string());
if let Some(seq) = expected_seq {
publish = publish.expected_last_subject_sequence(seq);
}
js.send_publish(subject(aggregate_type, aggregate_id), publish)
.await?
.await
}
/// Loads every event for one aggregate, in real JetStream append order.
/// Uses an ephemeral pull consumer scoped to this one aggregate's
/// subject, fetched once and torn down - same idiom dodrenett's
/// `getEvents` used, appropriate at this scale (single-digit events per
/// aggregate), not a durable/long-lived consumer.
pub async fn load_events(
js: &async_nats::jetstream::Context,
aggregate_type: &str,
aggregate_id: &str,
) -> anyhow::Result<Vec<Event>> {
use futures::TryStreamExt;
let stream = js.get_stream(STREAM_NAME).await?;
let subj = subject(aggregate_type, aggregate_id);
let consumer = stream
.create_consumer(async_nats::jetstream::consumer::pull::Config {
filter_subject: subj,
..Default::default()
})
.await?;
let mut events = Vec::new();
let mut messages = consumer.fetch().max_messages(10_000).messages().await?;
while let Some(message) = messages
.try_next()
.await
.map_err(|e| anyhow::anyhow!("fetching messages: {e}"))?
{
let info = message.info().map_err(|e| anyhow::anyhow!("reading message info: {e}"))?;
let mut event: Event = serde_json::from_slice(&message.payload)?;
event.sequence = info.stream_sequence;
events.push(event);
message.ack().await.map_err(|e| anyhow::anyhow!("acking: {e}"))?;
}
events.sort_by_key(|e| e.sequence);
// Best-effort teardown - an ephemeral consumer left behind expires
// on its own (default inactive_threshold), so a delete failure here
// isn't worth failing the read over.
let _ = stream.delete_consumer(&consumer.cached_info().name).await;
Ok(events)
}
+3
View File
@@ -6,6 +6,9 @@ pub mod content;
pub mod events;
pub mod resource;
#[cfg(feature = "ssr")]
pub mod aggregates;
#[cfg(feature = "ssr")]
pub mod server;
+2
View File
@@ -30,6 +30,7 @@ async fn main() -> anyhow::Result<()> {
let content_branch = std::env::var("CONTENT_BRANCH").unwrap_or_else(|_| "main".to_string());
let gitea_base = content::gitea_api_base(&content_repo)?;
let questions = content::load_questions_from_gitea(&content_repo, &content_branch, "questions").await?;
content::validate_questions(&questions)?;
tracing::info!(count = questions.len(), repo = %content_repo, branch = %content_branch, "loaded content");
let questions = Arc::new(arc_swap::ArcSwap::from_pointee(questions));
@@ -50,6 +51,7 @@ async fn main() -> anyhow::Result<()> {
// names whatever bucket an alternative's `record_as` should write
// into, and `answers::store_answer` creates it on first use.
let jetstream = async_nats::jetstream::new(nats.clone());
portal::events::store::ensure_stream(&jetstream).await?;
let oidc_state = Arc::new(oidc::Oidc::from_env().await?);
+273 -36
View File
@@ -1,24 +1,30 @@
//! Generic, authorized reads of NATS KV data declared in content
//! Generic, authorized reads of live data declared in content
//! (`content::ResourceSpec`, on a `Feature`). The one property that
//! matters: the bucket/key a resource reads is never a client-supplied
//! parameter, only something the server resolves from its own
//! trusted, YAML-loaded content - the client only ever names a
//! question id + feature name, so it can't probe an arbitrary bucket
//! by just naming it.
//! matters: what a resource reads is never a client-supplied parameter,
//! only something the server resolves from its own trusted, YAML-loaded
//! content - the client only ever names a question id + feature name
//! (plus, now, whatever form-field values it already has, forwarded as
//! query params), never a bucket/URL directly.
use leptos::prelude::*;
/// Fetches the live data for `question_id`'s `alternative`'s
/// `feature_name` feature. Fails closed: a resource with neither
/// `public: true` nor `requires_group` set is unreachable, not "open"
/// by omission.
/// by omission. `params` are the requesting form's current field
/// values (if any) - forwarded as query parameters to a live external
/// fetch (`GiteaStarred`/`GiteaOrgRepos`/`Url`), the mechanism behind
/// "a resource parameterized by other form fields"; a `Kv` resource
/// ignores them entirely, same as today.
#[server]
pub async fn get_resource(
question_id: String,
alternative: String,
feature_name: String,
params: std::collections::HashMap<String, String>,
) -> Result<serde_json::Value, ServerFnError> {
use crate::auth::{User, SESSION_USER_KEY};
use crate::content::ResourceSource;
use crate::server::AppState;
let state = expect_context::<AppState>();
@@ -66,39 +72,270 @@ pub async fn get_resource(
}
}
let store = state
.jetstream
.get_key_value(&resource.bucket)
.await
.map_err(|e| ServerFnError::new(format!("resource bucket unavailable: {e}")))?;
match &resource.key {
Some(key) => {
let bytes = store
.get(key)
let value = match &resource.source {
ResourceSource::Kv { bucket } => {
let store = state
.jetstream
.get_key_value(bucket)
.await
.map_err(|e| ServerFnError::new(e.to_string()))?
.ok_or_else(|| ServerFnError::new("resource key not found"))?;
serde_json::from_slice(&bytes).map_err(|e| ServerFnError::new(e.to_string()))
}
None => {
use futures::TryStreamExt;
let keys: Vec<String> = store
.keys()
.await
.map_err(|e| ServerFnError::new(e.to_string()))?
.try_collect()
.await
.map_err(|e| ServerFnError::new(e.to_string()))?;
let mut items = Vec::new();
for key in keys {
if let Ok(Some(bytes)) = store.get(&key).await {
if let Ok(value) = serde_json::from_slice::<serde_json::Value>(&bytes) {
items.push(value);
.map_err(|e| ServerFnError::new(format!("resource bucket unavailable: {e}")))?;
match &resource.key {
Some(key) => {
let bytes = store
.get(key)
.await
.map_err(|e| ServerFnError::new(e.to_string()))?
.ok_or_else(|| ServerFnError::new("resource key not found"))?;
serde_json::from_slice(&bytes).map_err(|e| ServerFnError::new(e.to_string()))?
}
None => {
use futures::TryStreamExt;
let keys: Vec<String> = store
.keys()
.await
.map_err(|e| ServerFnError::new(e.to_string()))?
.try_collect()
.await
.map_err(|e| ServerFnError::new(e.to_string()))?;
let mut items = Vec::new();
for key in keys {
if let Ok(Some(bytes)) = store.get(&key).await {
if let Ok(value) = serde_json::from_slice::<serde_json::Value>(&bytes) {
items.push(value);
}
}
}
serde_json::Value::Array(items)
}
}
Ok(serde_json::Value::Array(items))
}
ResourceSource::GiteaStarred { username } => {
fetch_gitea_json(&state, &format!("/api/v1/users/{username}/starred"), &params).await?
}
ResourceSource::GiteaOrgRepos { org } => {
fetch_gitea_json(&state, &format!("/api/v1/orgs/{org}/repos"), &params).await?
}
ResourceSource::Url { url } => fetch_url_json(url, &params).await?,
};
match &resource.jq {
Some(filter) => apply_jq(filter, &value),
None => Ok(value),
}
}
/// Fetches JSON from a path on the same Gitea instance content is
/// loaded from (`AppState.gitea_base`), attaching `GITEA_API_TOKEN` if
/// set - confirmed live against `project.uhhm.no` that at least the
/// starred-repos endpoint 401s unauthenticated, unlike the repo
/// -contents/repo-info endpoints `content.rs` already calls anonymously.
#[cfg(feature = "ssr")]
async fn fetch_gitea_json(
state: &crate::server::AppState,
path: &str,
params: &std::collections::HashMap<String, String>,
) -> Result<serde_json::Value, ServerFnError> {
let mut url = url::Url::parse(&format!("{}{}", state.gitea_base, path))
.map_err(|e| ServerFnError::new(e.to_string()))?;
for (k, v) in params {
url.query_pairs_mut().append_pair(k, v);
}
let client = openidconnect::reqwest::Client::new();
let mut req = client.get(url.clone());
if let Ok(token) = std::env::var("GITEA_API_TOKEN") {
if !token.is_empty() {
req = req.header("Authorization", format!("token {token}"));
}
}
let text = req
.send()
.await
.and_then(|r| r.error_for_status())
.map_err(|e| ServerFnError::new(format!("fetching {url}: {e}")))?
.text()
.await
.map_err(|e| ServerFnError::new(e.to_string()))?;
serde_json::from_str(&text).map_err(|e| ServerFnError::new(e.to_string()))
}
/// Fetches JSON from an arbitrary content-declared URL
/// (`content::ResourceSource::Url`). Content is trusted today (same
/// org owns both this app and its content repo), but a server-side
/// fetch of a content-supplied URL is real SSRF surface the moment
/// that stops being true - `assert_public_https_url` fails closed on
/// anything that isn't `https://` and actually resolves to a public
/// address, not just a syntactically-plausible one.
#[cfg(feature = "ssr")]
async fn fetch_url_json(
url: &str,
params: &std::collections::HashMap<String, String>,
) -> Result<serde_json::Value, ServerFnError> {
let mut parsed = assert_public_https_url(url)
.await
.map_err(|e| ServerFnError::new(e.to_string()))?;
for (k, v) in params {
parsed.query_pairs_mut().append_pair(k, v);
}
let client = openidconnect::reqwest::Client::new();
let text = client
.get(parsed.clone())
.send()
.await
.and_then(|r| r.error_for_status())
.map_err(|e| ServerFnError::new(format!("fetching {parsed}: {e}")))?
.text()
.await
.map_err(|e| ServerFnError::new(e.to_string()))?;
serde_json::from_str(&text).map_err(|e| ServerFnError::new(e.to_string()))
}
/// `https://` only, and only if every address the host resolves to is
/// public - rejects loopback (127.0.0.1, ::1), RFC1918/link-local
/// private ranges, and unspecified/multicast addresses. Resolves via
/// real DNS (not just a syntax check) so a hostname that *points at* an
/// internal address is caught too, not just a literal internal IP
/// spelled out directly.
#[cfg(feature = "ssr")]
async fn assert_public_https_url(url: &str) -> anyhow::Result<url::Url> {
let parsed = url::Url::parse(url)?;
if parsed.scheme() != "https" {
anyhow::bail!("only https:// resource urls are allowed");
}
let host = parsed
.host_str()
.ok_or_else(|| anyhow::anyhow!("resource url has no host"))?;
let port = parsed.port_or_known_default().unwrap_or(443);
let mut resolved = false;
for addr in tokio::net::lookup_host((host, port))
.await
.map_err(|e| anyhow::anyhow!("resolving resource host {host}: {e}"))?
{
resolved = true;
if !is_public_ip(&addr.ip()) {
anyhow::bail!("resource host {host} resolves to a non-public address");
}
}
if !resolved {
anyhow::bail!("resource host {host} did not resolve to any address");
}
Ok(parsed)
}
#[cfg(feature = "ssr")]
fn is_public_ip(ip: &std::net::IpAddr) -> bool {
match ip {
std::net::IpAddr::V4(v4) => {
!(v4.is_private()
|| v4.is_loopback()
|| v4.is_link_local()
|| v4.is_multicast()
|| v4.is_broadcast()
|| v4.is_unspecified()
|| v4.is_documentation())
}
std::net::IpAddr::V6(v6) => {
let is_unique_local = (v6.segments()[0] & 0xfe00) == 0xfc00;
let is_link_local = (v6.segments()[0] & 0xffc0) == 0xfe80;
!(v6.is_loopback()
|| v6.is_unspecified()
|| v6.is_multicast()
|| is_unique_local
|| is_link_local)
}
}
}
/// Runs `filter_src` (a jq filter) over `input`, collecting every value
/// it yields into a JSON array - e.g. `.[] | {name, url: .html_url}`
/// over a Gitea repo list yields one shaped object per repo. Pure Rust
/// via the `jaq` crate family, no shell-out to a `jq` binary.
#[cfg(feature = "ssr")]
pub fn apply_jq(
filter_src: &str,
input: &serde_json::Value,
) -> Result<serde_json::Value, ServerFnError> {
use jaq_core::load::{Arena, File, Loader};
use jaq_core::{data, unwrap_valr, Compiler, Ctx, Vars};
use jaq_json::{read, Val};
let input_text = input.to_string();
let input_val = read::parse_single(input_text.as_bytes())
.map_err(|e| ServerFnError::new(format!("internal: re-parsing json for jq: {e:?}")))?;
let program = File { code: filter_src, path: () };
let defs = jaq_core::defs().chain(jaq_std::defs()).chain(jaq_json::defs());
let funs = jaq_core::funs().chain(jaq_std::funs()).chain(jaq_json::funs());
let loader = Loader::new(defs);
let arena = Arena::default();
let modules = loader
.load(&arena, program)
.map_err(|e| ServerFnError::new(format!("jq filter parse error: {e:?}")))?;
let filter = Compiler::default()
.with_funs(funs)
.compile(modules)
.map_err(|e| ServerFnError::new(format!("jq filter compile error: {e:?}")))?;
let ctx = Ctx::<data::JustLut<Val>>::new(&filter.lut, Vars::new([]));
let mut outputs = Vec::new();
for result in filter.id.run((ctx, input_val)).map(unwrap_valr) {
let val = result.map_err(|e| ServerFnError::new(format!("jq filter runtime error: {e:?}")))?;
let text = val.to_string();
let json: serde_json::Value = serde_json::from_str(&text)
.map_err(|e| ServerFnError::new(format!("internal: parsing jq output: {e}")))?;
outputs.push(json);
}
Ok(serde_json::Value::Array(outputs))
}
#[cfg(test)]
mod tests {
use super::*;
/// A canned, Gitea-API-shaped fixture - the same fields the real
/// `/users/{username}/starred` endpoint returns - run through the
/// filter `index.yaml`'s "What we've built" resource actually
/// declares, confirming the `jaq` integration produces the shape
/// the frontend showcase card expects.
#[test]
fn jq_shapes_gitea_repo_list_for_the_showcase() {
let input = serde_json::json!([
{
"name": "cnats",
"description": "A NATS-backed chat client",
"html_url": "https://project.uhhm.no/bl/cnats",
"stargazers_count": 3,
"private": false
},
{
"name": "portal",
"description": "This app",
"html_url": "https://project.uhhm.no/uhhm/portal",
"stargazers_count": 1,
"private": false
}
]);
let filter = ".[] | {name: .name, description: .description, url: .html_url, stars: .stargazers_count}";
let shaped = apply_jq(filter, &input).expect("filter runs");
let items = shaped.as_array().expect("array output");
assert_eq!(items.len(), 2);
assert_eq!(items[0]["name"], "cnats");
assert_eq!(items[0]["url"], "https://project.uhhm.no/bl/cnats");
assert_eq!(items[0]["stars"], 3);
// The filter never mentions `private` - confirms shaping
// actually drops fields, not just passes the object through.
assert!(items[0].get("private").is_none());
}
#[test]
fn is_public_ip_rejects_loopback_and_private() {
let loopback: std::net::IpAddr = "127.0.0.1".parse().unwrap();
let private: std::net::IpAddr = "10.0.0.5".parse().unwrap();
let link_local: std::net::IpAddr = "169.254.1.1".parse().unwrap();
let public: std::net::IpAddr = "1.1.1.1".parse().unwrap();
assert!(!is_public_ip(&loopback));
assert!(!is_public_ip(&private));
assert!(!is_public_ip(&link_local));
assert!(is_public_ip(&public));
}
}