Event-sourced applicant/subscriber/project aggregates, generalized resources
Deploy / deploy (push) Failing after 3s
Deploy / deploy (push) Failing after 3s
Replaces the free-string, direct-KV-mutate state model in answers.rs
with a proper event log (events/store.rs, JetStream-backed, CAS via
expected_last_subject_sequence) and three pure state machines
(aggregates/{applicant,subscriber,project}.rs). Closes a real
lost-update race in the old transition_answer (concurrent decisions on
the same item could both win, publishing contradictory events). KV
buckets become best-effort read-model projections, not the source of
truth. Content-declared transition targets are now validated at
load/reload time against the real compiled transition tables, not
accepted as arbitrary strings.
Buckets renamed to describe their content, not their relation to the
app (portal_applicants -> applicants, etc); "inquiry" folded into a
richer "project" concept.
ResourceSpec generalized beyond a single KV bucket: Kv | GiteaStarred |
GiteaOrgRepos | Url sources, with an optional jq filter (via the jaq
crate) to shape live data for the frontend. Url source is SSRF-guarded
(https-only, rejects loopback/private/link-local, real DNS resolve).
New headless question_lint binary (validates content against compiled
transition tables with no NATS/OIDC/server involved) and a one-time
backfill_events binary (dry-run by default) for migrating existing KV
data onto the new event log.
Questions get an optional `responsible` contact plus a lightweight
"report this question" action.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
213b1130bb
commit
ebf4bf91b3
@@ -0,0 +1,40 @@
|
||||
//! Collaborator applicants (`/review`'s "Applicants" alternative,
|
||||
//! `record_as: applicants`). `Open -> Invited | Declined`, straight
|
||||
//! from the alternatives already declared in `questions/review.yaml`.
|
||||
use super::AggregateKind;
|
||||
|
||||
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
|
||||
pub enum State {
|
||||
Open,
|
||||
Invited,
|
||||
Declined,
|
||||
}
|
||||
|
||||
impl AggregateKind for State {
|
||||
const AGGREGATE_TYPE: &'static str = "applicant";
|
||||
const INITIAL_STATE: Self = State::Open;
|
||||
|
||||
fn from_event_type(event_type: &str) -> Option<Self> {
|
||||
match event_type {
|
||||
"applied" => Some(State::Open),
|
||||
"invited" => Some(State::Invited),
|
||||
"declined" => Some(State::Declined),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn event_type(self) -> &'static str {
|
||||
match self {
|
||||
State::Open => "applied",
|
||||
State::Invited => "invited",
|
||||
State::Declined => "declined",
|
||||
}
|
||||
}
|
||||
|
||||
fn allowed(self) -> &'static [Self] {
|
||||
match self {
|
||||
State::Open => &[State::Invited, State::Declined],
|
||||
State::Invited | State::Declined => &[],
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,294 @@
|
||||
//! Generic pure-replay + CAS-write machinery shared by every aggregate
|
||||
//! type (`applicant`, `subscriber`, `project`) - one small `AggregateKind`
|
||||
//! impl per type supplies its own states/transition table/event-type
|
||||
//! names, this module supplies the mechanics once. Mirrors dodrenett's
|
||||
//! `lib/machines/*.ts` shape (a `Record<State, State[]>` table + a pure
|
||||
//! `replay()`), but generalized across types instead of copy-pasted per
|
||||
//! type, and with real optimistic concurrency on write (dodrenett had
|
||||
//! none - see `crate::events::store`).
|
||||
#![cfg(feature = "ssr")]
|
||||
|
||||
pub mod applicant;
|
||||
pub mod project;
|
||||
pub mod subscriber;
|
||||
|
||||
use crate::events::store::{append_event, load_events, Event};
|
||||
|
||||
/// A state machine's shape: its states, its transition table, and how
|
||||
/// states map to/from the event-log's `event_type` strings. Kept as a
|
||||
/// trait (not a generic enum param alone) so each aggregate module can
|
||||
/// own its own `State` enum and transition table as plain, readable
|
||||
/// Rust, not a shared data structure everyone has to thread through.
|
||||
pub trait AggregateKind: Copy + Clone + PartialEq + Eq + std::fmt::Debug + 'static {
|
||||
/// The bucket-name -> aggregate-type key used both for `events.rs`'s
|
||||
/// subject scheme and for `content.rs`'s load-time transition
|
||||
/// validation (see `content::aggregate_type_for_bucket`).
|
||||
const AGGREGATE_TYPE: &'static str;
|
||||
/// The state a brand-new aggregate starts in - also the state a
|
||||
/// creation event (see `create`) asserts.
|
||||
const INITIAL_STATE: Self;
|
||||
|
||||
/// Maps an event-log `event_type` string to the state it represents
|
||||
/// - `None` for anything this aggregate type doesn't recognize
|
||||
/// (replay skips it, never errors).
|
||||
fn from_event_type(event_type: &str) -> Option<Self>;
|
||||
/// The event-log `event_type` string that represents transitioning
|
||||
/// *into* this state.
|
||||
fn event_type(self) -> &'static str;
|
||||
/// States reachable directly from this one. Checked at write time
|
||||
/// (`transition`); replay is deliberately more lenient (see below).
|
||||
fn allowed(self) -> &'static [Self];
|
||||
}
|
||||
|
||||
/// One aggregate's current view, replayed from its event history.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Aggregate<K> {
|
||||
pub id: String,
|
||||
pub state: K,
|
||||
pub payload: serde_json::Value,
|
||||
pub sequence: u64,
|
||||
}
|
||||
|
||||
/// Pure, no IO, never panics. An event that doesn't fit the transition
|
||||
/// table from wherever replay has gotten to so far is silently skipped
|
||||
/// - not an error, not even the *first* recognized event has anything
|
||||
/// to be "from" yet, so it's always accepted. This lenience (not
|
||||
/// enforcing the transition table during replay, only at write time -
|
||||
/// see `transition` below) is deliberate and matches dodrenett's proven
|
||||
/// "self-healing against malformed history" policy: a corrupted or
|
||||
/// out-of-order event never crashes a read, it's just ignored.
|
||||
pub fn replay<K: AggregateKind>(id: &str, events: &[Event]) -> Option<Aggregate<K>> {
|
||||
let mut current: Option<Aggregate<K>> = None;
|
||||
for event in events {
|
||||
let Some(next) = K::from_event_type(&event.event_type) else {
|
||||
continue;
|
||||
};
|
||||
let accept = match ¤t {
|
||||
None => true,
|
||||
Some(agg) => next == agg.state || agg.state.allowed().contains(&next),
|
||||
};
|
||||
if !accept {
|
||||
continue;
|
||||
}
|
||||
current = Some(Aggregate {
|
||||
id: id.to_string(),
|
||||
state: next,
|
||||
payload: event.payload.clone(),
|
||||
sequence: event.sequence,
|
||||
});
|
||||
}
|
||||
current
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum TransitionError {
|
||||
/// No creation event found for this id - nothing to transition.
|
||||
UnknownAggregate,
|
||||
DisallowedTransition { from: String, to: String },
|
||||
/// A concurrent writer already advanced this aggregate past the
|
||||
/// sequence this call read - the fix for the lost-update race
|
||||
/// `answers::transition_answer` used to have (see
|
||||
/// `crate::events::store::append_event`'s CAS).
|
||||
Conflict,
|
||||
Store(String),
|
||||
}
|
||||
|
||||
impl std::fmt::Display for TransitionError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
Self::UnknownAggregate => write!(f, "unknown item"),
|
||||
Self::DisallowedTransition { from, to } => {
|
||||
write!(f, "cannot go from {from} to {to}")
|
||||
}
|
||||
Self::Conflict => write!(f, "item was just changed by someone else, try again"),
|
||||
Self::Store(e) => write!(f, "storage error: {e}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn classify_publish_err(e: async_nats::jetstream::context::PublishError) -> TransitionError {
|
||||
use async_nats::jetstream::context::PublishErrorKind;
|
||||
match e.kind() {
|
||||
PublishErrorKind::WrongLastSequence | PublishErrorKind::WrongLastMessageId => {
|
||||
TransitionError::Conflict
|
||||
}
|
||||
_ => TransitionError::Store(e.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Appends the creation event for a brand-new aggregate. `expected_seq:
|
||||
/// Some(0)` (no message yet on this id's subject) doubles as a
|
||||
/// create-only-once guard - a double-submit race on the same id fails
|
||||
/// the same way a disallowed transition does, rather than silently
|
||||
/// overwriting.
|
||||
pub async fn create<K: AggregateKind>(
|
||||
js: &async_nats::jetstream::Context,
|
||||
id: &str,
|
||||
payload: serde_json::Value,
|
||||
occurred_at_ms: i64,
|
||||
) -> Result<Aggregate<K>, TransitionError> {
|
||||
let ack = append_event(
|
||||
js,
|
||||
K::AGGREGATE_TYPE,
|
||||
id,
|
||||
K::INITIAL_STATE.event_type(),
|
||||
payload.clone(),
|
||||
occurred_at_ms,
|
||||
Some(0),
|
||||
)
|
||||
.await
|
||||
.map_err(classify_publish_err)?;
|
||||
Ok(Aggregate {
|
||||
id: id.to_string(),
|
||||
state: K::INITIAL_STATE,
|
||||
payload,
|
||||
sequence: ack.sequence,
|
||||
})
|
||||
}
|
||||
|
||||
/// Loads and replays `id`'s current state, then - if `target` is
|
||||
/// actually reachable from it - appends the corresponding event with
|
||||
/// the aggregate's just-replayed sequence as the CAS guard.
|
||||
pub async fn transition<K: AggregateKind>(
|
||||
js: &async_nats::jetstream::Context,
|
||||
id: &str,
|
||||
target: K,
|
||||
payload: serde_json::Value,
|
||||
occurred_at_ms: i64,
|
||||
) -> Result<Aggregate<K>, TransitionError> {
|
||||
let events = load_events(js, K::AGGREGATE_TYPE, id)
|
||||
.await
|
||||
.map_err(|e| TransitionError::Store(e.to_string()))?;
|
||||
let current = replay::<K>(id, &events).ok_or(TransitionError::UnknownAggregate)?;
|
||||
if !current.state.allowed().contains(&target) {
|
||||
return Err(TransitionError::DisallowedTransition {
|
||||
from: format!("{:?}", current.state),
|
||||
to: format!("{target:?}"),
|
||||
});
|
||||
}
|
||||
let ack = append_event(
|
||||
js,
|
||||
K::AGGREGATE_TYPE,
|
||||
id,
|
||||
target.event_type(),
|
||||
payload.clone(),
|
||||
occurred_at_ms,
|
||||
Some(current.sequence),
|
||||
)
|
||||
.await
|
||||
.map_err(classify_publish_err)?;
|
||||
Ok(Aggregate {
|
||||
id: id.to_string(),
|
||||
state: target,
|
||||
payload,
|
||||
sequence: ack.sequence,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::events::store::Event;
|
||||
|
||||
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
|
||||
enum TestState {
|
||||
Open,
|
||||
A,
|
||||
B,
|
||||
}
|
||||
|
||||
impl AggregateKind for TestState {
|
||||
const AGGREGATE_TYPE: &'static str = "test";
|
||||
const INITIAL_STATE: Self = TestState::Open;
|
||||
|
||||
fn from_event_type(event_type: &str) -> Option<Self> {
|
||||
match event_type {
|
||||
"opened" => Some(TestState::Open),
|
||||
"a" => Some(TestState::A),
|
||||
"b" => Some(TestState::B),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
fn event_type(self) -> &'static str {
|
||||
match self {
|
||||
TestState::Open => "opened",
|
||||
TestState::A => "a",
|
||||
TestState::B => "b",
|
||||
}
|
||||
}
|
||||
fn allowed(self) -> &'static [Self] {
|
||||
match self {
|
||||
TestState::Open => &[TestState::A, TestState::B],
|
||||
TestState::A | TestState::B => &[],
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn event(seq: u64, event_type: &str) -> Event {
|
||||
Event {
|
||||
id: uuid::Uuid::new_v4(),
|
||||
aggregate_type: "test".to_string(),
|
||||
aggregate_id: "x".to_string(),
|
||||
event_type: event_type.to_string(),
|
||||
payload: serde_json::json!({}),
|
||||
occurred_at_ms: 0,
|
||||
sequence: seq,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_history_is_no_aggregate() {
|
||||
assert!(replay::<TestState>("x", &[]).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn valid_path_replays() {
|
||||
let events = vec![event(1, "opened"), event(2, "a")];
|
||||
let agg = replay::<TestState>("x", &events).unwrap();
|
||||
assert_eq!(agg.state, TestState::A);
|
||||
assert_eq!(agg.sequence, 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn disallowed_edge_is_skipped_not_fatal() {
|
||||
// "a" then "b" - B isn't reachable from A, so the second event
|
||||
// is silently ignored, not an error.
|
||||
let events = vec![event(1, "opened"), event(2, "a"), event(3, "b")];
|
||||
let agg = replay::<TestState>("x", &events).unwrap();
|
||||
assert_eq!(agg.state, TestState::A);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unrecognized_event_type_is_skipped() {
|
||||
let events = vec![event(1, "opened"), event(2, "bogus"), event(3, "a")];
|
||||
let agg = replay::<TestState>("x", &events).unwrap();
|
||||
assert_eq!(agg.state, TestState::A);
|
||||
}
|
||||
|
||||
proptest::proptest! {
|
||||
#[test]
|
||||
fn replay_never_panics_and_state_always_valid(
|
||||
types in proptest::collection::vec(
|
||||
proptest::sample::select(vec!["opened", "a", "b", "bogus"]),
|
||||
0..30,
|
||||
)
|
||||
) {
|
||||
let events: Vec<Event> = types
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, t)| event(i as u64, t))
|
||||
.collect();
|
||||
let result = replay::<TestState>("x", &events);
|
||||
if let Some(agg) = result {
|
||||
// Whatever state we landed in must itself be a real
|
||||
// state this machine declares - trivially true here
|
||||
// since TestState::allowed() only ever returns TestState
|
||||
// values, but this is the shape the real aggregates'
|
||||
// proptests repeat: replaying arbitrary event-type noise
|
||||
// must never produce anything but a legitimate state.
|
||||
let _: TestState = agg.state;
|
||||
assert!(agg.sequence <= events.len() as u64);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
//! Project submissions (`record_as: projects` - absorbs what used to be
|
||||
//! called "inquiries"). A visitor submits a project idea plus who it's
|
||||
//! for (org/contact details, captured as plain payload fields - no
|
||||
//! separate Organization aggregate yet, see the event-sourcing plan).
|
||||
//! `Open -> Accepted | Declined` - a real b2b-pipeline pair of terminal
|
||||
//! states rather than the old generic `Handled`.
|
||||
use super::AggregateKind;
|
||||
|
||||
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
|
||||
pub enum State {
|
||||
Open,
|
||||
Accepted,
|
||||
Declined,
|
||||
}
|
||||
|
||||
impl AggregateKind for State {
|
||||
const AGGREGATE_TYPE: &'static str = "project";
|
||||
const INITIAL_STATE: Self = State::Open;
|
||||
|
||||
fn from_event_type(event_type: &str) -> Option<Self> {
|
||||
match event_type {
|
||||
"submitted" => Some(State::Open),
|
||||
"accepted" => Some(State::Accepted),
|
||||
"declined" => Some(State::Declined),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn event_type(self) -> &'static str {
|
||||
match self {
|
||||
State::Open => "submitted",
|
||||
State::Accepted => "accepted",
|
||||
State::Declined => "declined",
|
||||
}
|
||||
}
|
||||
|
||||
fn allowed(self) -> &'static [Self] {
|
||||
match self {
|
||||
State::Open => &[State::Accepted, State::Declined],
|
||||
State::Accepted | State::Declined => &[],
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
//! Newsletter subscribers (`record_as: subscribers`). `Open ->
|
||||
//! Unsubscribed`, fired via `self_transition` (see `answers.rs`) rather
|
||||
//! than an owner-gated `transition`, so a subscriber never needs a
|
||||
//! signed-in session to leave.
|
||||
use super::AggregateKind;
|
||||
|
||||
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
|
||||
pub enum State {
|
||||
Open,
|
||||
Unsubscribed,
|
||||
}
|
||||
|
||||
impl AggregateKind for State {
|
||||
const AGGREGATE_TYPE: &'static str = "subscriber";
|
||||
const INITIAL_STATE: Self = State::Open;
|
||||
|
||||
fn from_event_type(event_type: &str) -> Option<Self> {
|
||||
match event_type {
|
||||
"subscribed" => Some(State::Open),
|
||||
"unsubscribed" => Some(State::Unsubscribed),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn event_type(self) -> &'static str {
|
||||
match self {
|
||||
State::Open => "subscribed",
|
||||
State::Unsubscribed => "unsubscribed",
|
||||
}
|
||||
}
|
||||
|
||||
fn allowed(self) -> &'static [Self] {
|
||||
match self {
|
||||
State::Open => &[State::Unsubscribed],
|
||||
State::Unsubscribed => &[],
|
||||
}
|
||||
}
|
||||
}
|
||||
+157
-20
@@ -1,15 +1,15 @@
|
||||
//! Durable, queryable storage for submitted answers - the KV-backed
|
||||
//! counterpart to the ephemeral `events::AnswerSubmitted` every
|
||||
//! submission already publishes. The runtime here has no notion of
|
||||
//! "applicant" or any other specific workflow: an `Alternative.record_as`
|
||||
//! names a NATS KV bucket (nothing more); on submit, the response gets
|
||||
//! wrapped in the same envelope (`Answer`) regardless of what content
|
||||
//! produced it. `resource::get_resource` reads it back generically, and
|
||||
//! `transition_answer` below moves an answer between states named in
|
||||
//! content (`ResourceSpec.transitions`), never states hardcoded here.
|
||||
//! Whatever workflow this ends up powering - applicant review, order
|
||||
//! triage, comment moderation - is entirely a property of some
|
||||
//! `portal-content` YAML, not of this file.
|
||||
//! Read-model storage for submitted answers - the KV-backed projection
|
||||
//! of the durable event log now underneath it (`events::store`,
|
||||
//! `aggregates`) for the three buckets migrated onto that machinery
|
||||
//! this pass (`content::aggregate_type_for_bucket`). An
|
||||
//! `Alternative.record_as`/`ResourceSpec`'s bucket is still just a
|
||||
//! bucket name as far as `resource::get_resource` is concerned - this
|
||||
//! file is where that name additionally gets checked against a real,
|
||||
//! Rust-defined transition table, for the buckets that have one.
|
||||
//! A bucket outside that three-entry map still works exactly as
|
||||
//! before (direct KV mutate-in-place, no event log, no CAS) - this is
|
||||
//! deliberately not a hard cutover, so content isn't forced to migrate
|
||||
//! before it needs to.
|
||||
|
||||
use leptos::prelude::*;
|
||||
use serde::{Deserialize, Serialize};
|
||||
@@ -71,7 +71,30 @@ pub async fn store_answer(
|
||||
decided_by: None,
|
||||
};
|
||||
let data = serde_json::to_vec(&answer)?;
|
||||
store.put(id, data.into()).await?;
|
||||
store.put(id.clone(), data.into()).await?;
|
||||
|
||||
// Also seed the event log for the three aggregate types migrated
|
||||
// onto it this pass - best-effort, same "a hiccup here shouldn't
|
||||
// fail a submission the NATS notification event has already
|
||||
// recorded" policy as the KV write above.
|
||||
if let Some(agg_type) = crate::content::aggregate_type_for_bucket(bucket) {
|
||||
use crate::aggregates::{applicant, create, project, subscriber};
|
||||
let seed = match agg_type {
|
||||
"applicant" => create::<applicant::State>(js, &id, responses.clone(), submitted_ms)
|
||||
.await
|
||||
.map(|_| ()),
|
||||
"subscriber" => create::<subscriber::State>(js, &id, responses.clone(), submitted_ms)
|
||||
.await
|
||||
.map(|_| ()),
|
||||
"project" => create::<project::State>(js, &id, responses.clone(), submitted_ms)
|
||||
.await
|
||||
.map(|_| ()),
|
||||
_ => Ok(()),
|
||||
};
|
||||
if let Err(e) = seed {
|
||||
tracing::error!(bucket, id, error = %e, "failed to seed event log for submission");
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -138,9 +161,12 @@ pub async fn transition_answer(
|
||||
return Err(ServerFnError::new("not authorized"));
|
||||
}
|
||||
|
||||
let bucket = resource
|
||||
.bucket()
|
||||
.ok_or_else(|| ServerFnError::new("resource is not kv-backed, has no transitions"))?;
|
||||
let store = state
|
||||
.jetstream
|
||||
.get_key_value(&resource.bucket)
|
||||
.get_key_value(bucket)
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(format!("resource bucket unavailable: {e}")))?;
|
||||
let bytes = store
|
||||
@@ -155,6 +181,22 @@ pub async fn transition_answer(
|
||||
}
|
||||
|
||||
let decided_ms = chrono::Utc::now().timestamp_millis();
|
||||
let decision_payload_for_event =
|
||||
serde_json::json!({ "to": to, "item": item_id, "by": user.username });
|
||||
|
||||
// For the three buckets migrated onto the event log this pass, the
|
||||
// real fix for the lost-update race two concurrent decisions on the
|
||||
// same item used to hit: append with CAS on the aggregate's
|
||||
// just-replayed sequence, so a second racing caller's write is
|
||||
// rejected instead of silently overwriting the first. A bucket
|
||||
// outside that map falls back to the direct KV mutate this always
|
||||
// did - not every resource has to be migrated to keep working.
|
||||
if let Some(agg_type) = crate::content::aggregate_type_for_bucket(bucket) {
|
||||
transition_by_aggregate(&state.jetstream, agg_type, &item_id, &to, decision_payload_for_event.clone(), decided_ms)
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?;
|
||||
}
|
||||
|
||||
answer.state = to.clone();
|
||||
answer.decided_ms = Some(decided_ms);
|
||||
answer.decided_by = Some(user.username.clone());
|
||||
@@ -169,15 +211,14 @@ pub async fn transition_answer(
|
||||
// submission hash, published the same way any other answer is -
|
||||
// question_id/alternative come from this call's own arguments and
|
||||
// the content-declared transition label, never a hardcoded value.
|
||||
let decision_payload = serde_json::json!({ "to": to, "item": item_id, "by": user.username });
|
||||
let parent_hashes = vec![item_id.clone()];
|
||||
let chain_hash = hash_node(&question_id, &parent_hashes, &decision_payload, decided_ms);
|
||||
let chain_hash = hash_node(&question_id, &parent_hashes, &decision_payload_for_event, decided_ms);
|
||||
let event = AnswerSubmitted {
|
||||
chain_hash,
|
||||
parent_hashes,
|
||||
question_id,
|
||||
alternative: transition.label,
|
||||
responses: decision_payload,
|
||||
responses: decision_payload_for_event,
|
||||
timestamp_ms: decided_ms,
|
||||
};
|
||||
emit_answer_submitted(&state.nats, &event)
|
||||
@@ -187,6 +228,43 @@ pub async fn transition_answer(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Dispatches to the right aggregate type's `transition()` by name -
|
||||
/// the one place `answers.rs` has to know the fixed, three-entry set of
|
||||
/// migrated aggregate types (`content::aggregate_type_for_bucket`'s
|
||||
/// counterpart), since `crate::aggregates` itself stays generic over
|
||||
/// any `AggregateKind` and has no list of its own to dispatch through.
|
||||
#[cfg(feature = "ssr")]
|
||||
async fn transition_by_aggregate(
|
||||
js: &async_nats::jetstream::Context,
|
||||
aggregate_type: &str,
|
||||
item_id: &str,
|
||||
to: &str,
|
||||
payload: serde_json::Value,
|
||||
occurred_at_ms: i64,
|
||||
) -> Result<(), crate::aggregates::TransitionError> {
|
||||
use crate::aggregates::{applicant, project, subscriber, transition, AggregateKind, TransitionError};
|
||||
|
||||
macro_rules! dispatch {
|
||||
($state:ty) => {{
|
||||
let target = <$state>::from_event_type(to).ok_or_else(|| {
|
||||
TransitionError::DisallowedTransition {
|
||||
from: "?".to_string(),
|
||||
to: to.to_string(),
|
||||
}
|
||||
})?;
|
||||
transition::<$state>(js, item_id, target, payload, occurred_at_ms).await?;
|
||||
}};
|
||||
}
|
||||
|
||||
match aggregate_type {
|
||||
"applicant" => dispatch!(applicant::State),
|
||||
"subscriber" => dispatch!(subscriber::State),
|
||||
"project" => dispatch!(project::State),
|
||||
_ => return Err(TransitionError::UnknownAggregate),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The self-service counterpart to `transition_answer`: no signed-in
|
||||
/// session, no group check - authorized instead by already holding
|
||||
/// `item_id` (a chain hash, opaque and unguessable) plus a matching
|
||||
@@ -244,6 +322,22 @@ pub async fn self_transition_answer(
|
||||
}
|
||||
|
||||
let decided_ms = chrono::Utc::now().timestamp_millis();
|
||||
let decision_payload_for_event =
|
||||
serde_json::json!({ "to": self_transition.to, "item": item_id, "by": email });
|
||||
|
||||
if let Some(agg_type) = crate::content::aggregate_type_for_bucket(&self_transition.bucket) {
|
||||
transition_by_aggregate(
|
||||
&state.jetstream,
|
||||
agg_type,
|
||||
&item_id,
|
||||
&self_transition.to,
|
||||
decision_payload_for_event.clone(),
|
||||
decided_ms,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?;
|
||||
}
|
||||
|
||||
answer.state = self_transition.to.clone();
|
||||
answer.decided_ms = Some(decided_ms);
|
||||
answer.decided_by = Some(email.clone());
|
||||
@@ -254,15 +348,14 @@ pub async fn self_transition_answer(
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?;
|
||||
|
||||
let decision_payload = serde_json::json!({ "to": self_transition.to, "item": item_id, "by": email });
|
||||
let parent_hashes = vec![item_id.clone()];
|
||||
let chain_hash = hash_node(&question_id, &parent_hashes, &decision_payload, decided_ms);
|
||||
let chain_hash = hash_node(&question_id, &parent_hashes, &decision_payload_for_event, decided_ms);
|
||||
let event = AnswerSubmitted {
|
||||
chain_hash,
|
||||
parent_hashes,
|
||||
question_id,
|
||||
alternative: self_transition.label,
|
||||
responses: decision_payload,
|
||||
responses: decision_payload_for_event,
|
||||
timestamp_ms: decided_ms,
|
||||
};
|
||||
emit_answer_submitted(&state.nats, &event)
|
||||
@@ -271,3 +364,47 @@ pub async fn self_transition_answer(
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// `app.rs`'s list-rendering path only shows the nice
|
||||
/// button-per-transition UI if a resource's raw JSON value
|
||||
/// deserializes as `Vec<Answer>` - otherwise it silently falls
|
||||
/// back to a plain read-only dump with no error. Guards that shape
|
||||
/// contract directly, since nothing else would catch a drift here.
|
||||
#[test]
|
||||
fn answer_list_round_trips_through_the_shape_app_rs_expects() {
|
||||
let answers = vec![
|
||||
Answer {
|
||||
id: "abc123".to_string(),
|
||||
question_id: "/review".to_string(),
|
||||
alternative: "Applicants".to_string(),
|
||||
responses: serde_json::json!({"name": "Ada"}),
|
||||
state: OPEN_STATE.to_string(),
|
||||
submitted_ms: 1_700_000_000_000,
|
||||
decided_ms: None,
|
||||
decided_by: None,
|
||||
},
|
||||
Answer {
|
||||
id: "def456".to_string(),
|
||||
question_id: "/review".to_string(),
|
||||
alternative: "Applicants".to_string(),
|
||||
responses: serde_json::json!({"name": "Grace"}),
|
||||
state: "invited".to_string(),
|
||||
submitted_ms: 1_700_000_001_000,
|
||||
decided_ms: Some(1_700_000_002_000),
|
||||
decided_by: Some("bl".to_string()),
|
||||
},
|
||||
];
|
||||
let value = serde_json::Value::Array(
|
||||
answers.iter().map(|a| serde_json::to_value(a).unwrap()).collect(),
|
||||
);
|
||||
let round_tripped: Vec<Answer> =
|
||||
serde_json::from_value(value).expect("Answer list round-trips");
|
||||
assert_eq!(round_tripped.len(), 2);
|
||||
assert_eq!(round_tripped[1].state, "invited");
|
||||
assert_eq!(round_tripped[1].decided_by.as_deref(), Some("bl"));
|
||||
}
|
||||
}
|
||||
|
||||
+91
-2
@@ -9,7 +9,7 @@ use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::answers::{Answer, SelfTransitionAnswer, TransitionAnswer};
|
||||
use crate::auth::{current_user, User};
|
||||
use crate::content::{is_qualified, Alternative, Question, Transition};
|
||||
use crate::content::{is_qualified, Alternative, Question, Responsible, Transition};
|
||||
use crate::resource::get_resource;
|
||||
|
||||
/// The visible site name/wordmark - "portal" is just this codebase's
|
||||
@@ -157,6 +157,7 @@ fn QuestionView(
|
||||
.into_any();
|
||||
}
|
||||
|
||||
let question_id_for_note = question_id.clone();
|
||||
view! {
|
||||
<Hero
|
||||
title=question.name.clone()
|
||||
@@ -197,10 +198,64 @@ fn QuestionView(
|
||||
}
|
||||
/>
|
||||
</div>
|
||||
{question
|
||||
.responsible
|
||||
.clone()
|
||||
.map(|r| {
|
||||
view! { <ResponsibleNote question_id=question_id_for_note.clone() responsible=r/> }
|
||||
})}
|
||||
}
|
||||
.into_any()
|
||||
}
|
||||
|
||||
/// "Asked by X — contact them if you get stuck", plus a one-click
|
||||
/// "report this question" action (see `report_question`) - mainly
|
||||
/// meant for a question that reads as unhelpfully LLM-generated, so
|
||||
/// whoever's responsible for it hears about it.
|
||||
#[component]
|
||||
fn ResponsibleNote(question_id: String, responsible: Responsible) -> impl IntoView {
|
||||
let report = ServerAction::<ReportQuestion>::new();
|
||||
let reported = RwSignal::new(false);
|
||||
Effect::new(move |_| {
|
||||
if report.value().get().is_some_and(|r| r.is_ok()) {
|
||||
reported.set(true);
|
||||
}
|
||||
});
|
||||
|
||||
view! {
|
||||
<p class="question-responsible">
|
||||
"Asked by " {responsible.name.clone()} " — "
|
||||
<a href=format!("mailto:{}", responsible.contact)>"contact them"</a>
|
||||
" if you get stuck."
|
||||
{move || {
|
||||
if reported.get() {
|
||||
view! { <span class="question-reported">" Reported — thanks."</span> }
|
||||
.into_any()
|
||||
} else {
|
||||
let question_id = question_id.clone();
|
||||
view! {
|
||||
<button
|
||||
type="button"
|
||||
class="question-report"
|
||||
disabled=move || report.pending().get()
|
||||
on:click=move |_| {
|
||||
report
|
||||
.dispatch(ReportQuestion {
|
||||
question_id: question_id.clone(),
|
||||
reason: None,
|
||||
});
|
||||
}
|
||||
>
|
||||
" Not helpful? Report this question."
|
||||
</button>
|
||||
}
|
||||
.into_any()
|
||||
}
|
||||
}}
|
||||
</p>
|
||||
}
|
||||
}
|
||||
|
||||
// Binds the actual "YES - Rasterized Lines" piece live at uhhm.no
|
||||
// (ported to `public/yes.js`, an ES module now instead of a page-owning
|
||||
// script) as a typed JS class - the idiomatic wasm-bindgen way to drive
|
||||
@@ -691,7 +746,11 @@ fn ResourceFeature(
|
||||
let feature_name = feature_name.clone();
|
||||
move || (question_id.clone(), alternative.clone(), feature_name.clone())
|
||||
},
|
||||
|(q, a, f)| get_resource(q, a, f),
|
||||
// Empty params for now - forwarding sibling requirement values
|
||||
// as query params (for a resource parameterized by other form
|
||||
// fields) is deferred, see the resource-backed multi-select
|
||||
// requirement UI follow-up.
|
||||
|(q, a, f)| get_resource(q, a, f, std::collections::HashMap::new()),
|
||||
);
|
||||
|
||||
view! {
|
||||
@@ -996,6 +1055,36 @@ pub async fn submit_answer(
|
||||
Ok(SubmitResult { next, chain_hash })
|
||||
}
|
||||
|
||||
/// Fire-and-forget "this question wasn't helpful" report - a plain log
|
||||
/// entry in the event store (`aggregate_type: "question_report"`, no
|
||||
/// state machine, nothing to transition), reviewable via
|
||||
/// `events::store::load_events(js, "question_report", question_id)` the
|
||||
/// same way any other aggregate's history is. No auth, same as reading
|
||||
/// a public question - a report is cheap, low-stakes signal, not
|
||||
/// something worth gating behind a session.
|
||||
#[server]
|
||||
pub async fn report_question(
|
||||
question_id: String,
|
||||
reason: Option<String>,
|
||||
) -> Result<(), ServerFnError> {
|
||||
use crate::server::AppState;
|
||||
|
||||
let state = expect_context::<AppState>();
|
||||
let occurred_at_ms = chrono::Utc::now().timestamp_millis();
|
||||
crate::events::store::append_event(
|
||||
&state.jetstream,
|
||||
"question_report",
|
||||
&question_id,
|
||||
"reported",
|
||||
serde_json::json!({ "reason": reason }),
|
||||
occurred_at_ms,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Uploads one file to `POST /upload` and returns its stored object key
|
||||
/// - the value a `type: file` requirement contributes to
|
||||
/// `responses_json`, same as any other field. A plain `fetch`, not a
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
//! One-time migration: for each of the three renamed buckets
|
||||
//! (`portal_applicants` -> `applicants`, `portal_subscribers` ->
|
||||
//! `subscribers`, `portal_project_inquiries` -> `projects`), seeds the
|
||||
//! new event log with a creation event (plus a decision event for any
|
||||
//! record already decided) and re-puts the record into the new-named
|
||||
//! KV bucket - then immediately replays what it just wrote and checks
|
||||
//! it reproduces the original record's state exactly, per the
|
||||
//! event-sourcing plan's verification step.
|
||||
//!
|
||||
//! Defaults to a dry run (lists what it *would* do, writes nothing) -
|
||||
//! pass `--apply` to actually write. The old-named buckets are never
|
||||
//! touched or deleted by this tool; remove them by hand once every
|
||||
//! record's replay has been confirmed to match.
|
||||
#![cfg(feature = "ssr")]
|
||||
|
||||
use portal::aggregates::{self, applicant, project, subscriber, AggregateKind};
|
||||
use portal::answers::{Answer, OPEN_STATE};
|
||||
use portal::events::store as event_store;
|
||||
|
||||
const MIGRATIONS: &[(&str, &str, &str)] = &[
|
||||
("portal_applicants", "applicants", "applicant"),
|
||||
("portal_subscribers", "subscribers", "subscriber"),
|
||||
("portal_project_inquiries", "projects", "project"),
|
||||
];
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> anyhow::Result<()> {
|
||||
let apply = std::env::args().any(|a| a == "--apply");
|
||||
|
||||
let nats_url =
|
||||
std::env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".to_string());
|
||||
let parsed = url::Url::parse(&nats_url)?;
|
||||
let mut opts = async_nats::ConnectOptions::new();
|
||||
if !parsed.username().is_empty() {
|
||||
opts = opts.user_and_password(
|
||||
parsed.username().to_string(),
|
||||
parsed.password().unwrap_or_default().to_string(),
|
||||
);
|
||||
}
|
||||
let nats = opts.connect(&nats_url).await?;
|
||||
let js = async_nats::jetstream::new(nats);
|
||||
event_store::ensure_stream(&js).await?;
|
||||
|
||||
println!("mode: {}\n", if apply { "APPLY (writing)" } else { "DRY RUN (no writes)" });
|
||||
|
||||
for (old_bucket, new_bucket, agg_type) in MIGRATIONS {
|
||||
println!("=== {old_bucket} -> {new_bucket} ({agg_type}) ===");
|
||||
let store = match js.get_key_value(*old_bucket).await {
|
||||
Ok(s) => s,
|
||||
Err(e) => {
|
||||
println!(" skip: bucket unavailable: {e}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
use futures::TryStreamExt;
|
||||
let keys: Vec<String> = store.keys().await?.try_collect().await?;
|
||||
if keys.is_empty() {
|
||||
println!(" (empty)");
|
||||
}
|
||||
for key in keys {
|
||||
let Some(bytes) = store.get(&key).await? else { continue };
|
||||
let answer: Answer = serde_json::from_slice(&bytes)?;
|
||||
println!(" {key}: state={:?} submitted_ms={}", answer.state, answer.submitted_ms);
|
||||
if apply {
|
||||
if let Err(e) = seed_and_verify(&js, agg_type, &key, &answer, new_bucket).await {
|
||||
println!(" ! {e}");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if !apply {
|
||||
println!("\nDry run only - no writes made. Re-run with --apply to actually backfill.");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn seed_and_verify(
|
||||
js: &async_nats::jetstream::Context,
|
||||
agg_type: &str,
|
||||
id: &str,
|
||||
answer: &Answer,
|
||||
new_bucket: &str,
|
||||
) -> anyhow::Result<()> {
|
||||
match agg_type {
|
||||
"applicant" => seed::<applicant::State>(js, id, answer).await?,
|
||||
"subscriber" => seed::<subscriber::State>(js, id, answer).await?,
|
||||
"project" => seed::<project::State>(js, id, answer).await?,
|
||||
other => anyhow::bail!("unknown aggregate type {other}"),
|
||||
}
|
||||
|
||||
let new_store = match js.get_key_value(new_bucket).await {
|
||||
Ok(s) => s,
|
||||
Err(_) => {
|
||||
js.create_key_value(async_nats::jetstream::kv::Config {
|
||||
bucket: new_bucket.to_string(),
|
||||
..Default::default()
|
||||
})
|
||||
.await?
|
||||
}
|
||||
};
|
||||
new_store.put(id, serde_json::to_vec(answer)?.into()).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Appends the initial event, then - if the original record was
|
||||
/// already decided - a second event carrying the real decision, before
|
||||
/// replaying and checking the result matches. Two events for a decided
|
||||
/// record, not one: a single "imported" event would show a decided
|
||||
/// record as forever open on replay.
|
||||
async fn seed<K: AggregateKind>(
|
||||
js: &async_nats::jetstream::Context,
|
||||
id: &str,
|
||||
answer: &Answer,
|
||||
) -> anyhow::Result<()> {
|
||||
aggregates::create::<K>(js, id, answer.responses.clone(), answer.submitted_ms)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("seeding initial event: {e}"))?;
|
||||
|
||||
if answer.state != OPEN_STATE {
|
||||
match K::from_event_type(&answer.state) {
|
||||
Some(target) => {
|
||||
let payload = serde_json::json!({
|
||||
"to": answer.state,
|
||||
"item": id,
|
||||
"by": answer.decided_by,
|
||||
});
|
||||
let occurred = answer.decided_ms.unwrap_or(answer.submitted_ms);
|
||||
aggregates::transition::<K>(js, id, target, payload, occurred)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("seeding decision event: {e}"))?;
|
||||
}
|
||||
None => println!(" ! unrecognized decided state {:?}, only seeded initial event", answer.state),
|
||||
}
|
||||
}
|
||||
|
||||
let events = event_store::load_events(js, K::AGGREGATE_TYPE, id).await?;
|
||||
match aggregates::replay::<K>(id, &events) {
|
||||
Some(agg) if agg.state.event_type() == answer.state => {
|
||||
println!(" OK: replay matches original state {:?}", answer.state);
|
||||
}
|
||||
Some(agg) => {
|
||||
println!(" MISMATCH: replayed {:?}, original was {:?}", agg.state, answer.state);
|
||||
}
|
||||
None => println!(" MISMATCH: no aggregate replayed at all"),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
//! Headless schema-check binary - loads content (from a Gitea repo URL
|
||||
//! or a local directory) and validates it exactly the way
|
||||
//! `content::watch_for_reload`/`main.rs`'s boot path do, with no NATS,
|
||||
//! OIDC, web server, or JetStream connection involved. Built once by
|
||||
//! portal's own deploy workflow and downloaded as a release artifact by
|
||||
//! `questions`' own CI, rather than compiled there - keeps that repo's
|
||||
//! CI coupling to "run a static binary," not "build a Rust workspace."
|
||||
#![cfg(feature = "ssr")]
|
||||
|
||||
use portal::content;
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> anyhow::Result<()> {
|
||||
let mut args = std::env::args().skip(1);
|
||||
let mut repo: Option<String> = None;
|
||||
let mut branch = "main".to_string();
|
||||
let mut subdir = "questions".to_string();
|
||||
let mut path: Option<String> = None;
|
||||
|
||||
while let Some(arg) = args.next() {
|
||||
match arg.as_str() {
|
||||
"--repo" => repo = args.next(),
|
||||
"--branch" => branch = args.next().unwrap_or(branch),
|
||||
"--subdir" => subdir = args.next().unwrap_or(subdir),
|
||||
"--path" => path = args.next(),
|
||||
other => {
|
||||
eprintln!("unknown argument: {other}");
|
||||
std::process::exit(2);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let questions = if let Some(dir) = path {
|
||||
load_from_dir(&dir)?
|
||||
} else if let Some(repo_url) = repo {
|
||||
content::load_questions_from_gitea(&repo_url, &branch, &subdir).await?
|
||||
} else {
|
||||
eprintln!(
|
||||
"usage: question-lint --repo <gitea-url> [--branch main] [--subdir questions] | --path <local-dir>"
|
||||
);
|
||||
std::process::exit(2);
|
||||
};
|
||||
|
||||
match content::validate_questions(&questions) {
|
||||
Ok(()) => {
|
||||
println!("OK: {} question(s) valid", questions.len());
|
||||
Ok(())
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("FAIL: {e}");
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The offline counterpart to `content::load_questions_from_gitea` -
|
||||
/// same "every `*.yaml` file becomes a `Question` keyed by its own
|
||||
/// `id`" shape, just reading a local checkout instead of Gitea's API,
|
||||
/// for linting a branch that hasn't been pushed yet.
|
||||
fn load_from_dir(
|
||||
dir: &str,
|
||||
) -> anyhow::Result<std::collections::HashMap<String, content::Question>> {
|
||||
let mut out = std::collections::HashMap::new();
|
||||
for entry in std::fs::read_dir(dir)? {
|
||||
let entry = entry?;
|
||||
let path = entry.path();
|
||||
if path.extension().and_then(|e| e.to_str()) != Some("yaml") {
|
||||
continue;
|
||||
}
|
||||
let raw = std::fs::read_to_string(&path)?;
|
||||
let question: content::Question = serde_yaml::from_str(&raw)
|
||||
.map_err(|e| anyhow::anyhow!("parsing {}: {e}", path.display()))?;
|
||||
out.insert(question.id.clone(), question);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
+151
-3
@@ -21,6 +21,20 @@ pub struct Question {
|
||||
pub qualifies: Option<String>,
|
||||
#[serde(default)]
|
||||
pub alternatives: Vec<Alternative>,
|
||||
/// Who to contact if a visitor gets stuck or finds this question
|
||||
/// unhelpful - rendered as a small line on the page, and the person
|
||||
/// a "report this question" action names in its own event payload
|
||||
/// (see `events::store`, `question_report` aggregate-less events).
|
||||
#[serde(default)]
|
||||
pub responsible: Option<Responsible>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Responsible {
|
||||
pub name: String,
|
||||
/// A mailto address or plain email - deliberately just a string,
|
||||
/// same as everywhere else content keeps contact info simple.
|
||||
pub contact: String,
|
||||
}
|
||||
|
||||
/// Whether `user` may view/submit `question`. `true` when the question
|
||||
@@ -105,8 +119,9 @@ pub struct Feature {
|
||||
/// item.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct ResourceSpec {
|
||||
pub bucket: String,
|
||||
/// A single item; omit to list the whole bucket.
|
||||
pub source: ResourceSource,
|
||||
/// A single item; omit to list the whole bucket. Only meaningful
|
||||
/// for a `Kv` source.
|
||||
#[serde(default)]
|
||||
pub key: Option<String>,
|
||||
/// Kanidm group required to read this resource.
|
||||
@@ -122,9 +137,59 @@ pub struct ResourceSpec {
|
||||
/// States a listed answer may move to, and the button label for
|
||||
/// each - empty means the resource is read-only. The allow-list a
|
||||
/// `transition_answer` call is checked against, so a client can
|
||||
/// never move an answer to a state content didn't declare.
|
||||
/// never move an answer to a state content didn't declare. Only
|
||||
/// meaningful for a `Kv` source - a live external pull is always
|
||||
/// read-only.
|
||||
#[serde(default)]
|
||||
pub transitions: Vec<Transition>,
|
||||
/// A jq filter reshaping the fetched value before it reaches the
|
||||
/// frontend - e.g. `.[] | {name, url: .html_url}` to pick just the
|
||||
/// fields a showcase card needs out of a raw Gitea API response.
|
||||
/// Evaluated via the `jaq` crate (`resource::apply_jq`), no shell
|
||||
/// -out. `None` (or a `Kv` source) returns the fetched value as-is.
|
||||
#[serde(default)]
|
||||
pub jq: Option<String>,
|
||||
}
|
||||
|
||||
/// Where a resource's live data actually comes from. `Kv` is the
|
||||
/// original (and still only mutable) behavior - a NATS KV bucket this
|
||||
/// server itself owns. The `Gitea*` variants are live pulls from the
|
||||
/// same Gitea instance content is loaded from (`AppState.gitea_base`),
|
||||
/// for surfacing things like a portfolio of starred repos without
|
||||
/// hand-curating a KV bucket for it. Not mutually exclusive with
|
||||
/// owner-curated resources - a hand-picked feature list is just a `Kv`
|
||||
/// resource an owner writes into; a `GiteaStarred`/`GiteaOrgRepos` pull
|
||||
/// is the automatic catalog to choose from. Both can appear as
|
||||
/// different features on the same page.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
#[serde(tag = "kind", rename_all = "snake_case")]
|
||||
pub enum ResourceSource {
|
||||
Kv { bucket: String },
|
||||
GiteaStarred { username: String },
|
||||
GiteaOrgRepos { org: String },
|
||||
/// Any other HTTPS JSON endpoint - the general escape hatch for a
|
||||
/// live resource that isn't this Gitea instance. Deliberately
|
||||
/// scheme-restricted and checked against loopback/private/link
|
||||
/// -local addresses at fetch time (`resource::fetch_url_resource`)
|
||||
/// - content is trusted today (same org owns both repos), but a
|
||||
/// server-side fetch of a content-supplied URL is real SSRF surface
|
||||
/// the moment that stops being true, so it fails closed rather than
|
||||
/// trusting every host by default.
|
||||
Url { url: String },
|
||||
}
|
||||
|
||||
impl ResourceSpec {
|
||||
/// The KV bucket this resource reads/writes, if it's `Kv`-sourced -
|
||||
/// `None` for a live external pull, which has no bucket and (see
|
||||
/// `transitions` above) can't be mutated through this mechanism.
|
||||
pub fn bucket(&self) -> Option<&str> {
|
||||
match &self.source {
|
||||
ResourceSource::Kv { bucket } => Some(bucket),
|
||||
ResourceSource::GiteaStarred { .. }
|
||||
| ResourceSource::GiteaOrgRepos { .. }
|
||||
| ResourceSource::Url { .. } => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
@@ -251,6 +316,85 @@ pub async fn load_questions_from_gitea(
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Maps a content-declared KV bucket name to the aggregate type
|
||||
/// (`crate::aggregates::*::State::AGGREGATE_TYPE`) it's event-sourced
|
||||
/// under - the fixed, three-entry table for this pass (see the
|
||||
/// event-sourcing plan). A bucket name not listed here is left alone
|
||||
/// entirely (no validation applied) - deliberate: only buckets actually
|
||||
/// migrated onto the generic aggregate machinery get their transitions
|
||||
/// checked.
|
||||
#[cfg(feature = "ssr")]
|
||||
pub fn aggregate_type_for_bucket(bucket: &str) -> Option<&'static str> {
|
||||
match bucket {
|
||||
"applicants" => Some("applicant"),
|
||||
"subscribers" => Some("subscriber"),
|
||||
"projects" => Some("project"),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "ssr")]
|
||||
fn is_valid_transition_target(aggregate_type: &str, to: &str) -> bool {
|
||||
use crate::aggregates::AggregateKind;
|
||||
fn check<K: AggregateKind>(to: &str) -> bool {
|
||||
K::INITIAL_STATE.allowed().iter().any(|s| s.event_type() == to)
|
||||
}
|
||||
match aggregate_type {
|
||||
"applicant" => check::<crate::aggregates::applicant::State>(to),
|
||||
"subscriber" => check::<crate::aggregates::subscriber::State>(to),
|
||||
"project" => check::<crate::aggregates::project::State>(to),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Validates every declared transition target (`SelfTransition.to`,
|
||||
/// `ResourceSpec.transitions[].to`) against the real compiled
|
||||
/// transition table for its bucket's aggregate type, for every bucket
|
||||
/// this pass actually migrated (see `aggregate_type_for_bucket`).
|
||||
/// Called on every content load/reload (`watch_for_reload`, `main.rs`'s
|
||||
/// boot path) and by the standalone `question-lint` binary, so a YAML
|
||||
/// typo becomes a caught, logged rejection instead of a silently
|
||||
/// -accepted, later-broken string.
|
||||
#[cfg(feature = "ssr")]
|
||||
pub fn validate_questions(
|
||||
questions: &std::collections::HashMap<String, Question>,
|
||||
) -> anyhow::Result<()> {
|
||||
for question in questions.values() {
|
||||
for alternative in &question.alternatives {
|
||||
if let Some(st) = &alternative.self_transition {
|
||||
if let Some(agg) = aggregate_type_for_bucket(&st.bucket) {
|
||||
if !is_valid_transition_target(agg, &st.to) {
|
||||
anyhow::bail!(
|
||||
"question {:?} alternative {:?}: self_transition.to {:?} is not a valid transition for bucket {:?} ({agg})",
|
||||
question.id, alternative.name, st.to, st.bucket
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
for feature in &alternative.features {
|
||||
let Some(resource) = &feature.resource else {
|
||||
continue;
|
||||
};
|
||||
let Some(bucket) = resource.bucket() else {
|
||||
continue;
|
||||
};
|
||||
let Some(agg) = aggregate_type_for_bucket(bucket) else {
|
||||
continue;
|
||||
};
|
||||
for transition in &resource.transitions {
|
||||
if !is_valid_transition_target(agg, &transition.to) {
|
||||
anyhow::bail!(
|
||||
"question {:?} alternative {:?} feature {:?}: transition.to {:?} is not a valid transition for bucket {:?} ({agg})",
|
||||
question.id, alternative.name, feature.name, transition.to, bucket
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Published by the content repo's own CI (after it lints a push) to
|
||||
/// tell every running instance to pick up the change - a plain fire
|
||||
/// and forget NATS publish, no payload, matching `events.rs`'s
|
||||
@@ -283,6 +427,10 @@ pub async fn watch_for_reload(
|
||||
while sub.next().await.is_some() {
|
||||
match load_questions_from_gitea(&repo_url, &branch, &subdir).await {
|
||||
Ok(loaded) => {
|
||||
if let Err(e) = validate_questions(&loaded) {
|
||||
tracing::error!(error = %e, "content reload failed validation, keeping last-good content");
|
||||
continue;
|
||||
}
|
||||
let count = loaded.len();
|
||||
questions.store(std::sync::Arc::new(loaded));
|
||||
tracing::info!(count, "reloaded content");
|
||||
|
||||
@@ -6,6 +6,8 @@
|
||||
//! change shape when that shows up, only gain a second subscriber.
|
||||
#![cfg(feature = "ssr")]
|
||||
|
||||
pub mod store;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
@@ -0,0 +1,150 @@
|
||||
//! Durable, replayable event log for the aggregate state machines in
|
||||
//! `crate::aggregates` - the source of truth `answers::Answer` KV
|
||||
//! records now only *project*. Ported from dodrenett's `lib/eventStore.ts`
|
||||
//! with two gaps closed rather than copied: dodrenett's `sequence`
|
||||
//! field was always hardcoded `0` (ordering relied entirely on
|
||||
//! JetStream's own append order anyway - here the real per-message
|
||||
//! sequence is used directly, nothing fake); dodrenett never used
|
||||
//! `Nats-Msg-Id`/CAS at all. This module uses both, via `async-nats`
|
||||
//! 0.38's native support - no new dependency.
|
||||
//!
|
||||
//! Subject scheme is `events.<aggregate_type>.<aggregate_id>`,
|
||||
//! deliberately *not* `events.<type>.<id>.<eventType>` like dodrenett -
|
||||
//! dropping the event type from the subject means
|
||||
//! `expected_last_subject_sequence` gives real optimistic concurrency
|
||||
//! across an aggregate's *entire* history, not just same-typed events.
|
||||
//! The cost: no cheap subject-wildcard query for "every `invited` event
|
||||
//! across all applicants" - nobody needs that today, and it's a filter
|
||||
//! over `load_events`'s output if it's ever needed.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
pub const STREAM_NAME: &str = "EVENTS";
|
||||
|
||||
/// One durably-stored fact about an aggregate. `sequence` is filled in
|
||||
/// from JetStream's own per-message sequence number on read
|
||||
/// (`load_events`) - never trust a value baked into the payload itself.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Event {
|
||||
pub id: uuid::Uuid,
|
||||
pub aggregate_type: String,
|
||||
pub aggregate_id: String,
|
||||
pub event_type: String,
|
||||
pub payload: serde_json::Value,
|
||||
pub occurred_at_ms: i64,
|
||||
#[serde(default)]
|
||||
pub sequence: u64,
|
||||
}
|
||||
|
||||
fn subject(aggregate_type: &str, aggregate_id: &str) -> String {
|
||||
format!("events.{aggregate_type}.{aggregate_id}")
|
||||
}
|
||||
|
||||
/// Idempotent - safe to call on every boot. `max_age` deliberately left
|
||||
/// at the default (unlimited) - dodrenett's 7-day retention was an
|
||||
/// explicit dev default, not something worth inheriting for what's
|
||||
/// meant to be the durable source of truth.
|
||||
pub async fn ensure_stream(
|
||||
js: &async_nats::jetstream::Context,
|
||||
) -> anyhow::Result<async_nats::jetstream::stream::Stream> {
|
||||
let stream = js
|
||||
.get_or_create_stream(async_nats::jetstream::stream::Config {
|
||||
name: STREAM_NAME.to_string(),
|
||||
subjects: vec!["events.>".to_string()],
|
||||
..Default::default()
|
||||
})
|
||||
.await?;
|
||||
Ok(stream)
|
||||
}
|
||||
|
||||
/// Appends one event. `expected_seq` set to the aggregate's last-known
|
||||
/// sequence turns this into a compare-and-swap: a concurrent caller
|
||||
/// racing against the same aggregate with a stale `expected_seq` gets
|
||||
/// `PublishError` back instead of silently winning a lost-update race
|
||||
/// (see `crate::aggregates`, `transition_answer`'s old direct
|
||||
/// KV-mutate-in-place bug this replaces). `event.id` is set as the
|
||||
/// JetStream `Nats-Msg-Id` for the stream's dedup window - only
|
||||
/// catches a genuine retry with the *same* id, not a fresh submission;
|
||||
/// callers that want real retry-dedup must pass a stable id themselves.
|
||||
/// Returns the raw `PublishError` (not wrapped in `anyhow`) so callers
|
||||
/// like `crate::aggregates::classify_publish_err` can distinguish a CAS
|
||||
/// conflict (`WrongLastSequence`) from every other failure mode.
|
||||
pub async fn append_event(
|
||||
js: &async_nats::jetstream::Context,
|
||||
aggregate_type: &str,
|
||||
aggregate_id: &str,
|
||||
event_type: &str,
|
||||
payload: serde_json::Value,
|
||||
occurred_at_ms: i64,
|
||||
expected_seq: Option<u64>,
|
||||
) -> Result<async_nats::jetstream::publish::PublishAck, async_nats::jetstream::context::PublishError>
|
||||
{
|
||||
let event = Event {
|
||||
id: uuid::Uuid::new_v4(),
|
||||
aggregate_type: aggregate_type.to_string(),
|
||||
aggregate_id: aggregate_id.to_string(),
|
||||
event_type: event_type.to_string(),
|
||||
payload,
|
||||
occurred_at_ms,
|
||||
sequence: 0, // ignored on write - the server assigns the real one
|
||||
};
|
||||
// Every field here is a plain, always-serializable type (strings,
|
||||
// an i64, a Uuid, a serde_json::Value) - matches `chain.rs::hash_node`'s
|
||||
// own "json values always serialize" precedent.
|
||||
let data = serde_json::to_vec(&event).expect("Event always serializes");
|
||||
|
||||
let mut publish = async_nats::jetstream::context::Publish::build()
|
||||
.payload(data.into())
|
||||
.message_id(event.id.to_string());
|
||||
if let Some(seq) = expected_seq {
|
||||
publish = publish.expected_last_subject_sequence(seq);
|
||||
}
|
||||
|
||||
js.send_publish(subject(aggregate_type, aggregate_id), publish)
|
||||
.await?
|
||||
.await
|
||||
}
|
||||
|
||||
/// Loads every event for one aggregate, in real JetStream append order.
|
||||
/// Uses an ephemeral pull consumer scoped to this one aggregate's
|
||||
/// subject, fetched once and torn down - same idiom dodrenett's
|
||||
/// `getEvents` used, appropriate at this scale (single-digit events per
|
||||
/// aggregate), not a durable/long-lived consumer.
|
||||
pub async fn load_events(
|
||||
js: &async_nats::jetstream::Context,
|
||||
aggregate_type: &str,
|
||||
aggregate_id: &str,
|
||||
) -> anyhow::Result<Vec<Event>> {
|
||||
use futures::TryStreamExt;
|
||||
|
||||
let stream = js.get_stream(STREAM_NAME).await?;
|
||||
let subj = subject(aggregate_type, aggregate_id);
|
||||
let consumer = stream
|
||||
.create_consumer(async_nats::jetstream::consumer::pull::Config {
|
||||
filter_subject: subj,
|
||||
..Default::default()
|
||||
})
|
||||
.await?;
|
||||
|
||||
let mut events = Vec::new();
|
||||
let mut messages = consumer.fetch().max_messages(10_000).messages().await?;
|
||||
while let Some(message) = messages
|
||||
.try_next()
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("fetching messages: {e}"))?
|
||||
{
|
||||
let info = message.info().map_err(|e| anyhow::anyhow!("reading message info: {e}"))?;
|
||||
let mut event: Event = serde_json::from_slice(&message.payload)?;
|
||||
event.sequence = info.stream_sequence;
|
||||
events.push(event);
|
||||
message.ack().await.map_err(|e| anyhow::anyhow!("acking: {e}"))?;
|
||||
}
|
||||
events.sort_by_key(|e| e.sequence);
|
||||
|
||||
// Best-effort teardown - an ephemeral consumer left behind expires
|
||||
// on its own (default inactive_threshold), so a delete failure here
|
||||
// isn't worth failing the read over.
|
||||
let _ = stream.delete_consumer(&consumer.cached_info().name).await;
|
||||
|
||||
Ok(events)
|
||||
}
|
||||
@@ -6,6 +6,9 @@ pub mod content;
|
||||
pub mod events;
|
||||
pub mod resource;
|
||||
|
||||
#[cfg(feature = "ssr")]
|
||||
pub mod aggregates;
|
||||
|
||||
#[cfg(feature = "ssr")]
|
||||
pub mod server;
|
||||
|
||||
|
||||
@@ -30,6 +30,7 @@ async fn main() -> anyhow::Result<()> {
|
||||
let content_branch = std::env::var("CONTENT_BRANCH").unwrap_or_else(|_| "main".to_string());
|
||||
let gitea_base = content::gitea_api_base(&content_repo)?;
|
||||
let questions = content::load_questions_from_gitea(&content_repo, &content_branch, "questions").await?;
|
||||
content::validate_questions(&questions)?;
|
||||
tracing::info!(count = questions.len(), repo = %content_repo, branch = %content_branch, "loaded content");
|
||||
let questions = Arc::new(arc_swap::ArcSwap::from_pointee(questions));
|
||||
|
||||
@@ -50,6 +51,7 @@ async fn main() -> anyhow::Result<()> {
|
||||
// names whatever bucket an alternative's `record_as` should write
|
||||
// into, and `answers::store_answer` creates it on first use.
|
||||
let jetstream = async_nats::jetstream::new(nats.clone());
|
||||
portal::events::store::ensure_stream(&jetstream).await?;
|
||||
|
||||
let oidc_state = Arc::new(oidc::Oidc::from_env().await?);
|
||||
|
||||
|
||||
+273
-36
@@ -1,24 +1,30 @@
|
||||
//! Generic, authorized reads of NATS KV data declared in content
|
||||
//! Generic, authorized reads of live data declared in content
|
||||
//! (`content::ResourceSpec`, on a `Feature`). The one property that
|
||||
//! matters: the bucket/key a resource reads is never a client-supplied
|
||||
//! parameter, only something the server resolves from its own
|
||||
//! trusted, YAML-loaded content - the client only ever names a
|
||||
//! question id + feature name, so it can't probe an arbitrary bucket
|
||||
//! by just naming it.
|
||||
//! matters: what a resource reads is never a client-supplied parameter,
|
||||
//! only something the server resolves from its own trusted, YAML-loaded
|
||||
//! content - the client only ever names a question id + feature name
|
||||
//! (plus, now, whatever form-field values it already has, forwarded as
|
||||
//! query params), never a bucket/URL directly.
|
||||
|
||||
use leptos::prelude::*;
|
||||
|
||||
/// Fetches the live data for `question_id`'s `alternative`'s
|
||||
/// `feature_name` feature. Fails closed: a resource with neither
|
||||
/// `public: true` nor `requires_group` set is unreachable, not "open"
|
||||
/// by omission.
|
||||
/// by omission. `params` are the requesting form's current field
|
||||
/// values (if any) - forwarded as query parameters to a live external
|
||||
/// fetch (`GiteaStarred`/`GiteaOrgRepos`/`Url`), the mechanism behind
|
||||
/// "a resource parameterized by other form fields"; a `Kv` resource
|
||||
/// ignores them entirely, same as today.
|
||||
#[server]
|
||||
pub async fn get_resource(
|
||||
question_id: String,
|
||||
alternative: String,
|
||||
feature_name: String,
|
||||
params: std::collections::HashMap<String, String>,
|
||||
) -> Result<serde_json::Value, ServerFnError> {
|
||||
use crate::auth::{User, SESSION_USER_KEY};
|
||||
use crate::content::ResourceSource;
|
||||
use crate::server::AppState;
|
||||
|
||||
let state = expect_context::<AppState>();
|
||||
@@ -66,39 +72,270 @@ pub async fn get_resource(
|
||||
}
|
||||
}
|
||||
|
||||
let store = state
|
||||
.jetstream
|
||||
.get_key_value(&resource.bucket)
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(format!("resource bucket unavailable: {e}")))?;
|
||||
|
||||
match &resource.key {
|
||||
Some(key) => {
|
||||
let bytes = store
|
||||
.get(key)
|
||||
let value = match &resource.source {
|
||||
ResourceSource::Kv { bucket } => {
|
||||
let store = state
|
||||
.jetstream
|
||||
.get_key_value(bucket)
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?
|
||||
.ok_or_else(|| ServerFnError::new("resource key not found"))?;
|
||||
serde_json::from_slice(&bytes).map_err(|e| ServerFnError::new(e.to_string()))
|
||||
}
|
||||
None => {
|
||||
use futures::TryStreamExt;
|
||||
let keys: Vec<String> = store
|
||||
.keys()
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?
|
||||
.try_collect()
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?;
|
||||
let mut items = Vec::new();
|
||||
for key in keys {
|
||||
if let Ok(Some(bytes)) = store.get(&key).await {
|
||||
if let Ok(value) = serde_json::from_slice::<serde_json::Value>(&bytes) {
|
||||
items.push(value);
|
||||
.map_err(|e| ServerFnError::new(format!("resource bucket unavailable: {e}")))?;
|
||||
match &resource.key {
|
||||
Some(key) => {
|
||||
let bytes = store
|
||||
.get(key)
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?
|
||||
.ok_or_else(|| ServerFnError::new("resource key not found"))?;
|
||||
serde_json::from_slice(&bytes).map_err(|e| ServerFnError::new(e.to_string()))?
|
||||
}
|
||||
None => {
|
||||
use futures::TryStreamExt;
|
||||
let keys: Vec<String> = store
|
||||
.keys()
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?
|
||||
.try_collect()
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?;
|
||||
let mut items = Vec::new();
|
||||
for key in keys {
|
||||
if let Ok(Some(bytes)) = store.get(&key).await {
|
||||
if let Ok(value) = serde_json::from_slice::<serde_json::Value>(&bytes) {
|
||||
items.push(value);
|
||||
}
|
||||
}
|
||||
}
|
||||
serde_json::Value::Array(items)
|
||||
}
|
||||
}
|
||||
Ok(serde_json::Value::Array(items))
|
||||
}
|
||||
ResourceSource::GiteaStarred { username } => {
|
||||
fetch_gitea_json(&state, &format!("/api/v1/users/{username}/starred"), ¶ms).await?
|
||||
}
|
||||
ResourceSource::GiteaOrgRepos { org } => {
|
||||
fetch_gitea_json(&state, &format!("/api/v1/orgs/{org}/repos"), ¶ms).await?
|
||||
}
|
||||
ResourceSource::Url { url } => fetch_url_json(url, ¶ms).await?,
|
||||
};
|
||||
|
||||
match &resource.jq {
|
||||
Some(filter) => apply_jq(filter, &value),
|
||||
None => Ok(value),
|
||||
}
|
||||
}
|
||||
|
||||
/// Fetches JSON from a path on the same Gitea instance content is
|
||||
/// loaded from (`AppState.gitea_base`), attaching `GITEA_API_TOKEN` if
|
||||
/// set - confirmed live against `project.uhhm.no` that at least the
|
||||
/// starred-repos endpoint 401s unauthenticated, unlike the repo
|
||||
/// -contents/repo-info endpoints `content.rs` already calls anonymously.
|
||||
#[cfg(feature = "ssr")]
|
||||
async fn fetch_gitea_json(
|
||||
state: &crate::server::AppState,
|
||||
path: &str,
|
||||
params: &std::collections::HashMap<String, String>,
|
||||
) -> Result<serde_json::Value, ServerFnError> {
|
||||
let mut url = url::Url::parse(&format!("{}{}", state.gitea_base, path))
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?;
|
||||
for (k, v) in params {
|
||||
url.query_pairs_mut().append_pair(k, v);
|
||||
}
|
||||
let client = openidconnect::reqwest::Client::new();
|
||||
let mut req = client.get(url.clone());
|
||||
if let Ok(token) = std::env::var("GITEA_API_TOKEN") {
|
||||
if !token.is_empty() {
|
||||
req = req.header("Authorization", format!("token {token}"));
|
||||
}
|
||||
}
|
||||
let text = req
|
||||
.send()
|
||||
.await
|
||||
.and_then(|r| r.error_for_status())
|
||||
.map_err(|e| ServerFnError::new(format!("fetching {url}: {e}")))?
|
||||
.text()
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?;
|
||||
serde_json::from_str(&text).map_err(|e| ServerFnError::new(e.to_string()))
|
||||
}
|
||||
|
||||
/// Fetches JSON from an arbitrary content-declared URL
|
||||
/// (`content::ResourceSource::Url`). Content is trusted today (same
|
||||
/// org owns both this app and its content repo), but a server-side
|
||||
/// fetch of a content-supplied URL is real SSRF surface the moment
|
||||
/// that stops being true - `assert_public_https_url` fails closed on
|
||||
/// anything that isn't `https://` and actually resolves to a public
|
||||
/// address, not just a syntactically-plausible one.
|
||||
#[cfg(feature = "ssr")]
|
||||
async fn fetch_url_json(
|
||||
url: &str,
|
||||
params: &std::collections::HashMap<String, String>,
|
||||
) -> Result<serde_json::Value, ServerFnError> {
|
||||
let mut parsed = assert_public_https_url(url)
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?;
|
||||
for (k, v) in params {
|
||||
parsed.query_pairs_mut().append_pair(k, v);
|
||||
}
|
||||
let client = openidconnect::reqwest::Client::new();
|
||||
let text = client
|
||||
.get(parsed.clone())
|
||||
.send()
|
||||
.await
|
||||
.and_then(|r| r.error_for_status())
|
||||
.map_err(|e| ServerFnError::new(format!("fetching {parsed}: {e}")))?
|
||||
.text()
|
||||
.await
|
||||
.map_err(|e| ServerFnError::new(e.to_string()))?;
|
||||
serde_json::from_str(&text).map_err(|e| ServerFnError::new(e.to_string()))
|
||||
}
|
||||
|
||||
/// `https://` only, and only if every address the host resolves to is
|
||||
/// public - rejects loopback (127.0.0.1, ::1), RFC1918/link-local
|
||||
/// private ranges, and unspecified/multicast addresses. Resolves via
|
||||
/// real DNS (not just a syntax check) so a hostname that *points at* an
|
||||
/// internal address is caught too, not just a literal internal IP
|
||||
/// spelled out directly.
|
||||
#[cfg(feature = "ssr")]
|
||||
async fn assert_public_https_url(url: &str) -> anyhow::Result<url::Url> {
|
||||
let parsed = url::Url::parse(url)?;
|
||||
if parsed.scheme() != "https" {
|
||||
anyhow::bail!("only https:// resource urls are allowed");
|
||||
}
|
||||
let host = parsed
|
||||
.host_str()
|
||||
.ok_or_else(|| anyhow::anyhow!("resource url has no host"))?;
|
||||
let port = parsed.port_or_known_default().unwrap_or(443);
|
||||
let mut resolved = false;
|
||||
for addr in tokio::net::lookup_host((host, port))
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("resolving resource host {host}: {e}"))?
|
||||
{
|
||||
resolved = true;
|
||||
if !is_public_ip(&addr.ip()) {
|
||||
anyhow::bail!("resource host {host} resolves to a non-public address");
|
||||
}
|
||||
}
|
||||
if !resolved {
|
||||
anyhow::bail!("resource host {host} did not resolve to any address");
|
||||
}
|
||||
Ok(parsed)
|
||||
}
|
||||
|
||||
#[cfg(feature = "ssr")]
|
||||
fn is_public_ip(ip: &std::net::IpAddr) -> bool {
|
||||
match ip {
|
||||
std::net::IpAddr::V4(v4) => {
|
||||
!(v4.is_private()
|
||||
|| v4.is_loopback()
|
||||
|| v4.is_link_local()
|
||||
|| v4.is_multicast()
|
||||
|| v4.is_broadcast()
|
||||
|| v4.is_unspecified()
|
||||
|| v4.is_documentation())
|
||||
}
|
||||
std::net::IpAddr::V6(v6) => {
|
||||
let is_unique_local = (v6.segments()[0] & 0xfe00) == 0xfc00;
|
||||
let is_link_local = (v6.segments()[0] & 0xffc0) == 0xfe80;
|
||||
!(v6.is_loopback()
|
||||
|| v6.is_unspecified()
|
||||
|| v6.is_multicast()
|
||||
|| is_unique_local
|
||||
|| is_link_local)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Runs `filter_src` (a jq filter) over `input`, collecting every value
|
||||
/// it yields into a JSON array - e.g. `.[] | {name, url: .html_url}`
|
||||
/// over a Gitea repo list yields one shaped object per repo. Pure Rust
|
||||
/// via the `jaq` crate family, no shell-out to a `jq` binary.
|
||||
#[cfg(feature = "ssr")]
|
||||
pub fn apply_jq(
|
||||
filter_src: &str,
|
||||
input: &serde_json::Value,
|
||||
) -> Result<serde_json::Value, ServerFnError> {
|
||||
use jaq_core::load::{Arena, File, Loader};
|
||||
use jaq_core::{data, unwrap_valr, Compiler, Ctx, Vars};
|
||||
use jaq_json::{read, Val};
|
||||
|
||||
let input_text = input.to_string();
|
||||
let input_val = read::parse_single(input_text.as_bytes())
|
||||
.map_err(|e| ServerFnError::new(format!("internal: re-parsing json for jq: {e:?}")))?;
|
||||
|
||||
let program = File { code: filter_src, path: () };
|
||||
let defs = jaq_core::defs().chain(jaq_std::defs()).chain(jaq_json::defs());
|
||||
let funs = jaq_core::funs().chain(jaq_std::funs()).chain(jaq_json::funs());
|
||||
let loader = Loader::new(defs);
|
||||
let arena = Arena::default();
|
||||
let modules = loader
|
||||
.load(&arena, program)
|
||||
.map_err(|e| ServerFnError::new(format!("jq filter parse error: {e:?}")))?;
|
||||
let filter = Compiler::default()
|
||||
.with_funs(funs)
|
||||
.compile(modules)
|
||||
.map_err(|e| ServerFnError::new(format!("jq filter compile error: {e:?}")))?;
|
||||
let ctx = Ctx::<data::JustLut<Val>>::new(&filter.lut, Vars::new([]));
|
||||
|
||||
let mut outputs = Vec::new();
|
||||
for result in filter.id.run((ctx, input_val)).map(unwrap_valr) {
|
||||
let val = result.map_err(|e| ServerFnError::new(format!("jq filter runtime error: {e:?}")))?;
|
||||
let text = val.to_string();
|
||||
let json: serde_json::Value = serde_json::from_str(&text)
|
||||
.map_err(|e| ServerFnError::new(format!("internal: parsing jq output: {e}")))?;
|
||||
outputs.push(json);
|
||||
}
|
||||
Ok(serde_json::Value::Array(outputs))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A canned, Gitea-API-shaped fixture - the same fields the real
|
||||
/// `/users/{username}/starred` endpoint returns - run through the
|
||||
/// filter `index.yaml`'s "What we've built" resource actually
|
||||
/// declares, confirming the `jaq` integration produces the shape
|
||||
/// the frontend showcase card expects.
|
||||
#[test]
|
||||
fn jq_shapes_gitea_repo_list_for_the_showcase() {
|
||||
let input = serde_json::json!([
|
||||
{
|
||||
"name": "cnats",
|
||||
"description": "A NATS-backed chat client",
|
||||
"html_url": "https://project.uhhm.no/bl/cnats",
|
||||
"stargazers_count": 3,
|
||||
"private": false
|
||||
},
|
||||
{
|
||||
"name": "portal",
|
||||
"description": "This app",
|
||||
"html_url": "https://project.uhhm.no/uhhm/portal",
|
||||
"stargazers_count": 1,
|
||||
"private": false
|
||||
}
|
||||
]);
|
||||
let filter = ".[] | {name: .name, description: .description, url: .html_url, stars: .stargazers_count}";
|
||||
|
||||
let shaped = apply_jq(filter, &input).expect("filter runs");
|
||||
let items = shaped.as_array().expect("array output");
|
||||
assert_eq!(items.len(), 2);
|
||||
assert_eq!(items[0]["name"], "cnats");
|
||||
assert_eq!(items[0]["url"], "https://project.uhhm.no/bl/cnats");
|
||||
assert_eq!(items[0]["stars"], 3);
|
||||
// The filter never mentions `private` - confirms shaping
|
||||
// actually drops fields, not just passes the object through.
|
||||
assert!(items[0].get("private").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_public_ip_rejects_loopback_and_private() {
|
||||
let loopback: std::net::IpAddr = "127.0.0.1".parse().unwrap();
|
||||
let private: std::net::IpAddr = "10.0.0.5".parse().unwrap();
|
||||
let link_local: std::net::IpAddr = "169.254.1.1".parse().unwrap();
|
||||
let public: std::net::IpAddr = "1.1.1.1".parse().unwrap();
|
||||
assert!(!is_public_ip(&loopback));
|
||||
assert!(!is_public_ip(&private));
|
||||
assert!(!is_public_ip(&link_local));
|
||||
assert!(is_public_ip(&public));
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user