WF-9 · Macro + ML
Point-in-time features with release lags, rolling robust z-scores, Global Economy Pulse, Ridge/Elastic Net/Huber/PCA+Ridge/gradient boosting/random forest and prequential selection.
Progressive path
From visible output to implementation.
Each level adds detail without forcing the reader to restart: enter at the most useful point and continue progressively.
Methods added to the laboratory
World Economy Engine and Etna Sentinel: two different calculation families.
WF-9 · Regimes and graph
Three-state Gaussian Mixture with causal fitting; NetworkX structural graph and damped propagation. Bootstrap, conformal intervals and graph ablation govern evidence.
WF-10 · Simplified physical model
Hourly kinematic advection at 700/500 hPa, Haversine distance to transport assets and alignment bands. It is not an ash or concentration model.
WF-10 · ML, RPA and gates
Isolation Forest only characterises wind-context anomaly; GitHub Actions, the VONA source gate, deduplication and AT Protocol/XRPC implement fail-closed orchestration.
WF-11/12 · Chemical balances + economics
HHV/LHV balances, SMR/CCS/electrolysis routes, the NH₃→urea chain, break-even and sensitivities; Ridge/Gradient Boosting adds the M+1 forecast while keeping it separate from the physical calculation.
WF-13 · Point-in-time renewables forecast
Capacity factor, ECMWF weather, 1/24/168h lags, clear-sky and Gradient Boosting with empirical intervals. The zero-secret proxy remains separate from observed actuals.
WF-14/15 · Controlled process AI
PCA T²/Q, process residuals, CUSUM and voting for anomaly detection; PLS/Ridge/Random Forest/Gradient Boosting, split-conformal and Isolation Forest OOD for the soft sensor.
WF-16 · Operations research
HP/MP/LP balances, HiGHS LP/MILP, min-load, startup cost and perturb/re-opt marginal values without mislabelling MILP values as duals.
Packages: WF-9 uses NumPy, pandas, scikit-learn, NetworkX, SciPy, Matplotlib, SQLite, PyYAML, requests, ReportLab/openpyxl. WF-10 uses pandas, NumPy, scikit-learn, Matplotlib, Pillow, BeautifulSoup, pypdf, pydantic, requests and Bluesky integration. WF-11–16 add SciPy HiGHS, scikit-learn Gradient Boosting/PLS/Isolation Forest, SQLite and PyYAML.
Before the technical matrix
Five questions for evaluating a workflow rigorously.
- Was the data actually available when the result was produced?
- Is the system compared with a simpler method?
- Are error and uncertainty measured?
- Does behaviour change over time?
- Can the result be reconstructed?
Point-in-time
A forecast made on Monday cannot use data published on Tuesday.
Baseline
A complex model should be compared with a simple rule, such as “tomorrow will be similar to today”.
Calibration
If many events receive 70% probability, about seven in ten should occur over time.
Data drift
A model may lose reliability when data, context or relationships change.
Ablation
A source or variable is removed to test whether it genuinely contributes.
Failure mode
Describe in advance how the system may fail, stop or become misleading.
Architecture map
Ten layers separate sources, inference and publication.
Data engineering
Data correctness before model complexity.
Time and point-in-time
- separate source, acquisition, run and publication timestamps;
- normalise to UTC and use local time only for presentation;
- features use only information available at prediction time;
- explicit as-of joins and lags reduce leakage.
Contract and lineage
- expected schema and units for each adapter;
- source identifier and provenance URL;
- parser version and numeric fingerprint;
- cache state and fallback rationale.
Missing data
- distinguish observed zero from missing information;
- missing masks when coverage matters to the model;
- graceful degradation only when semantically safe;
- fail closed for official gates or critical sources.
Persistence
- CSV/JSON/Parquet selected for audit, volume and compatibility;
- atomic writes and deterministic names;
- ex-ante history separated from matured outcomes;
- anti-duplicate state separated from temporary artifacts.
Software ecosystem
Frameworks, libraries and protocols by architectural layer.
Data layer
pandas, NumPy, SciPy, PyArrow, Pydantic, SQLite, JSON, CSV and Parquet support normalisation, typing, timestamps, snapshots and audit.
Model layer
scikit-learn, LightGBM, XGBoost, statsmodels, optional PyTorch and NetworkX cover classification, regression, anomalies, clustering, bootstrap and graphs.
Source layer
requests, httpx, pandas-datareader, Beautiful Soup, lxml, pypdf, yfinance and optional ObsPy integrate APIs, pages, PDFs and public series with retries and timeouts.
Presentation layer
Matplotlib, Pillow, Chart.js, Jinja2, ReportLab, openpyxl, HTML5, CSS and JavaScript generate accessible cards, dashboards and responsive galleries.
Automation layer
GitHub Actions, cron, workflow_dispatch, pytest, Ruff, uv and Hatch orchestrate runs, tests, packaging and artifacts.
Publishing layer
AT Protocol/XRPC, TID, Bluesky, Cloudflare Pages and Workers handle standalone posts, static delivery and language routing.
Implementation patterns
Four contracts reduce runtime ambiguity.
Adapter contract
fetch(source_date) -> RawPayload
parse(payload) -> TypedRecord
validate(record) -> QualityReportAcquisition, parsing and validation remain separate and testable.
Outcome state
PUBLISHED | DRY_RUN
SKIPPED | BLOCKED
DEGRADED | FAILEDA green run is not confused with an actual publication.
Point-in-time
feature_time <= prediction_time
source_time <= run_cutoff
outcome_time > prediction_timeExplicit timestamps and lags reduce leakage and look-ahead.
Idempotency key
key = hash(event_id,
source_time,
language,
content_version)Fingerprints and deterministic record keys make retries safe.
MLOps and observability
The lifecycle includes data, model, execution and publication.
Versioning
- Git for code, configuration and documentation;
- package version and release contract;
- data and card hashes;
- ex-ante records separated from matured outcomes.
Reproducibility
pyproject.tomland lockfiles;- offline fixtures and seeds where applicable;
- input and configuration snapshots;
- downloadable run artifacts.
Observability
run_summary.jsonand provenance;- source, run and publication timestamps;
- record URIs and persistent state;
- explicit messages for skip, block and degraded states.
Outside the current scope
- OCI/Docker containers and registries;
- DVC or MLflow for experimental lineage;
- OpenTelemetry and Prometheus/Grafana;
- FastAPI for controlled services.
NOT PRESENTED AS IMPLEMENTED
Model risk and validation
A complex model must beat a relevant baseline out of sample.
| Area | Expected implementation | Error avoided | Evidence |
|---|---|---|---|
| Temporal split | ordered train/validation/test with gap or purge where needed | look-ahead and contamination | dates and fold counts |
| Baseline | naïve, climatology, persistence or simple model | apparent skill without incremental value | model-minus-baseline metric |
| Calibration | Brier, reliability curve, coverage or error bands | overconfident probabilities | matured sample and period |
| Ablation | remove feature groups or modalities | unjustified complexity | metric delta and stability |
| Robustness | bootstrap, period analysis, sensitivity | result driven by few episodes | intervals and worst period |
| Drift | monitor distributions, errors and coverage | obsolete model left undetected | history and thresholds |
Software and runtime
Operating reliability is part of the result.
Modularity
Adapters, processing, modelling, rendering and publishing remain separate and testable. Configuration is not scattered through code.
Idempotency
Event IDs, source dates, fingerprints and visual hashes prevent duplicate posts and accidental reuse of old cards.
State machine
Outcomes distinguish published, dry run, skipped, blocked and failed; a green run does not necessarily mean a post.
CI/CD
Lint, unit tests, offline integration tests, structural contracts and artifacts precede scheduled execution.
Secrets and security
Credentials live in protected environments with least privilege, log redaction and staging/production separation.
Observability
run_summary.json, provenance, versions, timestamps and publication URIs support diagnosis and audit.
Workflow matrix
Implemented methods and project-specific criteria.
| WF | Data and features | Inference | Validation | Characteristic stack | Failure mode |
|---|---|---|---|---|---|
| WF-1 | multi-city forecasts; range, wind, humidity, rain and pressure | rule-based index with capped contributions | consistency checks and sensitivity analysis; no clinical validation | Python, pandas, Open-Meteo, Pillow/Matplotlib | treating an educational index as individual prediction |
| WF-2 | energy futures, FX, lags and rolling features; 3-2-1 crack KPI | tree models and multi-horizon forecast | temporal splits, baselines, MAE/bias, error bootstrap | pandas, scikit-learn/XGBoost, Parquet, HTML reports | incomplete margin proxy and unstable relationships |
| WF-3 | cross-asset series, returns, volatility, breadth and spreads | regime, directional classifier, Isolation Forest, composite index | walk-forward, baseline comparison, period stability | yfinance, pandas, scikit-learn, Chart.js | oversimplifying shocks and non-stationary dependencies |
| WF-4 | AI-chain baskets, z-scores, networks and simulation parameters | composite index, 30-day outlook, bootstrap, network analytics | model-vs-naïve, matured errors and basket stability | NumPy/pandas, scikit-learn, NetworkX, Matplotlib | financial proxies do not equal real industrial capacity |
| WF-5 | GVP event, INGV gate, earthquakes, FIRMS, 700/500 hPa winds | Lagrangian advection, regime MLP, DBSCAN and optional anomaly score | synthetic tests, freshness gate, duplicate guards, official fail-closed | requests/httpx, BeautifulSoup, pypdf, Pydantic, scikit-learn, optional PyTorch | mistaking wind trajectories for concentration or hazard |
| WF-6 | Brent, TTF, volatility, spreads, rates, fertilisers and shipping/proxies; z-scores and confidence | weighted 0–100 index, EWM, ridge forecast and state machine | walk-forward OOS, Dynamic versus Static, subperiods, stress windows and risk analytics | pandas, SciPy, scikit-learn, statsmodels, SQLite, Jinja2/ReportLab | mistaking market proxies for physical availability or an operating signal |
| WF-7 | INGV events, FDSN seismicity, FIRMS, tremor and clouds; point-in-time and censoring features | LightGBM by modality, climatology/persistence/Hawkes baselines, blend and modality gate | purged walk-forward, embargo, holdout, Brier, calibration and matured scorecard | pandas, PyArrow, SciPy, scikit-learn, LightGBM, pypdf, optional ObsPy | mistaking educational probabilities for alerts; hiding incomplete modalities |
| WF-8 | RSS/API feeds, source registry, independence groups, text and provenance metadata | sentiment and topic/entity/event classification, lexical similarity, event cluster or theme bundle, facts-only templating | source gates, 5-gram overlap threshold, communicability, idempotency, PDS/AppView verification and audit receipts | requests, transformers, PyTorch, pandas/NumPy, Pillow, regex, AT Protocol/XRPC, GitHub Actions | non-independent or non-admitted sources, generic content, excessive reuse, or one language not being published |
| WF-9 | World Bank WDI, FRED and OECD; release lags, lags/differences/rolling features, robust z-scores, Global Pulse components and graph-propagated features | Ridge, Elastic Net, Huber, PCA+Ridge, HistGradientBoosting, Random Forest; robust ensemble; 3-state GMM; damped graph-propagation scenarios | expanding backtest with purge gap, prequential selection, naive benchmark, bootstrap skill, 90% conformal intervals, graph ablation and anti-leakage/vintage audit | Python, NumPy, pandas, scikit-learn, NetworkX, SciPy, Matplotlib, SQLite, PyYAML, requests, ReportLab/openpyxl | mistaking an expert-hypothesis graph for proven causality; macro revisions, small OOS, structural breaks or experimental forecasts treated as advice |
| WF-10 | INGV VONA, Open-Meteo 700/500 hPa winds, FIRMS and GIBS VIIRS; airport/road/rail/port geometries and provenance | hourly 12h kinematic advection, Haversine distance and alignment bands; Isolation Forest on wind context; rule-based VONA relay | unit/end-to-end tests, data-quality and disclosure gates, deduplication/idempotency, fail-closed on uncertain/positive official status and immutable history | Python, pandas, NumPy, scikit-learn, Matplotlib, Pillow, BeautifulSoup, pypdf, pydantic, requests, GitHub Actions, AT Protocol/XRPC | interpreting direction/proximity as plume, concentration, impact or disruption probability; source latency or unavailability |
| WF-11 | EEX→BFE TTF with quality/provenance gate, Italian bidding-zone day-ahead power via Energy-Charts→euenergy/ENTSO-E, EUA and grid-carbon-intensity scenario; typed HHV/LHV SMR/CCS/electrolysis parameters | deterministic calculation; Current Variable Cost and LCOH; state-aware break-even | physical/dimensional invariants, sourcing coherence, lineage, snapshot replay | Python, NumPy, pandas, SciPy, Matplotlib, SQLite, PyYAML | mistaking PPA scenarios or mismatched emissions boundaries for observed facts |
| WF-12 | TTF, World Bank Pink Sheet, ECB FX, lags/seasonality; explicit HHV/LHV | deterministic NH₃/urea chain + Ridge/Gradient Boosting M+1 forecast | walk-forward, persistence, seasonality, gas-only and physical baselines; MAE/RMSE/bias/coverage/skill | Python, pandas, scikit-learn, SQLite, Matplotlib | mistaking a global benchmark for European cost or mixing nowcast and forecast |
| WF-13 | Terna Public API primary and ENTSO-E fallback for observed generation/capacity; without credentials, a non-publishable Open-Meteo proxy using declared 100 m wind nodes, local power curves before weighted aggregation, ECMWF D+1; 1/24/168h lags, seasonality and clear-sky | Gradient Boosting on capacity factor; empirical intervals | time split and persistence; proxy separated from actuals and never matured as a real forecast; prospective ENTSO-E A69 baseline when available; nMAE/RMSE/bias/coverage/skill | Python, pandas, scikit-learn, requests, SQLite, Matplotlib | mistaking the weather/capacity proxy for observed generation, missing forecast vintages or installed-capacity growth mistaken for skill |
| WF-14 | generic CSTR multivariate sensors, noise, missing data, step/drift/intermittent faults | drift-aware ensemble: high-specificity PCA T²/Q + steam, cooling and reaction residuals + slow steam-residual CUSUM; Isolation Forest/autoencoder as comparators; fault classifier | parameters frozen on a separate development simulation; same frozen 28-day synthetic benchmark; event recall, missed rate, false-alarm episodes/24h, P50/P90 delay, v1.4.16 ensemble comparison and anti-leakage | Python, NumPy, pandas, scikit-learn, SQLite, Matplotlib | performance applies only to the simulator; test-bed-specific process residuals and an uncalibrated classifier require validation before real-plant transfer |
| WF-15 | process sensors + delayed laboratory with noise/QC | Linear, PLS, Ridge, Random Forest, Gradient Boosting; Isolation Forest OOD | RMSE/MAE/bias/coverage, skill versus last lab, split-conformal and truth-vs-lab; risk-coverage diagnostic only | Python, pandas, scikit-learn, SQLite, Matplotlib | leakage from true values or a soft sensor answering outside its domain |
| WF-16 | synthetic steam demand, capacities/efficiencies, EEX TTF, licence-checked Energy-Charts Italian bidding-zone day-ahead power, EUA | HiGHS LP and MILP; dispatch and perturb/re-opt | feasibility, balances, optimality, solve time, stability and reference policy | Python, SciPy HiGHS, NumPy, pandas, SQLite, Matplotlib | calling MILP marginal values duals or presenting simulated savings as measured savings |
Academic and technical assessment
Useful questions for independent review.
- Can every source timestamp be reconstructed?
- Do features respect point-in-time availability?
- Is the baseline appropriate?
- Are hyperparameters selected without contaminating test data?
- Is the matured-outcome sample sufficient?
- Do metrics include denominator and period?
- Do missing data cause an explicit block or degradation?
- Are results reproducible from configuration and version?
- Are failure modes tested or only described?
- Is publication idempotent and auditable?
- Do source licences allow the intended reuse?
- Is implemented functionality separated from research extensions?
Technical observations or proposed checks
Feedback on methodology, tests, reproducibility and possible experiments is welcome.
Multi-source method for Bluesky
Technical components of workflow WF-8.
Source policy and copyright
- domain registry with source_class, risk_tier and reuse_mode;
- communicable facts only and locally rendered images;
- anti-reuse gate with 5-gram overlap checks.
NLP and comparison
- sentiment, topic, entity and event classification;
- source alignment through lexical similarity and domain signatures;
- explicit distinction between an event cluster and a theme bundle.
Templating and publication
- facts-only templates for title, takeaway, why-it-matters and technical lens;
- separate Italian and English root posts;
- facets to sources, no
reply_to, multiple images per post.
Packages used
- requests for feed and API collection;
- transformers and torch for NLP enrichment;
- Pillow, matplotlib, pandas, numpy, regex for rendering, data and Bluesky limits.