.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "gallery/experimental_workflow_demo.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_gallery_experimental_workflow_demo.py: Experimental-data workflow template ====================================== The recommended workflow for applying SFI to **real experimental data**. This demo loads a 2D optical-tweezer trajectory from a CSV file, infers the force and diffusion, sparsifies with PASTIS, and validates with a bootstrap trajectory. The data includes localization noise, a slow rotation (torque), and a weak Duffing-type cubic anharmonicity in :math:`x`. PASTIS automatically detects these non-trivial terms beyond the harmonic restoring force. .. admonition:: SFI's design philosophy Stochastic Force Inference is built for experimental trajectories where *no model pre-exists*. The linear regression backbone requires no initial guess, works with arbitrary basis functions, and the PASTIS information criterion rigorously identifies which terms are supported by the data. .. rubric:: Tags real-data · overdamped · experimental-workflow .. GENERATED FROM PYTHON SOURCE LINES 27-29 .. code-block:: Python :dedent: 1 .. GENERATED FROM PYTHON SOURCE LINES 54-62 Step 1 — Load trajectory from CSV ------------------------------------ ``TrajectoryCollection.load()`` handles CSV (with optional YAML header), Parquet, and HDF5. The CSV file here uses a ``# dt: 0.01`` YAML header so the timestep is automatically set. In your own workflow, replace the path with your data file. .. GENERATED FROM PYTHON SOURCE LINES 62-70 .. code-block:: Python from SFI.trajectory import TrajectoryCollection csv_path = _here.parent / "experimental_data" / "optical_tweezer.csv" coll = TrajectoryCollection.load(csv_path) print(f"Loaded: {coll.T} frames, d={coll.d}, dt={coll.dt}") .. rst-class:: sphx-glr-script-out .. code-block:: none Loaded: 20001 frames, d=2, dt=0.01 .. GENERATED FROM PYTHON SOURCE LINES 71-76 Step 2 — Visualize the data ------------------------------ Always inspect your data first: check for outliers, missing frames, or obvious drift. .. GENERATED FROM PYTHON SOURCE LINES 76-77 .. code-block:: Python :dedent: 1 .. image-sg:: /gallery/images/sphx_glr_experimental_workflow_demo_001.png :alt: Step 2 — Data inspection, Trajectory (2D), Time series :srcset: /gallery/images/sphx_glr_experimental_workflow_demo_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 100-110 Step 3 — Basis, inference, and sparsification ------------------------------------------------- Start with a polynomial basis (degree 3). For experimental data you typically do not know the diffusion tensor, so we estimate it. PASTIS then selects the simplest model the data supports. The data contains a weak Duffing cubic (:math:`-\alpha\,x^3`) in one axis and a slow rotation; PASTIS should recover these beyond the dominant harmonic terms. .. GENERATED FROM PYTHON SOURCE LINES 110-130 .. code-block:: Python from SFI.bases import monomials_up_to from SFI import OverdampedLangevinInference from SFI.utils import plotting degree = 3 B = monomials_up_to(order=degree, dim=2, rank='vector') inf = OverdampedLangevinInference(coll) inf.compute_diffusion_constant() inf.infer_force_linear(B) inf.compute_force_error() inf.sparsify_force(criterion="PASTIS") support = np.asarray(inf.force_support) k_sel = len(support) if k_sel > 0: inf.compute_force_error() inf.print_report() .. image-sg:: /gallery/images/sphx_glr_experimental_workflow_demo_002.png :alt: Step 3 — Sparsified inference, PASTIS selected 4 / 20 features, Inferred force field (sparse) :srcset: /gallery/images/sphx_glr_experimental_workflow_demo_002.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none --- StochasticForceInference Report --- Average diffusion tensor: [[0.48866063 0.00630378] [0.00630378 0.5060318 ]] Measurement noise tensor: [[ 5.5156957e-04 -6.7758949e-05] [-6.7758949e-05 5.2479864e-04]] Force estimated information: 197.58346557617188 Force: estimated normalized mean squared error (sampling only): 0.010122304486196823 Force Coefficient Table ───────────────────────────────────────────────────────────────────── # Label Coefficient Std.Err SNR Sig ───────────────────────────────────────────────────────────────────── 2 x0·e0 -1.27470e+00 2.77786e-01 4.6 * 4 x1·e0 7.40408e-01 1.05972e-01 7.0 * 5 x1·e1 -1.11140e+00 1.05428e-01 10.5 ** 12 x0^3·e0 -2.10942e+00 4.44976e-01 4.7 * ───────────────────────────────────────────────────────────────────── 4/20 basis functions in support, sig: 4* / 1** / 0*** (|SNR| ≥ 2 / 10 / 100) (Std.err. reflects sampling error only; discretization bias is not included.) Zeroed (16): 1·e0, 1·e1, x0·e1, x0^2·e0, x0^2·e1, (x0·x1)·e0, (x0·x1)·e1, x1^2·e0, x1^2·e1, x0^3·e1, (x0^2·x1)·e0, (x0^2·x1)·e1, (x0·x1^2)·e0, (x0·x1^2)·e1, x1^3·e0, x1^3·e1 .. GENERATED FROM PYTHON SOURCE LINES 153-158 Step 4 — Bootstrap validation -------------------------------- Simulate a trajectory from the inferred model and compare with the original data — a self-consistency check. .. GENERATED FROM PYTHON SOURCE LINES 158-162 .. code-block:: Python key = random.PRNGKey(123) coll_boot, _ = inf.simulate_bootstrapped_trajectory(key) .. image-sg:: /gallery/images/sphx_glr_experimental_workflow_demo_003.png :alt: Step 4 — Bootstrap vs data :srcset: /gallery/images/sphx_glr_experimental_workflow_demo_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 176-188 Step 5 — Consistency diagnostics ----------------------------------- The final check on real data: recompute the standardised residuals and test whether they behave like an i.i.d. :math:`\mathcal{N}(0, 1)` sample. Localization noise biases the linear Itô estimator (an errors-in-variables effect), so a residual-consistency flag here is a useful signal — not a failure of the workflow, but a sign that measurement noise matters for this dataset. When it appears, the parametric estimator (:meth:`~SFI.inference.OverdampedLangevinInference.infer_force`) profiles the noise level and removes the bias — see :doc:`/inference/noise_and_sampling`. Each flag below carries its one-line action hint inline. .. GENERATED FROM PYTHON SOURCE LINES 188-194 .. code-block:: Python from SFI.diagnostics import assess, plot_summary report = assess(inf, level="standard") report.print_summary() .. image-sg:: /gallery/images/sphx_glr_experimental_workflow_demo_004.png :alt: Step 5 — residual diagnostics, Q--Q plot, Residual histogram, Residual ACF :srcset: /gallery/images/sphx_glr_experimental_workflow_demo_004.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none === SFI diagnostics report === backend : linear regime : OD n_obs : 20000 n_particles: 1 d: 2 level : standard -- Residuals -- mean = +0.0039 std = 1.0469 skew = +0.003 kurt-3 = -0.027 (n=40000) ✗ normality ks stat=0.014 p=3.29e-07 ✗ autocorr ljung_box stat=90 p=7.48e-11 ✓ autocorr ljung_box_squared stat=22.4 p=0.318 predicted NMSE = 0.0101 realised NMSE = 9.41 χ² z = +13.58 (|z|>5 ⇒ bias) -- Flags -- ! [normality/ks] p=3.29e-07 < 0.01 — non-Gaussian residuals — rare events not captured by the basis, or a non-Gaussian noise structure ! [autocorr/ljung_box] p=7.48e-11 < 0.01 — missing time-correlated feature — widen the basis; if it persists, suspect coarse sampling: the parametric estimator (infer_force) extends the usable Δt ! [mse_consistency] residual chi^2 z-score = +13.58, realised/predicted NMSE = 9.3e+02 — realised error above predicted — model bias; on experimental data usually measurement noise: consider the parametric estimator (infer_force) .. GENERATED FROM PYTHON SOURCE LINES 200-202 Thumbnail --------- .. GENERATED FROM PYTHON SOURCE LINES 202-205 .. code-block:: Python stamp_output() .. image-sg:: /gallery/images/sphx_glr_experimental_workflow_demo_005.png :alt: experimental workflow demo :srcset: /gallery/images/sphx_glr_experimental_workflow_demo_005.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none [Generated: 2026-06-30 10:03] .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 20.512 seconds) .. rst-class:: sphx-glr-example-tags 🏷 Tags: real-data, overdamped, experimental-workflow .. _sphx_glr_download_gallery_experimental_workflow_demo.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: experimental_workflow_demo.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: experimental_workflow_demo.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: experimental_workflow_demo.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_