MeasurementEditing#
- class MeasurementEditing(datasets)#
Bases:
objectInteractive and automated editing of measurement residuals for orbit determination.
Provides these editing modes:
Lasso Selector — graphically select points to flag as outliers on a plot.
Chi-Squared — automatically flag points whose residuals exceed a chi-squared threshold.
Kurtosis — automatically trim the heaviest tails until the residual kurtosis falls below a target (3, the Gaussian value, by default).
Date Range — manually specify time windows to flag or unflag observations.
Isolation Forest — flag anomalous residuals by tree-isolation depth, with the rejection threshold taken from the knee of the ranked scores rather than from an assumed contamination fraction. Works jointly on the two components of a vector observable.
Huber / MAD — detrend each pass with a Huber regression, then threshold on a robust MAD scale.
Residual Interval — flag residuals outside a fixed interval, by default 5 sigma about the mean.
- Parameters:
- datasetslist of MeasurementDataSet
The datasets to edit. Nested lists are flattened one level.
- Attributes:
datasetsThe list of managed datasets.
isolation_forest_min_pointsFewest active, finite points
isolation_forest_filter()runs on; below this it flags only the non-finite points.outlier_flagsDict of {dataset_name: bool array} of current outlier flags.
pass_gapGap above which consecutive observations belong to separate passes in
huber_mad_filter(), each detrended on its own.pass_min_pointsFewest points a pass may have in
huber_mad_filter()before it is merged into its neighbour instead of being detrended alone.refit_sigmasDict of {dataset_name: float} sigmas re-estimated by huber_mad_filter.
Methods
Write the edited outlier_flag arrays back into each dataset's data dict.
chi_squared_filter([dataset_name, ...])Automatically flag measurements whose normalised residuals exceed a chi-squared threshold.
chi_squared_filter_all([dataset_name, ...])Chi-squared filtering with every preview figure shown AT ONCE.
date_range_filter(t_start, t_end[, ...])Flag or unflag all measurements whose time falls within [t_start, t_end].
Apply flags and return the list of edited datasets.
huber_mad_filter([dataset_name, n_sigma, ...])Flag outliers by detrending residuals against time with a Huber regression, then thresholding on a MAD-based robust scale.
interactive_date_range([dataset_name])Open an interactive GUI for manual date-range flagging and unflagging.
isolation_forest_filter([dataset_name, ...])Flag anomalous residuals with an isolation forest, cutting at the knee of the ranked anomaly scores.
isolation_forest_filter_all([dataset_name, ...])Isolation-forest filtering with every preview figure shown AT ONCE.
kurtosis_filter([dataset_name, ...])Trim the most extreme normalised residuals until the sample kurtosis of the remaining points drops below
target_kurtosis.kurtosis_filter_all([dataset_name, ...])Kurtosis filtering with every preview figure shown AT ONCE.
lasso_editor([dataset_name, x_field, ...])Open an interactive matplotlib figure for lasso-based outlier selection.
lasso_editor_all([dataset_name, x_field, ...])Open a SEPARATE lasso-editor figure for each dataset and display them all at once with a single
plt.show().reset([dataset_name])Clear all outlier flags.
residual_filter([dataset_name, ...])Flag measurements whose residuals fall outside a fixed interval.
summary()Print a per-dataset summary of flagged vs. total measurements.
See also
scarabaeus.FilterDataManagerCombines datasets and applies editing before indexing.
scarabaeus.FilterODBase filter that consumes the edited datasets.
Notes
Edits are non-destructive: original
outlier_flagarrays are copied internally and not written back untilapply_flags()orget_datasets()is called.Examples
editor = MeasurementEditing(datasets) editor.lasso_editor(dataset_name="DSN_Range") editor.chi_squared_filter(sigma_scale=3.0) editor.kurtosis_filter(target_kurtosis=3.0) editor.date_range_filter("2024-01-01T00:00:00", "2024-01-05T12:00:00") edited_datasets = editor.get_datasets()
lasso_editor_all(),chi_squared_filter_all()andkurtosis_filter_all()are the all-figures-at-once counterparts oflasso_editor(),chi_squared_filter()andkurtosis_filter(), for editing several datasets without blocking on one figure at a time.The pass splitting of
huber_mad_filter()and the minimum sample ofisolation_forest_filter()are set per editor throughpass_gap,pass_min_pointsandisolation_forest_min_points, which start from the class defaultsPASS_GAP,PASS_MIN_POINTSandISOLATION_FOREST_MIN_POINTS.- apply_flags()#
Write the edited outlier_flag arrays back into each dataset’s data dict. Call this when satisfied with edits, before passing datasets to the filter.
- chi_squared_filter(dataset_name: str = None, sigma_scale: float = 3.0, dof: int = 1, preview: bool = True)#
Automatically flag measurements whose normalised residuals exceed a chi-squared threshold.
Criterion: (residual / sigma)^2 > chi2.ppf(1 - alpha, dof) where alpha = 1 - CDF(sigma_scale^2, dof=1), equivalent to N-sigma.
- Parameters:
- dataset_namestr, list of str, or None
Name(s) of the dataset(s) to filter. If None, applies to all datasets.
- sigma_scalefloat
Equivalent Gaussian sigma multiplier (e.g. 3.0 for 3-sigma rejection).
- dofint
Degrees of freedom for chi-squared distribution (default 1).
- previewbool
If True, show a summary plot before applying flags.
- chi_squared_filter_all(dataset_name=None, sigma_scale: float = 3.0, dof: int = 1, preview: bool = True)#
Chi-squared filtering with every preview figure shown AT ONCE.
Same rejection criterion and same flag merging as
chi_squared_filter()— the only difference is presentation: all preview figures are built first and displayed together by a singleplt.show(), instead of one blocking figure per dataset.Use this instead of calling
chi_squared_filter()in a per-dataset loop. Looping blocks on each dataset’splt.show(), which is clunky on remote/browser backends: closing a plot tab there does not advance the loop to the next dataset.Flags are merged only after the previews are closed, so the previews still show the state before any flags are applied. Note the previews are informational, not a confirmation prompt — the flags are applied either way.
- Parameters:
- dataset_namestr or list of str or None
Dataset(s) to filter.
Nonefilters all managed datasets.- sigma_scalefloat
Equivalent Gaussian sigma multiplier (e.g. 3.0 for 3-sigma rejection).
- dofint
Degrees of freedom for chi-squared distribution (default 1).
- previewbool
If True, show the summary plots before applying flags.
- date_range_filter(t_start, t_end, dataset_name: str = None, action: str = 'flag', time_field: str = 't2')#
Flag or unflag all measurements whose time falls within [t_start, t_end].
- Parameters:
- t_startfloat or str
Start of the window. Either a numeric epoch (seconds) or ISO 8601 string (e.g. “2024-01-01T00:00:00”).
- t_endfloat or str
End of the window (same format as t_start).
- dataset_namestr, list of str, or None
Dataset(s) to apply to.
Noneapplies to all datasets.- actionstr
"flag"marks measurements in the window as outliers;"unflag"removes the outlier flag from them.- time_fieldstr
Time field to use for comparison (default: ‘t2’).
- get_datasets()#
Apply flags and return the list of edited datasets.
- Returns:
- list of MeasurementDataSet
- huber_mad_filter(dataset_name=None, n_sigma: float = 5.0, mad_scale: float = 1.4826, refit_sigma: bool = True, pass_gap_sec: float = None, min_pass_points: int = None)#
Flag outliers by detrending residuals against time with a Huber regression, then thresholding on a MAD-based robust scale.
A tracking pass usually carries a slope that a plain mean-and-sigma test reads as spread, so the trend is removed first; the median absolute deviation then keeps the scale estimate from being dragged by the outliers it is meant to find. Unlike
chi_squared_filter, this judges the residuals by their own scatter rather than by the sigma already attached to the measurements, which is what makes it usable before a trustworthy sigma exists.Has no preview figure, like
date_range_filter.- Parameters:
- dataset_namestr, list of str, or None
Datasets to filter.
None= all.- n_sigmafloat
Acceptance width, in units of the robust scale below.
- mad_scalefloat
Factor turning the median absolute deviation into that scale. Defaults to 1.4826, which makes the MAD a consistent estimator of the standard deviation for normal data, so
n_sigmareads as a number of sigmas. Pass 1.0 to threshold on the raw MAD instead.- refit_sigmabool
Re-estimate a measurement sigma from the surviving points and record it in
refit_sigmas. The editor does not own the measurement models, so applying it is left to the caller.- pass_gap_secfloat, optional
Gap [s] above which consecutive observations belong to separate passes, each detrended on its own, for this call only. Defaults to
pass_gap.- min_pass_pointsint, optional
Fewest points a pass may have before it is merged into its neighbour instead of being detrended alone, for this call only. Defaults to
pass_min_points.
- Returns:
- dict
{dataset_name: float}of re-estimated sigmas, empty whenrefit_sigmais False.
- interactive_date_range(dataset_name: str = None)#
Open an interactive GUI for manual date-range flagging and unflagging.
Displays a residuals scatter plot for the first targeted dataset and provides two text boxes (
t_start,t_end) and three buttons:Flag — apply
date_range_filter()withaction='flag'to all target datasets for the entered epoch range.Unflag — apply
date_range_filter()withaction='unflag'.Done — close the figure and return.
Both text boxes accept numeric epoch strings (seconds, e.g.
"705427200.0") or ISO 8601 date strings (e.g."2024-01-01T00:00:00").The plot updates immediately after each Flag/Unflag action to reflect the current outlier state of the first dataset.
- Parameters:
- dataset_namestr, list of str, or None
Dataset(s) to target. If None, all managed datasets are targeted (the plot preview always shows the first one).
- isolation_forest_filter(dataset_name=None, standardize: bool = True, scale_by_cos_dec: bool = False, n_estimators: int = 500, random_state: int = 0, preview: bool = True)#
Flag anomalous residuals with an isolation forest, cutting at the knee of the ranked anomaly scores.
An isolation forest scores a point by how few random splits it takes to separate it from the rest: outliers isolate quickly. Unlike
chi_squared_filter()no sigma threshold is assumed, and unlikekurtosis_filter()no distribution shape is assumed — but the forest still has to be told how much to reject, and rather than fixing a contamination fraction the threshold is taken from the knee of the ranked scores (maximum drop below the endpoint chord), so the data decides where the tail begins.A two-component observable is scored jointly: the two scalar datasets an optical track is split into are recombined into one 2-D point per observation, and a rejection flags both components — a bad image is bad in both coordinates, not in half of one.
- Parameters:
- dataset_namestr or list of str or None
Dataset(s) to filter. Components of the same observable are grouped automatically. If None, every dataset is considered.
- standardizebool
Robustly rescale each coordinate by its median and MAD before fitting (default True). Without it the forest isolates on whichever component happens to be noisier.
- scale_by_cos_decbool
For a right-ascension/declination pair, multiply the RA residual by
cos(dec)so the pair is compared on the tangent plane. Ignored for any other observable. Defaults to False.- n_estimatorsint
Number of trees (default 500).
- random_stateint
Seed, so a rejection is reproducible (default 0).
- previewbool
If True, show the residuals and the knee before applying flags.
- Raises:
- InvalidIsolationForestSettingsError
If
n_estimatorsis below 1.
Notes
An observable with fewer than
isolation_forest_min_points(eight by default) active points is left untouched — a forest cannot say anything useful about a handful of points.A statistical label is not proof that a measurement is wrong. Refit and recompute residuals after rejecting, and check that the solution is not unduly sensitive to what was cut.
References
[1]Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008). Isolation Forest. Eighth IEEE International Conference on Data Mining, 413-422.
[2]Satopää, V., Albrecht, J., Irwin, D., & Raghavan, B. (2011). Finding a “Kneedle” in a Haystack. 31st International Conference on Distributed Computing Systems Workshops, 166-171.
- isolation_forest_filter_all(dataset_name=None, standardize: bool = True, scale_by_cos_dec: bool = False, n_estimators: int = 500, random_state: int = 0, preview: bool = True)#
Isolation-forest filtering with every preview figure shown AT ONCE.
Same rejection as
isolation_forest_filter(); the figures are all built first and displayed by a single blockingplt.show()instead of one figure per observable. Flags are merged only after the previews are closed, so each preview still shows the state before anything was applied.- Parameters:
- dataset_namestr or list of str or None
Dataset(s) to filter. See
isolation_forest_filter().- standardize, scale_by_cos_dec, n_estimators, random_stateoptional
As in
isolation_forest_filter().- previewbool
If True, show all preview figures together.
- Raises:
- MeasurementEditing.InvalidIsolationForestSettingsError
If
n_estimatorsis less than 1.
- kurtosis_filter(dataset_name: str = None, target_kurtosis: float = 3.0, max_fraction: float = 0.25, preview: bool = True)#
Trim the most extreme normalised residuals until the sample kurtosis of the remaining points drops below
target_kurtosis.A Gaussian residual population has kurtosis 3 (Pearson definition, i.e. zero excess kurtosis); anything above that is heavy-tailed, which for tracking residuals means outliers rather than genuine noise. Unlike
chi_squared_filter(), no absolute sigma threshold is assumed — the data decides how much of the tail has to go.Procedure: normalised residuals
z = residual / sigmaare ordered by|z - median(z)|(most extreme first) and removed one at a time; the kurtosis of the surviving points is evaluated after every removal, and the smallest number of removals that brings it totarget_kurtosisis the one applied. Only points that are still active are considered — already-flagged points would inflate the kurtosis and drive over-trimming — and points with zero or non-finite sigma are ignored (never flagged), as inchi_squared_filter().- Parameters:
- dataset_namestr, list of str, or None
Name(s) of the dataset(s) to filter. If None, applies to all datasets.
- target_kurtosisfloat
Kurtosis to trim down to (default 3.0, the Gaussian value). Must be at least 1.0, the smallest kurtosis any distribution can have.
- max_fractionfloat
Safety cap on the fraction of active points that may be removed (default 0.25). If the target is not reached within the cap, the capped number of points is flagged and a warning is printed.
- previewbool
If True, show a summary plot before applying flags.
- Raises:
- InvalidKurtosisSettingsError
If
target_kurtosis < 1ormax_fractionis outside [0, 1).
Notes
Datasets with fewer than four active points are left untouched: the fourth moment is not meaningful there.
- kurtosis_filter_all(dataset_name=None, target_kurtosis: float = 3.0, max_fraction: float = 0.25, preview: bool = True)#
Kurtosis filtering with every preview figure shown AT ONCE.
Same trimming criterion and same flag merging as
kurtosis_filter()— the only difference is presentation: all preview figures are built first and displayed together by a singleplt.show(), instead of one blocking figure per dataset.Use this instead of calling
kurtosis_filter()in a per-dataset loop. Looping blocks on each dataset’splt.show(), which is clunky on remote/browser backends: closing a plot tab there does not advance the loop to the next dataset.Flags are merged only after the previews are closed, so the previews still show the state before any flags are applied. Note the previews are informational, not a confirmation prompt — the flags are applied either way.
- Parameters:
- dataset_namestr or list of str or None
Dataset(s) to filter.
Nonefilters all managed datasets.- target_kurtosisfloat
Kurtosis to trim down to (default 3.0, the Gaussian value).
- max_fractionfloat
Safety cap on the fraction of active points that may be removed.
- previewbool
If True, show the summary plots before applying flags.
- lasso_editor(dataset_name=None, x_field: str = 't2', y_field: str = 'residuals', title: str = None)#
Open an interactive matplotlib figure for lasso-based outlier selection.
Left-click drag draws a lasso; enclosed visible points are toggled as outliers. Buttons:
[Confirm] — commit the current flags to the editor, hide the rejected points, and rescale the axes to the remaining points. The figure stays open so editing can continue.
[Show/Hide Rejected] — toggle visibility of rejected points (including points whose
outlier_flagwas already True when the editor opened), rescaling the axes each time. Hidden points cannot be lassoed.[Reset] — clear all flags for the displayed dataset(s).
[Done] — commit and close the figure.
Multiple datasets of the same data type can be edited together (e.g. all range measurements regardless of which measurement model produced them) by passing a list of names or
Nonefor all managed datasets. Each dataset’s active points get their own color; flags are committed back to each dataset individually.- Parameters:
- dataset_namestr or list of str or None
Name(s) of the dataset(s) to edit (must match
MeasurementDataSet.set_name).Noneedits all managed datasets at once.- x_fieldstr
Field to use for x-axis (default: ‘t2’).
- y_fieldstr
Field to use for y-axis (default: ‘residuals’).
- titlestr, optional
Plot title override.
- lasso_editor_all(dataset_name=None, x_field: str = 't2', y_field: str = 'residuals', title: str = None)#
Open a SEPARATE lasso-editor figure for each dataset and display them all at once with a single
plt.show().Each figure is fully interactive on its own (its own lasso plus the Confirm / Show-Hide Rejected / Reset / Done buttons); committing writes the flags back to that dataset. Control returns once every figure has been closed.
Use this instead of calling
lasso_editor()in a per-dataset loop. Looping blocks on each dataset’splt.show(), which is clunky on remote/browser backends: closing a plot tab there does not advance the loop to the next dataset. Here a singleplt.show()drives every dataset’s editor together.- Parameters:
- dataset_namestr or list of str or None
Dataset(s) to edit — one figure each.
Noneedits all managed datasets.- x_fieldstr
Field to use for the x-axis (default: ‘t2’).
- y_fieldstr
Field to use for the y-axis (default: ‘residuals’).
- titlestr, optional
Title override applied to every figure.
- reset(dataset_name: str = None)#
Clear all outlier flags.
- Parameters:
- dataset_namestr or None
If None, resets all datasets.
- residual_filter(dataset_name=None, mean_outlier: float = None, sigma_outlier: float = None)#
Flag measurements whose residuals fall outside a fixed interval.
A measurement is rejected when its residual lies outside
[mean_outlier - sigma_outlier, mean_outlier + sigma_outlier]. With neither bound given this is a 5-sigma test about the mean. Both statistics come from the points not already flagged: including known-bad points would inflate the spread and hide the outliers this is meant to find, which matters because flags accumulate across passes.The default is not robust. An outlier inflates the spread it is tested against, so a lone outlier among n - 1 tight points is only caught when n > 26, whatever its size. Use
huber_mad_filter()unless explicit bounds are what is wanted.Has no preview figure, like
date_range_filter.- Parameters:
- dataset_namestr, list of str, or None
Datasets to filter.
None= all.- mean_outlierfloat, optional
Centre of the accepted interval, in residual units. Defaults to the mean of the unflagged residuals.
- sigma_outlierfloat, optional
Half-width of the accepted interval, in residual units. Defaults to 5 times the standard deviation of the unflagged residuals.
Both are plain numbers by definition: they are compared directly with the
residualscolumn, which is in the units of the dataset’s measurements (km for range, Hz for Doppler, RU for sequential ranging, …). The same values are applied to every dataset filtered, so give explicit bounds one measurement type at a time (dataset_name).
- Returns:
- dict
{dataset_name: numpy.ndarray}of the indices flagged by this call, excluding any that were already flagged.
- Raises:
- KeyError
If a dataset has no
residualscolumn.
- summary()#
Print a per-dataset summary of flagged vs. total measurements.
- ISOLATION_FOREST_MIN_POINTS = 8#
Default fewest active, finite points the isolation-forest outlier filter runs on; below this it flags only the non-finite points. The starting
isolation_forest_min_pointsof an editor.
- PASS_GAP = 21600.0 sec#
Default gap that splits measurements into passes for the Huber/MAD outlier filter; the starting
pass_gapof an editor.Longer than any gap inside one tracking pass: over a 6.2 h pass the ranging has a median gap of 523 s and a largest one of 3300 s, the Doppler 60 s and 2498 s. A 300 s gap would cut that pass into 38 pieces.
- PASS_MIN_POINTS = 5#
Default fewest points a pass segment may have before the Huber/MAD filter merges it into its neighbour rather than detrending it on its own: a fit to a near-empty segment absorbs the outliers it should flag. The starting
pass_min_pointsof an editor.
- property datasets: list#
The list of managed datasets.
- Returns:
- list of MeasurementDataSet
Datasets as stored internally (flags not yet written back unless
apply_flags()orget_datasets()has been called).
- property isolation_forest_min_points: int#
Fewest active, finite points
isolation_forest_filter()runs on; below this it flags only the non-finite points. Starts asISOLATION_FOREST_MIN_POINTS.- Raises:
- InvalidIsolationForestSettingsError
If the value is not a whole number of at least 1.
- property pass_gap: ArrayWUnits#
Gap above which consecutive observations belong to separate passes in
huber_mad_filter(), each detrended on its own. Starts asPASS_GAP(6 h); raise or lower it for a mission whose tracking cadence differs from DSN deep-space passes.Set an
ArrayWUnitstime, or a bare number of seconds.- Raises:
- InvalidHuberMadSettingsError
If the value is not one finite, positive time.
- property pass_min_points: int#
Fewest points a pass may have in
huber_mad_filter()before it is merged into its neighbour instead of being detrended alone. Starts asPASS_MIN_POINTS.- Raises:
- InvalidHuberMadSettingsError
If the value is not a whole number of at least 1.
- exception InvalidHuberMadSettingsError#
Bases:
ValueErrorRaised when the Huber/MAD pass settings are not usable.
- exception InvalidIsolationForestSettingsError#
Bases:
ValueErrorRaised when the isolation-forest settings cannot produce a fit.