MeasurementEditing#

class MeasurementEditing(datasets)#

Bases: object

Interactive and automated editing of measurement residuals for orbit determination.

Provides these editing modes:

  1. Lasso Selector — graphically select points to flag as outliers on a plot.

  2. Chi-Squared — automatically flag points whose residuals exceed a chi-squared threshold.

  3. Kurtosis — automatically trim the heaviest tails until the residual kurtosis falls below a target (3, the Gaussian value, by default).

  4. Date Range — manually specify time windows to flag or unflag observations.

  5. Isolation Forest — flag anomalous residuals by tree-isolation depth, with the rejection threshold taken from the knee of the ranked scores rather than from an assumed contamination fraction. Works jointly on the two components of a vector observable.

  6. Huber / MAD — detrend each pass with a Huber regression, then threshold on a robust MAD scale.

  7. Residual Interval — flag residuals outside a fixed interval, by default 5 sigma about the mean.

Parameters:
datasetslist of MeasurementDataSet

The datasets to edit. Nested lists are flattened one level.

Attributes:
datasets

The list of managed datasets.

isolation_forest_min_points

Fewest active, finite points isolation_forest_filter() runs on; below this it flags only the non-finite points.

outlier_flags

Dict of {dataset_name: bool array} of current outlier flags.

pass_gap

Gap above which consecutive observations belong to separate passes in huber_mad_filter(), each detrended on its own.

pass_min_points

Fewest points a pass may have in huber_mad_filter() before it is merged into its neighbour instead of being detrended alone.

refit_sigmas

Dict of {dataset_name: float} sigmas re-estimated by huber_mad_filter.

Methods

apply_flags()

Write the edited outlier_flag arrays back into each dataset's data dict.

chi_squared_filter([dataset_name, ...])

Automatically flag measurements whose normalised residuals exceed a chi-squared threshold.

chi_squared_filter_all([dataset_name, ...])

Chi-squared filtering with every preview figure shown AT ONCE.

date_range_filter(t_start, t_end[, ...])

Flag or unflag all measurements whose time falls within [t_start, t_end].

get_datasets()

Apply flags and return the list of edited datasets.

huber_mad_filter([dataset_name, n_sigma, ...])

Flag outliers by detrending residuals against time with a Huber regression, then thresholding on a MAD-based robust scale.

interactive_date_range([dataset_name])

Open an interactive GUI for manual date-range flagging and unflagging.

isolation_forest_filter([dataset_name, ...])

Flag anomalous residuals with an isolation forest, cutting at the knee of the ranked anomaly scores.

isolation_forest_filter_all([dataset_name, ...])

Isolation-forest filtering with every preview figure shown AT ONCE.

kurtosis_filter([dataset_name, ...])

Trim the most extreme normalised residuals until the sample kurtosis of the remaining points drops below target_kurtosis.

kurtosis_filter_all([dataset_name, ...])

Kurtosis filtering with every preview figure shown AT ONCE.

lasso_editor([dataset_name, x_field, ...])

Open an interactive matplotlib figure for lasso-based outlier selection.

lasso_editor_all([dataset_name, x_field, ...])

Open a SEPARATE lasso-editor figure for each dataset and display them all at once with a single plt.show().

reset([dataset_name])

Clear all outlier flags.

residual_filter([dataset_name, ...])

Flag measurements whose residuals fall outside a fixed interval.

summary()

Print a per-dataset summary of flagged vs. total measurements.

See also

scarabaeus.FilterDataManager

Combines datasets and applies editing before indexing.

scarabaeus.FilterOD

Base filter that consumes the edited datasets.

Notes

Edits are non-destructive: original outlier_flag arrays are copied internally and not written back until apply_flags() or get_datasets() is called.

Examples

editor = MeasurementEditing(datasets)
editor.lasso_editor(dataset_name="DSN_Range")
editor.chi_squared_filter(sigma_scale=3.0)
editor.kurtosis_filter(target_kurtosis=3.0)
editor.date_range_filter("2024-01-01T00:00:00", "2024-01-05T12:00:00")
edited_datasets = editor.get_datasets()

lasso_editor_all(), chi_squared_filter_all() and kurtosis_filter_all() are the all-figures-at-once counterparts of lasso_editor(), chi_squared_filter() and kurtosis_filter(), for editing several datasets without blocking on one figure at a time.

The pass splitting of huber_mad_filter() and the minimum sample of isolation_forest_filter() are set per editor through pass_gap, pass_min_points and isolation_forest_min_points, which start from the class defaults PASS_GAP, PASS_MIN_POINTS and ISOLATION_FOREST_MIN_POINTS.

apply_flags()#

Write the edited outlier_flag arrays back into each dataset’s data dict. Call this when satisfied with edits, before passing datasets to the filter.

chi_squared_filter(dataset_name: str = None, sigma_scale: float = 3.0, dof: int = 1, preview: bool = True)#

Automatically flag measurements whose normalised residuals exceed a chi-squared threshold.

Criterion: (residual / sigma)^2 > chi2.ppf(1 - alpha, dof) where alpha = 1 - CDF(sigma_scale^2, dof=1), equivalent to N-sigma.

Parameters:
dataset_namestr, list of str, or None

Name(s) of the dataset(s) to filter. If None, applies to all datasets.

sigma_scalefloat

Equivalent Gaussian sigma multiplier (e.g. 3.0 for 3-sigma rejection).

dofint

Degrees of freedom for chi-squared distribution (default 1).

previewbool

If True, show a summary plot before applying flags.

chi_squared_filter_all(dataset_name=None, sigma_scale: float = 3.0, dof: int = 1, preview: bool = True)#

Chi-squared filtering with every preview figure shown AT ONCE.

Same rejection criterion and same flag merging as chi_squared_filter() — the only difference is presentation: all preview figures are built first and displayed together by a single plt.show(), instead of one blocking figure per dataset.

Use this instead of calling chi_squared_filter() in a per-dataset loop. Looping blocks on each dataset’s plt.show(), which is clunky on remote/browser backends: closing a plot tab there does not advance the loop to the next dataset.

Flags are merged only after the previews are closed, so the previews still show the state before any flags are applied. Note the previews are informational, not a confirmation prompt — the flags are applied either way.

Parameters:
dataset_namestr or list of str or None

Dataset(s) to filter. None filters all managed datasets.

sigma_scalefloat

Equivalent Gaussian sigma multiplier (e.g. 3.0 for 3-sigma rejection).

dofint

Degrees of freedom for chi-squared distribution (default 1).

previewbool

If True, show the summary plots before applying flags.

date_range_filter(t_start, t_end, dataset_name: str = None, action: str = 'flag', time_field: str = 't2')#

Flag or unflag all measurements whose time falls within [t_start, t_end].

Parameters:
t_startfloat or str

Start of the window. Either a numeric epoch (seconds) or ISO 8601 string (e.g. “2024-01-01T00:00:00”).

t_endfloat or str

End of the window (same format as t_start).

dataset_namestr, list of str, or None

Dataset(s) to apply to. None applies to all datasets.

actionstr

"flag" marks measurements in the window as outliers; "unflag" removes the outlier flag from them.

time_fieldstr

Time field to use for comparison (default: ‘t2’).

get_datasets()#

Apply flags and return the list of edited datasets.

Returns:
list of MeasurementDataSet
huber_mad_filter(dataset_name=None, n_sigma: float = 5.0, mad_scale: float = 1.4826, refit_sigma: bool = True, pass_gap_sec: float = None, min_pass_points: int = None)#

Flag outliers by detrending residuals against time with a Huber regression, then thresholding on a MAD-based robust scale.

A tracking pass usually carries a slope that a plain mean-and-sigma test reads as spread, so the trend is removed first; the median absolute deviation then keeps the scale estimate from being dragged by the outliers it is meant to find. Unlike chi_squared_filter, this judges the residuals by their own scatter rather than by the sigma already attached to the measurements, which is what makes it usable before a trustworthy sigma exists.

Has no preview figure, like date_range_filter.

Parameters:
dataset_namestr, list of str, or None

Datasets to filter. None = all.

n_sigmafloat

Acceptance width, in units of the robust scale below.

mad_scalefloat

Factor turning the median absolute deviation into that scale. Defaults to 1.4826, which makes the MAD a consistent estimator of the standard deviation for normal data, so n_sigma reads as a number of sigmas. Pass 1.0 to threshold on the raw MAD instead.

refit_sigmabool

Re-estimate a measurement sigma from the surviving points and record it in refit_sigmas. The editor does not own the measurement models, so applying it is left to the caller.

pass_gap_secfloat, optional

Gap [s] above which consecutive observations belong to separate passes, each detrended on its own, for this call only. Defaults to pass_gap.

min_pass_pointsint, optional

Fewest points a pass may have before it is merged into its neighbour instead of being detrended alone, for this call only. Defaults to pass_min_points.

Returns:
dict

{dataset_name: float} of re-estimated sigmas, empty when refit_sigma is False.

interactive_date_range(dataset_name: str = None)#

Open an interactive GUI for manual date-range flagging and unflagging.

Displays a residuals scatter plot for the first targeted dataset and provides two text boxes (t_start, t_end) and three buttons:

  • Flag — apply date_range_filter() with action='flag' to all target datasets for the entered epoch range.

  • Unflag — apply date_range_filter() with action='unflag'.

  • Done — close the figure and return.

Both text boxes accept numeric epoch strings (seconds, e.g. "705427200.0") or ISO 8601 date strings (e.g. "2024-01-01T00:00:00").

The plot updates immediately after each Flag/Unflag action to reflect the current outlier state of the first dataset.

Parameters:
dataset_namestr, list of str, or None

Dataset(s) to target. If None, all managed datasets are targeted (the plot preview always shows the first one).

isolation_forest_filter(dataset_name=None, standardize: bool = True, scale_by_cos_dec: bool = False, n_estimators: int = 500, random_state: int = 0, preview: bool = True)#

Flag anomalous residuals with an isolation forest, cutting at the knee of the ranked anomaly scores.

An isolation forest scores a point by how few random splits it takes to separate it from the rest: outliers isolate quickly. Unlike chi_squared_filter() no sigma threshold is assumed, and unlike kurtosis_filter() no distribution shape is assumed — but the forest still has to be told how much to reject, and rather than fixing a contamination fraction the threshold is taken from the knee of the ranked scores (maximum drop below the endpoint chord), so the data decides where the tail begins.

A two-component observable is scored jointly: the two scalar datasets an optical track is split into are recombined into one 2-D point per observation, and a rejection flags both components — a bad image is bad in both coordinates, not in half of one.

Parameters:
dataset_namestr or list of str or None

Dataset(s) to filter. Components of the same observable are grouped automatically. If None, every dataset is considered.

standardizebool

Robustly rescale each coordinate by its median and MAD before fitting (default True). Without it the forest isolates on whichever component happens to be noisier.

scale_by_cos_decbool

For a right-ascension/declination pair, multiply the RA residual by cos(dec) so the pair is compared on the tangent plane. Ignored for any other observable. Defaults to False.

n_estimatorsint

Number of trees (default 500).

random_stateint

Seed, so a rejection is reproducible (default 0).

previewbool

If True, show the residuals and the knee before applying flags.

Raises:
InvalidIsolationForestSettingsError

If n_estimators is below 1.

Notes

An observable with fewer than isolation_forest_min_points (eight by default) active points is left untouched — a forest cannot say anything useful about a handful of points.

A statistical label is not proof that a measurement is wrong. Refit and recompute residuals after rejecting, and check that the solution is not unduly sensitive to what was cut.

References

[1]

Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008). Isolation Forest. Eighth IEEE International Conference on Data Mining, 413-422.

[2]

Satopää, V., Albrecht, J., Irwin, D., & Raghavan, B. (2011). Finding a “Kneedle” in a Haystack. 31st International Conference on Distributed Computing Systems Workshops, 166-171.

isolation_forest_filter_all(dataset_name=None, standardize: bool = True, scale_by_cos_dec: bool = False, n_estimators: int = 500, random_state: int = 0, preview: bool = True)#

Isolation-forest filtering with every preview figure shown AT ONCE.

Same rejection as isolation_forest_filter(); the figures are all built first and displayed by a single blocking plt.show() instead of one figure per observable. Flags are merged only after the previews are closed, so each preview still shows the state before anything was applied.

Parameters:
dataset_namestr or list of str or None

Dataset(s) to filter. See isolation_forest_filter().

standardize, scale_by_cos_dec, n_estimators, random_stateoptional

As in isolation_forest_filter().

previewbool

If True, show all preview figures together.

Raises:
MeasurementEditing.InvalidIsolationForestSettingsError

If n_estimators is less than 1.

kurtosis_filter(dataset_name: str = None, target_kurtosis: float = 3.0, max_fraction: float = 0.25, preview: bool = True)#

Trim the most extreme normalised residuals until the sample kurtosis of the remaining points drops below target_kurtosis.

A Gaussian residual population has kurtosis 3 (Pearson definition, i.e. zero excess kurtosis); anything above that is heavy-tailed, which for tracking residuals means outliers rather than genuine noise. Unlike chi_squared_filter(), no absolute sigma threshold is assumed — the data decides how much of the tail has to go.

Procedure: normalised residuals z = residual / sigma are ordered by |z - median(z)| (most extreme first) and removed one at a time; the kurtosis of the surviving points is evaluated after every removal, and the smallest number of removals that brings it to target_kurtosis is the one applied. Only points that are still active are considered — already-flagged points would inflate the kurtosis and drive over-trimming — and points with zero or non-finite sigma are ignored (never flagged), as in chi_squared_filter().

Parameters:
dataset_namestr, list of str, or None

Name(s) of the dataset(s) to filter. If None, applies to all datasets.

target_kurtosisfloat

Kurtosis to trim down to (default 3.0, the Gaussian value). Must be at least 1.0, the smallest kurtosis any distribution can have.

max_fractionfloat

Safety cap on the fraction of active points that may be removed (default 0.25). If the target is not reached within the cap, the capped number of points is flagged and a warning is printed.

previewbool

If True, show a summary plot before applying flags.

Raises:
InvalidKurtosisSettingsError

If target_kurtosis < 1 or max_fraction is outside [0, 1).

Notes

Datasets with fewer than four active points are left untouched: the fourth moment is not meaningful there.

kurtosis_filter_all(dataset_name=None, target_kurtosis: float = 3.0, max_fraction: float = 0.25, preview: bool = True)#

Kurtosis filtering with every preview figure shown AT ONCE.

Same trimming criterion and same flag merging as kurtosis_filter() — the only difference is presentation: all preview figures are built first and displayed together by a single plt.show(), instead of one blocking figure per dataset.

Use this instead of calling kurtosis_filter() in a per-dataset loop. Looping blocks on each dataset’s plt.show(), which is clunky on remote/browser backends: closing a plot tab there does not advance the loop to the next dataset.

Flags are merged only after the previews are closed, so the previews still show the state before any flags are applied. Note the previews are informational, not a confirmation prompt — the flags are applied either way.

Parameters:
dataset_namestr or list of str or None

Dataset(s) to filter. None filters all managed datasets.

target_kurtosisfloat

Kurtosis to trim down to (default 3.0, the Gaussian value).

max_fractionfloat

Safety cap on the fraction of active points that may be removed.

previewbool

If True, show the summary plots before applying flags.

lasso_editor(dataset_name=None, x_field: str = 't2', y_field: str = 'residuals', title: str = None)#

Open an interactive matplotlib figure for lasso-based outlier selection.

Left-click drag draws a lasso; enclosed visible points are toggled as outliers. Buttons:

  • [Confirm] — commit the current flags to the editor, hide the rejected points, and rescale the axes to the remaining points. The figure stays open so editing can continue.

  • [Show/Hide Rejected] — toggle visibility of rejected points (including points whose outlier_flag was already True when the editor opened), rescaling the axes each time. Hidden points cannot be lassoed.

  • [Reset] — clear all flags for the displayed dataset(s).

  • [Done] — commit and close the figure.

Multiple datasets of the same data type can be edited together (e.g. all range measurements regardless of which measurement model produced them) by passing a list of names or None for all managed datasets. Each dataset’s active points get their own color; flags are committed back to each dataset individually.

Parameters:
dataset_namestr or list of str or None

Name(s) of the dataset(s) to edit (must match MeasurementDataSet.set_name). None edits all managed datasets at once.

x_fieldstr

Field to use for x-axis (default: ‘t2’).

y_fieldstr

Field to use for y-axis (default: ‘residuals’).

titlestr, optional

Plot title override.

lasso_editor_all(dataset_name=None, x_field: str = 't2', y_field: str = 'residuals', title: str = None)#

Open a SEPARATE lasso-editor figure for each dataset and display them all at once with a single plt.show().

Each figure is fully interactive on its own (its own lasso plus the Confirm / Show-Hide Rejected / Reset / Done buttons); committing writes the flags back to that dataset. Control returns once every figure has been closed.

Use this instead of calling lasso_editor() in a per-dataset loop. Looping blocks on each dataset’s plt.show(), which is clunky on remote/browser backends: closing a plot tab there does not advance the loop to the next dataset. Here a single plt.show() drives every dataset’s editor together.

Parameters:
dataset_namestr or list of str or None

Dataset(s) to edit — one figure each. None edits all managed datasets.

x_fieldstr

Field to use for the x-axis (default: ‘t2’).

y_fieldstr

Field to use for the y-axis (default: ‘residuals’).

titlestr, optional

Title override applied to every figure.

reset(dataset_name: str = None)#

Clear all outlier flags.

Parameters:
dataset_namestr or None

If None, resets all datasets.

residual_filter(dataset_name=None, mean_outlier: float = None, sigma_outlier: float = None)#

Flag measurements whose residuals fall outside a fixed interval.

A measurement is rejected when its residual lies outside [mean_outlier - sigma_outlier, mean_outlier + sigma_outlier]. With neither bound given this is a 5-sigma test about the mean. Both statistics come from the points not already flagged: including known-bad points would inflate the spread and hide the outliers this is meant to find, which matters because flags accumulate across passes.

The default is not robust. An outlier inflates the spread it is tested against, so a lone outlier among n - 1 tight points is only caught when n > 26, whatever its size. Use huber_mad_filter() unless explicit bounds are what is wanted.

Has no preview figure, like date_range_filter.

Parameters:
dataset_namestr, list of str, or None

Datasets to filter. None = all.

mean_outlierfloat, optional

Centre of the accepted interval, in residual units. Defaults to the mean of the unflagged residuals.

sigma_outlierfloat, optional

Half-width of the accepted interval, in residual units. Defaults to 5 times the standard deviation of the unflagged residuals.

Both are plain numbers by definition: they are compared directly with the residuals column, which is in the units of the dataset’s measurements (km for range, Hz for Doppler, RU for sequential ranging, …). The same values are applied to every dataset filtered, so give explicit bounds one measurement type at a time (dataset_name).

Returns:
dict

{dataset_name: numpy.ndarray} of the indices flagged by this call, excluding any that were already flagged.

Raises:
KeyError

If a dataset has no residuals column.

summary()#

Print a per-dataset summary of flagged vs. total measurements.

ISOLATION_FOREST_MIN_POINTS = 8#

Default fewest active, finite points the isolation-forest outlier filter runs on; below this it flags only the non-finite points. The starting isolation_forest_min_points of an editor.

PASS_GAP = 21600.0 sec#

Default gap that splits measurements into passes for the Huber/MAD outlier filter; the starting pass_gap of an editor.

Longer than any gap inside one tracking pass: over a 6.2 h pass the ranging has a median gap of 523 s and a largest one of 3300 s, the Doppler 60 s and 2498 s. A 300 s gap would cut that pass into 38 pieces.

PASS_MIN_POINTS = 5#

Default fewest points a pass segment may have before the Huber/MAD filter merges it into its neighbour rather than detrending it on its own: a fit to a near-empty segment absorbs the outliers it should flag. The starting pass_min_points of an editor.

property datasets: list#

The list of managed datasets.

Returns:
list of MeasurementDataSet

Datasets as stored internally (flags not yet written back unless apply_flags() or get_datasets() has been called).

property isolation_forest_min_points: int#

Fewest active, finite points isolation_forest_filter() runs on; below this it flags only the non-finite points. Starts as ISOLATION_FOREST_MIN_POINTS.

Raises:
InvalidIsolationForestSettingsError

If the value is not a whole number of at least 1.

property outlier_flags: dict#

Dict of {dataset_name: bool array} of current outlier flags.

property pass_gap: ArrayWUnits#

Gap above which consecutive observations belong to separate passes in huber_mad_filter(), each detrended on its own. Starts as PASS_GAP (6 h); raise or lower it for a mission whose tracking cadence differs from DSN deep-space passes.

Set an ArrayWUnits time, or a bare number of seconds.

Raises:
InvalidHuberMadSettingsError

If the value is not one finite, positive time.

property pass_min_points: int#

Fewest points a pass may have in huber_mad_filter() before it is merged into its neighbour instead of being detrended alone. Starts as PASS_MIN_POINTS.

Raises:
InvalidHuberMadSettingsError

If the value is not a whole number of at least 1.

property refit_sigmas: dict#

Dict of {dataset_name: float} sigmas re-estimated by huber_mad_filter.

exception InvalidHuberMadSettingsError#

Bases: ValueError

Raised when the Huber/MAD pass settings are not usable.

exception InvalidIsolationForestSettingsError#

Bases: ValueError

Raised when the isolation-forest settings cannot produce a fit.

exception InvalidKurtosisSettingsError#

Bases: Exception

Raised when the kurtosis filter receives an unusable setting.