legend_data_monitor.excel package¶
Submodules¶
legend_data_monitor.excel.config_io module¶
Low-level I/O for legend-datasets status config files and validity.yaml.
- legend_data_monitor.excel.config_io._dump(data) str¶
Serialise data to a YAML string with double-quoted strings.
- Return type:
- legend_data_monitor.excel.config_io.append_to_config(path: Path, ged: str, entry: dict) None¶
Append a new detector block to a config file.
Raises ValueError if ged is already present in the file — call update_in_config instead.
- legend_data_monitor.excel.config_io.ensure_validity_entry(validity: list, timestamp: str, config_name: str, categories: list) bool¶
Insert or update a validity ‘append’ entry for config_name at timestamp.
The validity file looks as follows: - valid_from: YYYYMMDDTHHMMSSZ
- category:
all
cal
fft
mode: append apply:
l200-pXX-rYYY-T%-{all/phy}-config.yaml
If an entry already exists at timestamp, config_name is added to its apply list (if not already present). Otherwise a new entry is inserted in chronological order.
Returns True if validity was modified.
Call validity_blocked() first to ensure the timestamp is safe to write.
- Return type:
- legend_data_monitor.excel.config_io.read_config(path: Path) dict¶
Load a YAML config file. Returns {} if the file does not exist.
- Return type:
- legend_data_monitor.excel.config_io.remove_from_config(path: Path, ged: str) bool¶
Remove a detector block from a config file.
Returns True if the entry was found and removed, False if not present.
- Return type:
- legend_data_monitor.excel.config_io.remove_from_validity_apply(validity: list, timestamp: str, config_name: str) bool¶
Remove config_name from the apply list of the validity entry at timestamp.
Returns True if the list was modified. Does not remove the validity entry itself if the apply list becomes empty — leave that to a human.
- Return type:
- legend_data_monitor.excel.config_io.update_in_config(path: Path, ged: str, entry: dict) None¶
Update (or insert) a detector block in a config file.
Reads the full file, merges entry into the existing detector block and rewrites the file.
- legend_data_monitor.excel.config_io.validity_blocked(validity: list, timestamp: str) str | None¶
Return a reason if a new ‘append’ entry cannot safely be added at timestamp, or None if the timestamp is clear to write.
- Blocked conditions:
A ‘remove’ entry already exists at this timestamp.
More than one entry already exists at this timestamp (ambiguous merge).
- Return type:
str | None
legend_data_monitor.excel.core module¶
- legend_data_monitor.excel.core.expand_run_list(value: list | str) list[str]¶
Expand a YAML run value to a flat list of run strings.
- legend_data_monitor.excel.core.generate_dashboard(auto_dir_path: str, period: str, output: str) None¶
Generate the LEGEND usability dashboard for one period.
- legend_data_monitor.excel.core.get_geds(key: str, datasets_path: Path) dict[int, list[tuple[str, float]]]¶
- legend_data_monitor.excel.core.get_periods(key: str, datasets_path: Path) dict[str, list[tuple[str, str]]]¶
- legend_data_monitor.excel.core.get_runs_for_a_period(auto_dir_path: str, period: str) dict[str, list[tuple[str, str]]]¶
Discover all runs for period by scanning the DSP tier directories.
- Returns {period: [(run_type, run), …]} as:
cal r000, phy r000, cal r001, phy r001, …, cal rN
phy is only added when phy DSP data actually exists for that run.
legend_data_monitor.excel.detector_history module¶
Build per-detector transition histories from either the on-disk validity + config files, or from the Excel dashboard.
A Transition represents a genuine change in a detector’s usability at a specific (period, run, run_type). The ordered list of Transitions for a detector fully describes its history over the tracked periods.
- class legend_data_monitor.excel.detector_history.Transition¶
Bases:
objectA class to hold a bunch of info for a change in a detector’s usability at a specific run.
- legend_data_monitor.excel.detector_history._all_runinfo_entries(runinfo: dict) list[tuple[str, str, str]]¶
Return all (period, run, run_type) tuples across every period in runinfo, sorted chronologically by start_key.
Used by build_from_disk so that prev_val tracking is always correct regardless of which periods the Excel sheet covers.
- legend_data_monitor.excel.detector_history._find_source_and_reason(statuses_dir: Path, validity: list, timestamp: str, ged: str) tuple[str | None, str | None]¶
Scan validity entries at timestamp to find the config file that explicitly sets ged.
Returns (source_file_basename, reason). When multiple config files at the same timestamp contain ged, the last one wins (mirrors TextDB append order).
- legend_data_monitor.excel.detector_history._ordered_entries(periods: dict, runinfo: dict) list[tuple[str, str, str]]¶
Return all (period, run, run_type) tuples in the order they appear in periods, filtered to entries that exist in runinfo.
- legend_data_monitor.excel.detector_history.build_from_disk(datasets: Path, strings: dict, excel_periods: dict, runinfo: dict) tuple[dict, dict[str, list[Transition]]]¶
Build a per-detector transition list from the on-disk validity + config files.
Walks ALL periods in runinfo chronologically so that prev_val tracking is always correct — regardless of which periods the Excel sheet covers.
The returned usability dict is filtered to excel_periods only, so it can be compared directly with the usability dict from build_from_excel.
source_file and reason are read from the config file that explicitly sets the value at the transition timestamp. A Transition with source_file=None means the value is inherited — no config file at this exact timestamp.
- legend_data_monitor.excel.detector_history.build_from_excel(xlsx_path: str, strings: dict, periods: dict, runinfo: dict, prev_usab_seed: dict[str, str | None] | None = None) dict[str, list[Transition]]¶
Build a per-detector transition list from the Excel usability matrix.
Walks events in period-column order, reading each cell value. Records a Transition wherever the value changes relative to the previous event. Cell comments are read as the reason for that transition.
prev_usab_seed, if provided, is used to initialise the per-detector “last seen” value before the first Excel event. Pass the on-disk usability at the first Excel entry
- Return type:
dict[str, list[Transition]]
legend_data_monitor.excel.make_dashboard module¶
Excel dashboard generator.
Call make_excel(strings, periods, data, output_path).
Inputs¶
- stringsdict[int, list[tuple[str, float]]]
{string_number: [(ged_name, mass_g), ...]}— detectors in top-to-bottom order.- periodsdict[str, list[tuple[str, str]]]
{period: [(run_type, run), ...]}— columns in display order. The last run of each period should be cal-only (no trailing phy entry).- datadict[tuple, any]
{(string_num, ged_name, period, run, run_type, usability_type): value}where usability_type is"E"(energy scale) or"P"(PSD). Missing keys are treated as None (blank cell).
- legend_data_monitor.excel.make_dashboard._border(left=None, right=None, top=None, bottom=None) Border¶
- Return type:
Border
- legend_data_monitor.excel.make_dashboard._fill(hex_color: str) PatternFill¶
- Return type:
PatternFill
- legend_data_monitor.excel.make_dashboard._make_qcp_detail_sheet(work_book, sheet_name: str, run_type_filter: str, checks: list, strings: dict, periods: dict, qcp_data: dict) None¶
- legend_data_monitor.excel.make_dashboard._qcp_result(det_qcp: dict, run_type: str) tuple[str | None, list[str]]¶
Check all checks for a ged for a run type.
Returns (result, failed_checks) where result is “pass”, “fail”, or None. None means all checks were null (no data available).
- legend_data_monitor.excel.make_dashboard.add_summary_rows(work_sheet, strings: dict, periods: dict, col_index: dict, livetimes: dict) None¶
Append livetime and exposure summary rows below the detector data.
Livetime [days] — actual values, phy columns only Exposure ON [kg·yr] — Excel SUMPRODUCT formula Exposure AC [kg·yr] — Excel SUMPRODUCT formula Exposure OFF [kg·yr] — Excel SUMPRODUCT formula Exposure PSD valid+ON [kg·yr] — detectors with E=on AND P=valid
- legend_data_monitor.excel.make_dashboard.generate_period_colors(period_list: list[str]) dict[str, str]¶
Generate distinct colors for periods.
Preferred colors are used for known periods. Unknown periods get auto-generated colors distributed evenly around the color wheel.
- legend_data_monitor.excel.make_dashboard.make_excel(strings: dict, periods: dict, data: dict, output_path: str = 'dashboard_output.xlsx', livetimes: dict | None = None) None¶
Generate the usability dashboard Excel file.
- Parameters:
strings (see module docstring — detector layout per string)
periods (see module docstring — (run_type, run) columns per period)
data (see module docstring — usability values)
output_path (path to write the .xlsx file)
livetimes (optional {(period, run): livetime_in_seconds} — if supplied,) – summary exposure rows are appended below the detector data
- legend_data_monitor.excel.make_dashboard.make_qcp_sheet(work_book_path: str, strings: dict, periods: dict, qcp_data: dict) None¶
Add a ‘QCP Summary’ sheet to an existing workbook at wb_path.
One row per detector. Columns: cal r000, phy r000, cal r001, phy r001, … Each cell shows “pass” or “fail” if any fail — with a comment listing the failing check names.
- Parameters:
wb_path (path to an existing .xlsx file (produced by make_excel))
strings (same dict passed to make_excel)
periods (same dict passed to make_excel)
qcp_data ({period: {run: {detector: {"cal": {...}, "phy": {...}}}}}) – as returned by read_qcp.get_qcp_data()
legend_data_monitor.excel.read_qcp module¶
Reads QCP summary YAML files from monitoring/temp/.
- Returns qcp_data[period][run][detector][section][check] = True | False | None
section = “cal” or “phy” checks (cal) : FEP_gain_stab, fwhm_ok, npeak, const_stab, PSD checks (phy) : baseln_spike, baseln_stab, pulser_stab
- legend_data_monitor.excel.read_qcp.get_qcp_data(output: str, periods: dict) dict¶
Get QCP data for all periods and runs.
legend_data_monitor.excel.read_usability module¶
Reads escale-usability, PSD status, the reason field, and PSD notes from legend-datasets.
Main output is: data dict keys: (string_num, ged_name, period, run, run_type, field)
field = “E” -> “on” | “ac” | “off” | None field = “P” -> PSD status: “valid” | “present” | “missing” | None (complicated for various geds) field = “reason” -> str | None — only at the cal/phy event where it was written field = “PSD_note” -> str | None — formatted per-cut statuses, only at cal events
where PSD changed (or first cal run of period)
- PSD notes are:
low_aoe: valid high_aoe: present lq: valid ann: missing (redundant now) coax_rt: missing (redundant now)
- legend_data_monitor.excel.read_usability._build_psd_note_map(statuses: TextDB, runinfo: dict, periods: dict) dict¶
Build {(ged_name, cal_timestamp): note_str} for every cal run where PSD cut statuses changed since the previous cal run.
- Return type:
- legend_data_monitor.excel.read_usability._build_reason_map(datasets: Path, validity: list) dict¶
Build {(ged_name, valid_from_timestamp): reason} for entries where a config file explicitly sets a non-empty reason.
- Return type:
- legend_data_monitor.excel.read_usability._format_psd_note(cut_statuses: dict) str¶
Format {cut: status} as a multi-line note string.
- Return type:
- legend_data_monitor.excel.read_usability._psd_cut_statuses(ged_data: dict) dict | None¶
Return the full status dict {cut: status}, or None if no PSD block.
- Return type:
dict | None
- legend_data_monitor.excel.read_usability._psd_status(ged_data: dict) str | None¶
PSD status, or None if no PSD block.
- Return type:
str | None
- legend_data_monitor.excel.read_usability.correct_runinfo(datasets, run_info, period, run)¶
- legend_data_monitor.excel.read_usability.data(typ, tier, run, period, prod_cycle='auto/latest', server='nersc')¶
- legend_data_monitor.excel.read_usability.get_live_time(period, run)¶
- legend_data_monitor.excel.read_usability.get_run_start_timestamp(period, run, run_type)¶
- legend_data_monitor.excel.read_usability.get_usability_data(strings: dict, periods: dict, datasets: str, alter_mode: bool = False) dict¶
Build the usability data dict for the Excel dashboard.
- Parameters:
strings ({string_num: [(ged_name, mass_g), ...]})
periods ({period: [(run_type, run), ...]})
- Returns:
field = “E” → “on” | “ac” | “off” | None field = “P” → “valid” | “present” | “missing” | None field = “reason” → str | None (only where written) field = “PSD_note” → str | None (only at cal runs where PSD changed)
- Return type:
dict keyed by (string_num, ged_name, period, run, run_type, field)
- legend_data_monitor.excel.read_usability.write_runinfo(datasets, runinfo)¶
legend_data_monitor.excel.sync_to_datasets module¶
Sync the Excel usability dashboard back to legend-datasets.
Builds a Transition history from both the on-disk config files and the Excel sheet, diffs them per detector, and applies any differences.
Three kinds of change¶
ADD — Excel records a transition that disk does not -> write config + validity UPDATE — Both record a transition at the same run, but value or reason differs
-> update the existing source config file
- REMOVE — Disk records a transition that Excel does not want → remove detector
entry from its source config file
- legend_data_monitor.excel.sync_to_datasets._apply_adds(adds: list[Transition], statuses_dir: Path, validity: list, runinfo: dict) tuple[int, bool]¶
- legend_data_monitor.excel.sync_to_datasets._apply_removes(removes: list[Transition], statuses_dir: Path, validity: list) int¶
- Return type:
- legend_data_monitor.excel.sync_to_datasets._apply_updates(updates: list[tuple[Transition, Transition]], statuses_dir: Path, validity: list, runinfo: dict) tuple[int, bool]¶
- legend_data_monitor.excel.sync_to_datasets._build_entry(transition: Transition) dict¶
- Return type:
- legend_data_monitor.excel.sync_to_datasets._cal_config_removed_at_phy(period: str, run: str, validity: list, runinfo: dict) bool¶
Return True if validity already strips the cal-config at the phy start of this run.
When this is the case, a cal-only usability change should be written into the cal-config rather than the all-config: the remove entry will automatically revert the value for the phy run without needing a separate phy-config override.
- Return type:
- legend_data_monitor.excel.sync_to_datasets._diff(disk_usab: dict, excel_usab: dict, disk_hist: dict[str, list[Transition]], excel_hist: dict[str, list[Transition]], all_geds: list[str], excel_periods: dict, runinfo: dict) tuple[list, list, list]¶
Compare disk and Excel transition histories per detector.
Only ADD/UPDATE/REMOVE transitions that fall within the Excel period window. Disk transitions outside that window are never touched.
- Returns (adds, updates, removes) where:
adds — list[Transition] (Excel transition, no disk match) updates — list[tuple[Transition, Transition]] (disk, excel) removes — list[Transition] (disk transition, no Excel match)
- legend_data_monitor.excel.sync_to_datasets._is_reset_source(validity: list, config_name: str) bool¶
Return True if config_name appears in a mode:reset validity entry.
Reset configs are the canonical full-state documents written at period boundaries by the legend-datasets team. The sync tool must never remove individual detector entries from them — doing so leaves detectors with no status and causes blank cells in the dashboard.
- Return type:
- legend_data_monitor.excel.sync_to_datasets._needs_update(disk_t: Transition, excel_t: Transition) bool¶
Return True if the disk transition should be updated to match Excel.
A reason difference only triggers an update when Excel explicitly provides a reason — we never write a config entry purely to clear a reason that Excel left blank.
- Return type:
- legend_data_monitor.excel.sync_to_datasets._target_config_name(transition: Transition, validity: list, runinfo: dict) str¶
Derive the intended config file name for a new (ADD) transition.
Cal transitions go to all-config by default so the change persists into the phy run. Exception: if validity already has a remove entry that strips the cal-config at phy start, the cal change is cal-only and should live in the cal-config instead — the remove handles the revert.
- Return type: