logml.analysis.base_item

Functions

`load_analysis_metadata`(global_params)	Look for analysis metadata files, which describe results of high-level analysis.
`load_analysis_results`(metadata)	Load analysis results, which metadata is pointing to.

Classes

`AnalysisItem`(cfg[, global_params, logger, ...])	Base class for analysis steps.
`AnalysisMetadata`(analysis_name, status[, ...])	Analysis metadata.
`AnalysisPipelineContext`(steps, global_params)	Allows steps to load data from other steps without access to AnalysisRunner directly.
`AnalysisResult`([analysis_name, unique_name, ...])	Reportable summary of the analysis.
`ReleasePath`(src_path, dest_path)	Paths of artifacts to be released.

class logml.analysis.base_item.AnalysisMetadata(analysis_name: str, status: str, result_path: Optional[str] = None, unique_name: Optional[str] = None, report_link: Optional[str] = None)

Bases: object

Analysis metadata.

analysis_name: str

status: str

result_path: str = None

unique_name: str = None

report_link: str = None

class logml.analysis.base_item.AnalysisResult(analysis_name: Optional[str] = None, unique_name: Optional[str] = None, status: Optional[str] = None, **_kwargs)

Bases: object

Reportable summary of the analysis.

get_summary(short: bool = True) → Any

Get summary object.

Parameters: short – When true, return short (one-line) summary. Else return full summary.
Returns: format-able summary result.
Return type: object

dict(): Convert object to (json-dumpable) dictionary.

class logml.analysis.base_item.ReleasePath(src_path: Union[str, pathlib.Path], dest_path: Union[str, pathlib.Path])

Bases: object

Paths of artifacts to be released.

src_path: Union[str, pathlib.Path]

dest_path: Union[str, pathlib.Path]

class logml.analysis.base_item.AnalysisItem(cfg, global_params=None, logger=None, context: Optional[logml.analysis.base_item.AnalysisPipelineContext] = None, output_structure=None, unique_name=None, global_cfg=None, release_artifacts: bool = False, **kwargs)

Bases: object

Base class for analysis steps.

LABEL = None

PARAMS_CLS: Type = None

RESULT_CLS: Type = None

classmethod estimate_resources(res: JobResourcesReqs, cfg: logml.GlobalConfig = None, df: Optional[pd.DataFrame] = None, strata_shapes: Optional[Dict[str, tuple]] = None, item_params: Any = None) → None

Estimates resources for the step, result should be set to res: JobResourcesReqs.

Override in specific item classes to provide estimates. Default resources are: 1 CPU, 2Gb Mem.

NOTE from Slurm cluster: A memory size specification of zero is usually treated as a special case and grants the job access to all of the memory on each node. Never set memory limit to zero.

Default units are megabytes.

classmethod prepare_params(params: dict, global_cfg): Convert params, if they are not of PARAMS_CLS class.

abstract run(): Run analysis. Result is fetched later by get_result()

abstract get_result(): Return step final result. Should be of RESULT_CLS type, if RESULT_CLS is declared.

dump_metadata() → None: Dump finished analysis metadata, if needed.

dump_result(): Dumpt step result as pickle.

load_result(): Load serialized result.

generate_analysis_metadata() → Optional[logml.analysis.base_item.AnalysisMetadata]

Creates metadata object.

Major use for metadata object is to point to high-level analysis results (for example, ultimate result of Survival Modeling, as opposed to sub-level analysis step, like model search). Those results are then gathered and rendered at the summary report page.

get_paths_to_release() → Optional[List[logml.analysis.base_item.ReleasePath]]

Specify files/folders to be copied to the run release folder.

By default, returns current item’s output folder to be copied. Note that result path should be either subpath of release folder, or relative paths that will be placed into the release folder.

class logml.analysis.base_item.AnalysisPipelineContext(steps: Dict[str, logml.analysis.base_item.AnalysisItem], global_params: Dict)

Bases: object

Allows steps to load data from other steps without access to AnalysisRunner directly.

get_step_output(name): Load result of a step by name.

logml.analysis.base_item.load_analysis_metadata(global_params: dict) → List[logml.analysis.base_item.AnalysisMetadata]

Look for analysis metadata files, which describe results of high-level analysis.

Files produced by AnalysisItem.dump_metadata. No special naming convention is used: load all yaml files in the analysis_metadata folder. They point to result_path, which is the pickled dump of specific AnalysisResult subclass, which should contain all information needed to create analysis summary/result representation.

logml.analysis.base_item.load_analysis_results(metadata: List[logml.analysis.base_item.AnalysisMetadata]) → Dict[str, List[logml.analysis.base_item.AnalysisResult]]: Load analysis results, which metadata is pointing to.