logml.analysis.common
Classes
|
Defines output structure for DAG tracking data. |
|
Status of job completion. |
- class logml.analysis.common.DagOutputStructure(global_params: dict, *unused_args, **unused_kwargs)
Bases:
logml.common.OutputStructureDefines output structure for DAG tracking data.
- DATA_FOLDER: str = '_dag'
- STEPS_FOLDER: str = 'steps'
- PROFILE_FILENAME: str = 'dag.profile'
- DAG_CONFIG_FILENAME: str = 'dag.yaml'
- DAG_SHEDULE_FILENAME: str = 'schedule.json'
- DAG_IMAGE_FILENAME: str = 'dag.png'
- DAG_STATUS_FILENAME: str = 'status.json'
- get_job_result_filename(name: str) pathlib.Path
Generate file name for a job result
- class logml.analysis.common.JobResourcesReqs
Bases:
pydantic.main.BaseModelMinimal requirements for job resources.
Show JSON schema
{ "title": "JobResourcesReqs", "description": "Minimal requirements for job resources.", "type": "object", "properties": { "cpu": { "title": "Cpu", "default": 1, "type": "integer" }, "mem": { "title": "Mem", "default": -1, "type": "integer" }, "timeout": { "title": "Timeout", "default": 36000, "type": "integer" } } }
- Fields
- field cpu: int = 1
- field mem: int = -1
- field timeout: int = 36000
- class logml.analysis.common.JobStatus(value)
Bases:
str,enum.EnumStatus of job completion.
- WAIT_START = 'wait_start'
- OK = 'ok'
- FAILED = 'failed'
- DEPENDENCY_FAILED = 'dep_failed'
- RUNNING = 'running'
- class logml.analysis.common.JobResult
Bases:
pydantic.main.BaseModelResult of a job.
Show JSON schema
{ "title": "JobResult", "description": "Result of a job.", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "status": { "default": "wait_start", "allOf": [ { "$ref": "#/definitions/JobStatus" } ] }, "start_time": { "title": "Start Time", "anyOf": [ { "type": "string" }, { "type": "string", "format": "date-time" } ] }, "end_time": { "title": "End Time", "anyOf": [ { "type": "string" }, { "type": "string", "format": "date-time" } ] }, "duration": { "title": "Duration", "anyOf": [ { "type": "string" }, { "type": "number", "format": "time-delta" } ] }, "error": { "title": "Error", "type": "string" } }, "required": [ "name" ], "definitions": { "JobStatus": { "title": "JobStatus", "description": "Status of job completion.", "enum": [ "wait_start", "ok", "failed", "dep_failed", "running" ], "type": "string" } } }
- Fields
- field name: str [Required]
- field status: logml.analysis.common.JobStatus = JobStatus.WAIT_START
- field start_time: Optional[Union[str, pandas._libs.tslibs.timestamps.Timestamp]] = None
- field end_time: Optional[Union[str, pandas._libs.tslibs.timestamps.Timestamp]] = None
- field duration: Optional[Union[str, pandas._libs.tslibs.timedeltas.Timedelta]] = None
- field error: str = None
- dict(*args, **kwargs) dict
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
- reset() None
Reset state
- class logml.analysis.common.JobDefinition
Bases:
pydantic.main.BaseModelJob definition for the LogML DAG
Show JSON schema
{ "title": "JobDefinition", "description": "Job definition for the LogML DAG", "type": "object", "properties": { "unique_id": { "title": "Unique Id", "type": "string" }, "type": { "title": "Type", "type": "string" }, "depends_on": { "title": "Depends On", "default": [], "type": "array", "items": { "type": "string" } }, "resources": { "title": "Resources", "default": { "cpu": 1, "mem": -1, "timeout": 36000 }, "allOf": [ { "$ref": "#/definitions/JobResourcesReqs" } ] } }, "required": [ "unique_id", "type" ], "definitions": { "JobResourcesReqs": { "title": "JobResourcesReqs", "description": "Minimal requirements for job resources.", "type": "object", "properties": { "cpu": { "title": "Cpu", "default": 1, "type": "integer" }, "mem": { "title": "Mem", "default": -1, "type": "integer" }, "timeout": { "title": "Timeout", "default": 36000, "type": "integer" } } } } }
- Fields
- field unique_id: str [Required]
- field type: str [Required]
- field depends_on: List[str] = []
- field resources: logml.analysis.common.JobResourcesReqs = JobResourcesReqs(cpu=1, mem=-1, timeout=36000)
- class logml.analysis.common.JobsSchedule
Bases:
pydantic.main.BaseModelSchedule for LogML DAG
Show JSON schema
{ "title": "JobsSchedule", "description": "Schedule for LogML DAG", "type": "object", "properties": { "jobs": { "title": "Jobs", "type": "array", "items": { "$ref": "#/definitions/JobDefinition" } }, "common_params": { "title": "Common Params", "default": {}, "type": "object" } }, "required": [ "jobs" ], "definitions": { "JobResourcesReqs": { "title": "JobResourcesReqs", "description": "Minimal requirements for job resources.", "type": "object", "properties": { "cpu": { "title": "Cpu", "default": 1, "type": "integer" }, "mem": { "title": "Mem", "default": -1, "type": "integer" }, "timeout": { "title": "Timeout", "default": 36000, "type": "integer" } } }, "JobDefinition": { "title": "JobDefinition", "description": "Job definition for the LogML DAG", "type": "object", "properties": { "unique_id": { "title": "Unique Id", "type": "string" }, "type": { "title": "Type", "type": "string" }, "depends_on": { "title": "Depends On", "default": [], "type": "array", "items": { "type": "string" } }, "resources": { "title": "Resources", "default": { "cpu": 1, "mem": -1, "timeout": 36000 }, "allOf": [ { "$ref": "#/definitions/JobResourcesReqs" } ] } }, "required": [ "unique_id", "type" ] } } }
- field jobs: List[logml.analysis.common.JobDefinition] [Required]
- field common_params: dict = {}