mirar.pipelines package

Central location for all pipelines. This is where you should add new pipelines.

exception mirar.pipelines.PipelineConfigError

Bases: ProcessorError, KeyError

Error raised when a pipeline is not found

mirar.pipelines.get_pipeline(instrument: str, *args, selected_configurations=None, **kwargs) → Pipeline

Function to get pipeline

Parameters:

• instrument – Name of instrument

• args – args

• selected_configurations – Configurations to use

• kwargs – kwargs

Returns:

pipeline

Subpackages

• mirar.pipelines.gmos package
  • Subpackages
    • mirar.pipelines.gmos.config package
      • Submodules
      • mirar.pipelines.gmos.config.constants module
  • Submodules
  • mirar.pipelines.gmos.blocks module
  • mirar.pipelines.gmos.generator module
    • gmos_astrometric_catalog_generator()
    • gmos_photometric_catalog_generator()
    • gmos_reference_image_generator()
    • gmos_reference_image_resampler()
    • gmos_reference_psfex()
    • gmos_reference_sextractor()
    • gmos_sdss_reference_cat_purifier()
    • gmos_zogy_catalogs_purifier()
  • mirar.pipelines.gmos.gmos_mask module
    • generate_gmos_mask()
  • mirar.pipelines.gmos.gmos_pipeline module
    • GMOSPipeline
      • GMOSPipeline.all_pipeline_configurations
      • GMOSPipeline.download_raw_images_for_night()
      • GMOSPipeline.name
      • GMOSPipeline.non_linear_level
  • mirar.pipelines.gmos.load_gmos_image module
    • load_detrended_gmos_fits()
    • load_detrended_gmos_image()
• mirar.pipelines.lmi package
  • Subpackages
    • mirar.pipelines.lmi.config package
      • Submodules
      • mirar.pipelines.lmi.config.constants module
    • mirar.pipelines.lmi.generator package
      • lmi_astrometric_catalog_generator()
      • lmi_photometric_catalog_generator()
      • lmi_reference_image_generator()
      • lmi_reference_image_resampler()
      • lmi_reference_psfex()
      • lmi_reference_sextractor()
      • lmi_sdss_reference_cat_purifier()
      • lmi_zogy_catalogs_purifier()
      • Submodules
      • mirar.pipelines.lmi.generator.sources module
      • mirar.pipelines.lmi.generator.stacks module
      • mirar.pipelines.lmi.generator.target module
  • Submodules
  • mirar.pipelines.lmi.blocks module
  • mirar.pipelines.lmi.lmi_pipeline module
    • LMIPipeline
      • LMIPipeline.all_pipeline_configurations
      • LMIPipeline.default_cal_requirements
      • LMIPipeline.download_raw_images_for_night()
      • LMIPipeline.name
      • LMIPipeline.non_linear_level
  • mirar.pipelines.lmi.load_lmi_image module
    • load_raw_lmi_fits()
    • load_raw_lmi_image()
• mirar.pipelines.sedmv2 package
  • Subpackages
    • mirar.pipelines.sedmv2.config package
      • Submodules
      • mirar.pipelines.sedmv2.config.constants module
  • Submodules
  • mirar.pipelines.sedmv2.blocks module
  • mirar.pipelines.sedmv2.generator module
    • sedmv2_astrometric_catalog_generator()
    • sedmv2_color_function_ps1()
    • sedmv2_photcal_catalog_purifier()
    • sedmv2_photometric_catalog_generator()
    • sedmv2_reference_image_generator()
    • sedmv2_reference_image_resampler()
    • sedmv2_reference_psfex()
    • sedmv2_reference_sextractor()
    • sedmv2_zogy_catalogs_purifier()
  • mirar.pipelines.sedmv2.load_sedmv2_image module
    • clean_cal_header()
    • clean_science_header()
    • date_obs_to_mjd()
    • load_raw_sedmv2_mef()
    • load_sedmv2_mef_image()
  • mirar.pipelines.sedmv2.sedmv2_pipeline module
    • SEDMv2Pipeline
      • SEDMv2Pipeline.all_pipeline_configurations
      • SEDMv2Pipeline.default_cal_requirements
      • SEDMv2Pipeline.download_raw_images_for_night()
      • SEDMv2Pipeline.name
      • SEDMv2Pipeline.non_linear_level
• mirar.pipelines.summer package
  • Subpackages
    • mirar.pipelines.summer.config package
      • Submodules
      • mirar.pipelines.summer.config.constants module
    • mirar.pipelines.summer.models package
      • set_up_summer_databases()
      • Submodules
      • mirar.pipelines.summer.models.base_model module
  • Submodules
  • mirar.pipelines.summer.blocks module
  • mirar.pipelines.summer.generator module
    • summer_astrometric_catalog_generator()
    • summer_photometric_catalog_generator()
    • summer_reference_image_generator()
    • summer_reference_image_resampler()
    • summer_reference_psfex()
    • summer_reference_sextractor()
    • summer_zogy_catalogs_purifier()
  • mirar.pipelines.summer.load_summer_image module
    • load_proc_summer_image()
    • load_raw_summer_fits()
    • load_raw_summer_image()
  • mirar.pipelines.summer.summer_pipeline module
    • SummerPipeline
      • SummerPipeline.all_pipeline_configurations
      • SummerPipeline.default_cal_requirements
      • SummerPipeline.download_raw_images_for_night()
      • SummerPipeline.name
      • SummerPipeline.non_linear_level
      • SummerPipeline.set_up_pipeline()
• mirar.pipelines.wasp package
  • Subpackages
    • mirar.pipelines.wasp.config package
      • Submodules
      • mirar.pipelines.wasp.config.constants module
    • mirar.pipelines.wasp.generator package
      • wasp_astrometric_catalog_generator()
      • wasp_photometric_catalog_generator()
      • wasp_reference_image_generator()
      • wasp_reference_image_resampler()
      • wasp_reference_psfex()
      • wasp_reference_sextractor()
      • wasp_sdss_reference_cat_purifier()
      • wasp_zogy_catalogs_purifier()
      • Submodules
      • mirar.pipelines.wasp.generator.stacks module
      • mirar.pipelines.wasp.generator.target module
  • Submodules
  • mirar.pipelines.wasp.blocks module
  • mirar.pipelines.wasp.load_wasp_image module
    • load_raw_wasp_fits()
    • load_raw_wasp_image()
  • mirar.pipelines.wasp.wasp_pipeline module
    • WASPPipeline
      • WASPPipeline.all_pipeline_configurations
      • WASPPipeline.defalut_cal_requirements
      • WASPPipeline.download_raw_images_for_night()
      • WASPPipeline.name
      • WASPPipeline.non_linear_level
• mirar.pipelines.winter package
  • Subpackages
    • mirar.pipelines.winter.config package
    • mirar.pipelines.winter.generator package
      • Submodules
      • mirar.pipelines.winter.generator.astrometry module
      • mirar.pipelines.winter.generator.candidates module
      • mirar.pipelines.winter.generator.photometry module
      • mirar.pipelines.winter.generator.realbogus module
      • mirar.pipelines.winter.generator.reduce module
      • mirar.pipelines.winter.generator.references module
      • mirar.pipelines.winter.generator.utils module
    • mirar.pipelines.winter.models package
      • set_up_q3c()
      • set_up_winter_databases()
      • Submodules
      • mirar.pipelines.winter.models.base_model module
  • Submodules
  • mirar.pipelines.winter.blocks module
  • mirar.pipelines.winter.build_references module
    • dummy_split_image_batch_generator()
    • run_winter_reference_build_pipeline()
    • write_subfields_file()
  • mirar.pipelines.winter.constants module
  • mirar.pipelines.winter.fix_headers module
    • fix_headers()
  • mirar.pipelines.winter.fourier_bkg_model module
    • subtract_fourier_background_model()
  • mirar.pipelines.winter.ingest_references module
    • export_image_to_db()
    • get_logger()
  • mirar.pipelines.winter.load_winter_image module
    • annotate_winter_subdet_headers()
    • clean_header()
    • get_raw_winter_mask()
    • load_astrometried_winter_image()
    • load_raw_winter_mef()
    • load_stacked_winter_image()
    • load_test_winter_image()
    • load_winter_mef_image()
    • load_winter_stack()
  • mirar.pipelines.winter.nlc module
    • apply_winter_nlc()
  • mirar.pipelines.winter.run module
    • run_stack_of_stacks()
    • run_winter()
  • mirar.pipelines.winter.validator module
    • CondensationError
    • DarkOverSubtractionError
    • TooManyMaskedPixelsError
    • is_condensation_in_image()
    • masked_images_rejector()
    • poor_astrometric_quality_rejector()
    • winter_condensation_rejector()
    • winter_dark_oversubtraction_rejector()
  • mirar.pipelines.winter.winter_pipeline module
    • WINTERPipeline
      • WINTERPipeline.all_pipeline_configurations
      • WINTERPipeline.default_cal_requirements
      • WINTERPipeline.download_raw_images_for_night()
      • WINTERPipeline.name
      • WINTERPipeline.non_linear_level
      • WINTERPipeline.set_up_pipeline()
  • mirar.pipelines.winter.write_obslog module
    • write_observation_log()
• mirar.pipelines.wirc package
  • Subpackages
    • mirar.pipelines.wirc.generator package
      • Submodules
      • mirar.pipelines.wirc.generator.calibration module
      • mirar.pipelines.wirc.generator.imsub module
      • mirar.pipelines.wirc.generator.references module
      • mirar.pipelines.wirc.generator.stacks module
      • mirar.pipelines.wirc.generator.target module
    • mirar.pipelines.wirc.wirc_files package
  • Submodules
  • mirar.pipelines.wirc.blocks module
  • mirar.pipelines.wirc.load_wirc_image module
    • load_raw_wirc_fits()
    • load_raw_wirc_image()
  • mirar.pipelines.wirc.wirc_pipeline module
    • WircPipeline
      • WircPipeline.all_pipeline_configurations
      • WircPipeline.download_raw_images_for_night()
      • WircPipeline.name
      • WircPipeline.non_linear_level

Submodules

mirar.pipelines.base_pipeline module

Module containing the base of the Pipeline

class.

Each Pipeline will have several configurations. A configuration corresponds to a list of BaseProcessor objects.

The pipeline will process data using a chosen list of these individual BaseProcessor objects.

class mirar.pipelines.base_pipeline.Pipeline(selected_configurations: str | list[str] = 'default', night: int | str = '')

Bases: object

Base class for pipelines.

Each pipeline must have the following class variables:

• a name (the name of the instrument

• pipeline configurations

• gain

• a _load_raw_image() function

to load raw images and modify the headers etc as required

add_configuration(configuration_name: str, configuration: list[BaseProcessor])

Add a new configuration to the pipeline.

Parameters:

• configuration_name – Name of new configuration

• configuration – the list of processors

Returns:

None

property all_pipeline_configurations

Dictionary containing all pipeline configurations

static configure_processors(processors: list[BaseProcessor], sub_dir: str = '') → list[BaseProcessor]

Propagates the correct nightly setting to a list of processors.

Parameters:

• processors – Processors to configure

• sub_dir – night sub directory to use

Returns:

Updated processors

default_cal_requirements = None

static download_raw_images_for_night(night: str | int)

Function to download images from a remote server

Parameters:

night – Night of data to download

Returns:

None

get_error_output_path() → Path

Generates a unique path for the error summary, in the output data directory. Makes the parent directory structure if needed.

Returns:

path for error summary

get_flowchart_output_path() → Path

Generates a unique path for the flowchart summary, in the output data directory. Makes the parent directory structure if needed.

Returns:

path for error summary

get_latest_configuration() → list[BaseProcessor]

Get the latest configuration used by the pipeline

Returns:

list of processors

load_pipeline_configuration(configuration: str = 'default') → list[BaseProcessor]

Load a particular named configuration from self.all_pipeline_configurations

Parameters:

configuration – configuration to be used

Returns:

list of processors

load_raw_image(path: str) → ImageBatch

Function to load in a raw image and create an Image object which can then be processed further

Parameters:

path – path of raw image

Returns:

Image object

property name

Unique name of pipeline , used to call it from the command via get_pipeline(). Should be the name of the instrument.

property non_linear_level

Non-linear level of instrument

pipelines = {'git': <class 'mirar.pipelines.git.git_pipeline.GITPipeline'>, 'gmos': <class 'mirar.pipelines.gmos.gmos_pipeline.GMOSPipeline'>, 'lmi': <class 'mirar.pipelines.lmi.lmi_pipeline.LMIPipeline'>, 'sedmv2': <class 'mirar.pipelines.sedmv2.sedmv2_pipeline.SEDMv2Pipeline'>, 'summer': <class 'mirar.pipelines.summer.summer_pipeline.SummerPipeline'>, 'wasp': <class 'mirar.pipelines.wasp.wasp_pipeline.WASPPipeline'>, 'winter': <class 'mirar.pipelines.winter.winter_pipeline.WINTERPipeline'>, 'wirc': <class 'mirar.pipelines.wirc.wirc_pipeline.WircPipeline'>}

postprocess_configuration(errorstack: ErrorStack, selected_configurations: str | list[str], processed_images: list[str] | None = None) → list[BaseProcessor]

Generate a postprocessing/cleanup processor sequence, Used by Monitor class for realtime processing. The first step is to update the header of images with any saved errors in errorstack.

Parameters:

• errorstack – Caught errors

• selected_configurations – Configurations to use.

• processed_images – list of processed images

Returns:

list of postprocess processors

reduce_images(dataset: Dataset | None = None, output_error_path: str | None = None, catch_all_errors: bool = True, selected_configurations: str | list[str] | None = None) → tuple[Dataset, ErrorStack]

Function to process a given dataset.

Parameters:

• dataset – dataset to process (can be empty)

• output_error_path – optional path to write error summary

• catch_all_errors – Either catch errors, or just immediately raise them

• selected_configurations – Configuration to use

Returns:

Post-processing dataset and summary of errors caught

set_configuration(new_configuration: str = 'default') → list[BaseProcessor]

Loads a new configuration for the pipeline

Parameters:

new_configuration – name of configuration

Returns:

list of corresponding processors

set_up_pipeline()

Function to do any additional pipeline setup.

Returns:

None

unpack_raw_image(path: str) → Image | list[Image]

Function to load in a raw image and ensure it has

the correct format for the code.

This function, unlike _load_raw_image() is not protected.

Parameters:

path – path of raw image

Returns:

The image data and image header