mirar.processors.sources package

Central module for candidate detection and extraction.

Subpackages

• mirar.processors.sources.machine_learning package
  • Submodules
  • mirar.processors.sources.machine_learning.pytorch module
    • Pytorch
      • Pytorch.base_key
      • Pytorch.description()
      • Pytorch.download_model()
      • Pytorch.get_ml_path()
      • Pytorch.get_model()
      • Pytorch.load_model()
  • mirar.processors.sources.machine_learning.xgboost module
    • XGBoost
      • XGBoost.base_key
      • XGBoost.description()
      • XGBoost.download_model()
      • XGBoost.get_ml_path()
      • XGBoost.get_model()
      • XGBoost.load_model()
• mirar.processors.sources.utils package
  • Submodules
  • mirar.processors.sources.utils.regions_writer module
    • RegionsWriter
      • RegionsWriter.base_key
      • RegionsWriter.description()

Submodules

mirar.processors.sources.csv_exporter module

Module with classes to write a source table to CSV

class mirar.processors.sources.csv_exporter.CSVExporter(output_dir_name: str | None = None, output_dir: str | Path = PosixPath('/home/docs'), export_keys: list[str] | None = None)

Bases: BaseSourceProcessor

Class to export a source table to CSV

base_key = 'CSVEXPORT'

description() → str

Return a description of the processor

Returns:

A description of the processor

mirar.processors.sources.edge_mask module

Edge mask processor

class mirar.processors.sources.edge_mask.EdgeSourcesMask(edge_boundary_size: float, image_xsize_column_key: str = 'X_SHAPE', image_ysize_column_key: str = 'Y_SHAPE', x_column_key: str = 'X_IMAGE', y_column_key: str = 'Y_IMAGE')

Bases: BaseSourceProcessor

Class to mask sources near the edge of the image

base_key = 'egdemask'

description() → str

Return a description of the processor

Returns:

A description of the processor

mirar.processors.sources.forced_photometry module

Module to extract a candidates table from an image header

class mirar.processors.sources.forced_photometry.ForcedPhotometryDetector(calculate_image_coordinates: bool = True, ra_header_key: str = 'ra', dec_header_key: str = 'dec', name_header_key: str | None = None)

Bases: BaseSourceGenerator

Class to create a candidates table for performing forced photometry

base_key = 'forcedphot'

description() → str

Return a description of the processor

Returns:

A description of the processor

exception mirar.processors.sources.forced_photometry.HeaderKeyMissingError

Bases: ProcessorError

Error relating to missing keys in headers

exception mirar.processors.sources.forced_photometry.TableFromHeaderError

Bases: ProcessorError

Error relating to writing a source table from a header

mirar.processors.sources.image_updater module

Module to update the header of images with data in a source table

class mirar.processors.sources.image_updater.ImageUpdater(modify_dir_name: str | None = None, input_dir: str | Path = PosixPath('/home/docs'), update_keys: list[str] | str | None = None, overwrite: bool = False, include_table: bool = True)

Bases: BaseSourceProcessor

Update the header of images with data in a source table

base_key = 'IMGUPDATE'

description() → str

Return a description of the processor

Returns:

A description of the processor

mirar.processors.sources.json_exporter module

Module with classes to write a source table to Json

class mirar.processors.sources.json_exporter.JSONExporter(output_dir_name: str | None = None, output_dir: str | Path = PosixPath('/home/docs'), export_keys: list[str] | None = None)

Bases: BaseSourceProcessor

Class to export a source table to JSON

base_key = 'JSONEXPORT'

description() → str

Return a description of the processor

Returns:

A description of the processor

mirar.processors.sources.json_loader module

Module with classes to read a source table from a json file

class mirar.processors.sources.json_loader.JSONLoader(input_dir_name: str | None = None, input_dir: str | Path = PosixPath('/home/docs'))

Bases: BaseSourceProcessor

Class to convert a parquet file to a source table

base_key = 'JSONLOAD'

description() → str

Return a description of the processor

Returns:

A description of the processor

mirar.processors.sources.json_loader.load_json_table(input_path: Path, metadata_key='metadata', source_key='sources') → SourceTable

Function to load a source table from a json file.

Parameters:

• input_path – Path to the json file

• metadata_key – Metadata key to use

• source_key – Source key to use

Returns:

SourceTable

mirar.processors.sources.namer module

Module containing a processor for assigning names to sources

class mirar.processors.sources.namer.CandidateNamer(base_name: str, name_start: str = 'aaaaa', db_name_field: str = 'objectid', name_key: str = 'objectid', **kwargs)

Bases: SingleSpatialCrossmatchSource

Processor to sequentially assign names to sources, of the form a, aa, aba…

base_key = 'namer'

description() → str

Return a description of the processor

Returns:

A description of the processor

extract_last_year(last_name: str) → int

Extract the year from the last name

Parameters:

last_name – last name

Returns:

year

get_next_name(detection_time: Time, last_name: str = None) → str

Function to get a new candidate name

Parameters:

• detection_time – detection time (Astropy Time object)

• last_name – last name

Returns:

new name

static increment_string(string: str)

Parameters

string

Returns

An incremented string, eg. aaa -> aab, aaz -> aba, azz -> baa, zzz-> aaaa

max_n_cpu: int = 1

mirar.processors.sources.parquet_loader module

Module with classes to read a source table from a parquet file

class mirar.processors.sources.parquet_loader.ParquetLoader(input_dir_name: str | None = None, input_dir: str | Path = PosixPath('/home/docs'))

Bases: BaseSourceProcessor

Class to convert a parquet file to a source table

base_key = 'PRQULOAD'

description() → str

Return a description of the processor

Returns:

A description of the processor

mirar.processors.sources.parquet_loader.load_parquet_table(input_path: Path, metadata_key='mirar_metadata') → SourceTable

Function to load a source table from a parquet file.

Parameters:

• input_path – Path to the parquet file

• metadata_key – Metadata key to use

Returns:

SourceTable

mirar.processors.sources.parquet_writer module

Module with classes to save a source table as a parquet file

class mirar.processors.sources.parquet_writer.NpEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: JSONEncoder

Class to encode numpy objects to json

default(o)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return super().default(o)

class mirar.processors.sources.parquet_writer.ParquetWriter(output_dir_name: str | None = None, output_dir: str | Path = PosixPath('/home/docs'))

Bases: BaseSourceProcessor

Class to save a source table as a parquet file

base_key = 'PARQUETWRITE'

description() → str

Return a description of the processor

Returns:

A description of the processor

mirar.processors.sources.parquet_writer.export_parquet(source_table: DataFrame, metadata: dict, parquet_path: Path)

Function to export a source table to parquet

Parameters:

• source_table – Table dataframe to export

• metadata – Metadata to add to the parquet file

• parquet_path – Output path

Returns:

None

mirar.processors.sources.sextractor_source_detector module

Module to detect candidates in an image

class mirar.processors.sources.sextractor_source_detector.SextractorSourceDetector(output_sub_dir: str = 'sources', target_only: bool = True)

Bases: BaseSourceGenerator

Class that retrieves a sextractor catalog and saves all sources to a sourcetable

base_key = 'DETSOURC'

check_prerequisites()

Check to see if any prerequisite processors are missing

Returns:

None

description() → str

Return a description of the processor

Returns:

A description of the processor

get_sub_output_dir() → Path

Returns: output sub-directory

mirar.processors.sources.sextractor_source_detector.generate_candidates_table(image: Image, sextractor_catalog_path: str | Path, target_only: bool = True) → DataFrame

Generate a candidates table from a sextractor catalog

Parameters:

• image – Image object

• sextractor_catalog_path – Path to the sextractor catalog

• target_only – Whether to return only the target source

Returns:

Candidates table

mirar.processors.sources.sextractor_source_detector.isolate_target(image: Image, sextractor_catalog: Table) → Table

Args:

image: Image object containing the target source coordinates sextractor_catalog: sextractor catalog as an astropy Table

Returns: Table with len=1, the target source from the sextractor catalog

mirar.processors.sources.source_detector module

Module to detect candidates in an image

class mirar.processors.sources.source_detector.ZOGYSourceDetector(cand_det_sextractor_config: str, cand_det_sextractor_filter: str, cand_det_sextractor_nnw: str, cand_det_sextractor_params: str, output_sub_dir: str = 'candidates', write_regions: bool = False, detect_negative_sources: bool = False)

Bases: BaseSourceGenerator

Class to detect candidates by running sourceextractor on a difference image and scorr image from ZOGY

base_key = 'DETCANDS'

check_prerequisites()

Check to see if any prerequisite processors are missing

Returns:

None

description() → str

Return a description of the processor

Returns:

A description of the processor

get_sub_output_dir() → Path

Get output sub-directory Returns:

mirar.processors.sources.source_detector.generate_candidates_table(diff: Image, scorr_catalog_path: str | Path, sci_resamp_image_path: str | Path, ref_resamp_image_path: str | Path, diff_scorr_path: str | Path, isdiffpos: bool = True) → DataFrame

Generate a candidates table from a difference image and scorr catalog :param diff: Difference image :param scorr_catalog_path: Path to the scorr catalog :param sci_resamp_image_path: Path to the resampled science image :param ref_resamp_image_path: Path to the resampled reference image :param diff_scorr_path: Path to the scorr image :param isdiffpos: Is the difference image positive? :return: Candidates table

mirar.processors.sources.source_exporter module

Module with classes to write a source table

class mirar.processors.sources.source_exporter.SourceWriter(output_dir_name: str | None = None, output_dir: str | Path = PosixPath('/home/docs'))

Bases: BaseSourceProcessor

Class to write a source table

base_key = 'SRCWRITE'

description() → str

Return a description of the processor

Returns:

A description of the processor

mirar.processors.sources.source_exporter.save_source_table(source_table: SourceTable, out_path: Path)

Function to save a source table to a json file.

Parameters:

• source_table – SourceTable to save

• out_path – Path to save to

Returns:

None

mirar.processors.sources.source_filter module

Base class for source filters

class mirar.processors.sources.source_filter.BaseSourceFilter

Bases: BaseSourceProcessor, ABC

Base class for source filters

base_key = 'filter'

mirar.processors.sources.source_loader module

Module with classes to read a source table from a json file

class mirar.processors.sources.source_loader.SourceLoader(input_dir_name: str | None = None, input_dir: str | Path = PosixPath('/home/docs'))

Bases: BaseSourceProcessor

Class to write a candidate table to a pandas dataframe

base_key = 'SRCLOAD'

description() → str

Return a description of the processor

Returns:

A description of the processor

mirar.processors.sources.source_loader.load_source_table(input_path: Path) → SourceTable

Function to load a source table from a pickle file.

WARNING: Usual rules with pickle apply. Only load files you trust.

Parameters:

input_path – Path to the pickle file

Returns:

SourceTable

mirar.processors.sources.source_selector module

Module containing processors and functions to select a subset of sources from a batch

class mirar.processors.sources.source_selector.SourceBatcher(split_key: str | list[str])

Bases: BaseSourceProcessor

Module to split SourceBatch object into multiple DataBatch objects.

Sources are batched using the split_key argument. For example, you can batch by filter, like this:

SourceBatcher(split_key=”filter”)

which will return N batches for the N different filters present in the directory you are reducing. If you do not require batching at some point in your reductions, you can split by BASE_NAME_KEY:

SourceBatcher(split_key=BASE_NAME_KEY)

which returns SourceBatches of length 1, one for each file in the directory you’re working with.

base_key = 'batch'

description() → str

Return a description of the processor

Returns:

A description of the processor

update_dataset(dataset: Dataset) → Dataset

Update a dataset after processing

Parameters:

dataset – Initial dataset

Returns:

Updated dataset

class mirar.processors.sources.source_selector.SourceDebatcher

Bases: BaseSourceProcessor

Processor to group all incoming SourceBatch objects into a single batch. This is helpful if you’ve already batched at an earlier stage in your workflow, and you want to start over and batch by a different split key.

base_key = 'debatch'

description() → str

Return a description of the processor

Returns:

A description of the processor

update_dataset(dataset: Dataset) → Dataset

Update a dataset after processing

Parameters:

dataset – Initial dataset

Returns:

Updated dataset

class mirar.processors.sources.source_selector.SourceRebatcher(split_key: str | list[str])

Bases: SourceBatcher

Processor to regroup all incoming SourceBatch objects into a single batch, and then split by new keys. This is helpful if you’ve already batched at an earlier stage in your workflow, and you want to start over and batch by a different split key.

base_key = 'rebatch'

description() → str

Return a description of the processor

Returns:

A description of the processor

update_dataset(dataset: Dataset) → Dataset

Update a dataset after processing

Parameters:

dataset – Initial dataset

Returns:

Updated dataset

class mirar.processors.sources.source_selector.SourceSelector(*args: tuple[str, str | list[str]])

Bases: BaseSourceProcessor, CleanupProcessor

Processor to only select a subset of sources from a batch. Sources can be selected using header keywords. For example, using:

SourceSelector((“OBSCLASS”, “SCIENCE”))

selects Sources with header[“OBSCLASS”]==”SCIENCE”

base_key = 'select'

description()

Return a description of the processor

Returns:

A description of the processor

mirar.processors.sources.source_selector.select_from_sources(batch: SourceBatch, key: str = 'TARGET', target_values: str | list[str] = 'science') → SourceBatch

Returns a subset of sources in a batch with have values of <key> equal to a value in <target values>

Parameters:

• batch – source batch to sort

• key – header key to filter on

• target_values – accepted value(s) for key

Returns:

source batch containing the subset of sources which pass

mirar.processors.sources.source_selector.split_sources_into_batches(sources: SourceBatch, split_key: str | list[str]) → Dataset

Function to split a single SourceBatch object into multiple DataBatch objects. Each new batch will have the same value of <split_key>. Returns a dataset containing the new batches

Parameters:

• sources – Source batch to split

• split_key – Key to split batch

Returns:

Dataset containing new source batches

mirar.processors.sources.source_table_modifier module

Module for modifying a source table.

class mirar.processors.sources.source_table_modifier.CustomSourceTableModifier(modifier_function: Callable[[SourceBatch], SourceBatch])

Bases: BaseSourceProcessor

Class to modify a source table based on a function

base_key = 'custom_source_modifier'

description() → str

Return a description of the processor

Returns:

A description of the processor