mirar.processors.database package

Central module for interactions with postgres databases

Submodules

mirar.processors.database.base_database_processor module

Module containing base database processor class

class mirar.processors.database.base_database_processor.BaseDatabaseProcessor(db_table: ~typing.Type[~mirar.database.base_model.BaseDB], pg_user: ~mirar.database.user.postgres_user.PostgresUser = <mirar.database.user.postgres_user.PostgresUser object>, pg_admin: ~mirar.database.user.postgres_admin.PostgresAdmin = <mirar.database.user.postgres_admin.PostgresAdmin object>)

Bases: BaseProcessor, ABC

Base class for processors which interact with a postgres database

max_n_cpu: int = 1

mirar.processors.database.database_inserter module

Module containing processors for exporting to databases

class mirar.processors.database.database_inserter.BaseDatabaseInserter(*args, duplicate_protocol: str = 'fail', **kwargs)

Bases: BaseDatabaseProcessor, ABC

Base class for DB inserter

base_key = 'dbinserter'

description()

Return a description of the processor

Returns:

A description of the processor

max_n_cpu: int = 1

class mirar.processors.database.database_inserter.DatabaseImageBatchInserter(*args, duplicate_protocol: str = 'fail', **kwargs)

Bases: DatabaseImageInserter

Processor for creating a single entry per batch of images in a database

check_prerequisites()

Check to see if any prerequisite processors are missing

Returns:

None

class mirar.processors.database.database_inserter.DatabaseImageInserter(*args, duplicate_protocol: str = 'fail', **kwargs)

Bases: BaseDatabaseInserter, BaseImageProcessor

Processor for exporting images to a database

static generate_value_dict(image: Image) → dict

Generate a dictionary of lower case keys and values from an image

Parameters:

image – Image

Returns:

Dictionary of lower case keys and values

class mirar.processors.database.database_inserter.DatabaseSourceInserter(*args, duplicate_protocol: str = 'fail', **kwargs)

Bases: BaseDatabaseInserter, BaseSourceProcessor

Processor for exporting sources to a database

exception mirar.processors.database.database_inserter.ImageBatchDatabaseExporterError

Bases: BaseProcessorError

Error for ImageBatchDatabaseExporter

mirar.processors.database.database_selector module

Module containing processors which import values from a database

class mirar.processors.database.database_selector.BaseDatabaseSelector(*args, boolean_match_key: str | None = None, **kwargs)

Bases: BaseDatabaseProcessor, ABC

Base Class for any database selector

base_key = 'dbselector'

get_constraints(data: dict) → None | DBQueryConstraints

Get db query constraints for a given datablock object

Parameters:

data – data block

Returns:

db query constraints object

class mirar.processors.database.database_selector.BaseDatabaseSourceSelector(db_output_columns: str | list[str], max_num_results: int | None = None, additional_query_constraints: DBQueryConstraints | None = None, **kwargs)

Bases: BaseDatabaseSelector, BaseSourceProcessor, ABC

Base Class for dataframe DB importers

query_for_source(source: Series, metadata: dict) → DataFrame

Query the database for a single source

Parameters:

• source – Source data

• metadata – Source Batch metadata

Returns:

Results from the database

update_dataframe(candidate_table: DataFrame, results: list[DataFrame]) → DataFrame

Update a dataframe with db results

Parameters:

• candidate_table – pandas table

• results – results from db query

Returns:

updated dataframe

class mirar.processors.database.database_selector.BaseImageDatabaseSelector(db_output_columns: str | list[str], output_alias_map: str | list[str] | None = None, update_header: ~collections.abc.Callable[[~mirar.data.image_data.Image, list[dict]], ~mirar.data.image_data.Image] = <function update_header_with_single_match>, **kwargs)

Bases: BaseDatabaseSelector, BaseImageProcessor, ABC

Processor to import data from images

class mirar.processors.database.database_selector.BaseSpatialCrossmatchSource(crossmatch_radius_arcsec: float, ra_field_name: str = 'ra', dec_field_name: str = 'dec', order_field_name: str | None = None, order_ascending: bool = False, query_dist: bool = False, **kwargs)

Bases: BaseDatabaseSourceSelector, ABC

Processor to crossmatch to sources in a database using spatial search

get_constraints(data: dict) → DBQueryConstraints

Get db query constraints for a given datablock object

Parameters:

data – data block

Returns:

db query constraints object

get_source_crossmatch_constraints(data: dict) → DBQueryConstraints

Apply constraints to a single source, using q3c

Parameters:

data – Dictionary containing source data

Returns:

DBQueryConstraints

class mirar.processors.database.database_selector.BaseValuesCrossmatch(db_query_columns: str | list[str], *args, **kwargs)

Bases: BaseDatabaseSelector, ABC

Processor to crossmatch to a database

get_accepted_values(data: dict) → list[str | float | int]

Get list of accepted values for crossmatch query

Parameters:

data – datablock

Returns:

accepted values from datablock

get_constraints(data: dict) → DBQueryConstraints

Get db query constraints for a datablock

Parameters:

data – datablock

Returns:

list of constraints

class mirar.processors.database.database_selector.CrossmatchDatabaseWithHeader(db_output_columns: str | list[str], output_alias_map: str | list[str] | None = None, update_header: ~collections.abc.Callable[[~mirar.data.image_data.Image, list[dict]], ~mirar.data.image_data.Image] = <function update_header_with_single_match>, **kwargs)

Bases: BaseImageDatabaseSelector, BaseValuesCrossmatch

Processor to crossmatch to a database using keys

description() → str

Return a description of the processor

Returns:

A description of the processor

class mirar.processors.database.database_selector.DatabaseHistorySelector(history_duration_days: float, time_field_name: str = 'jd', **kwargs)

Bases: SpatialCrossmatchSourceWithDatabase

Processor to import previous detections of a source from a database

get_constraints(data: dict) → DBQueryConstraints

Get db query constraints for a given datablock object

Parameters:

data – data block

Returns:

db query constraints object

class mirar.processors.database.database_selector.DatabaseMultimatchSelector(*args, base_output_column: str = 'prv_candidates', **kwargs)

Bases: BaseDatabaseSourceSelector, ABC

Processor to import multiple matches from a database

update_dataframe(candidate_table: DataFrame, results: list[DataFrame]) → DataFrame

Update a pandas dataframe with the number of matches

Parameters:

• candidate_table – Pandas dataframe

• results – db query results

Returns:

updated pandas dataframe

class mirar.processors.database.database_selector.DatabaseSingleMatchSelector(*args, **kwargs)

Bases: BaseDatabaseSourceSelector, ABC

Processor to import a single match from a database

update_dataframe(candidate_table: DataFrame, results: list[DataFrame]) → DataFrame

Update a dataframe with db results

Parameters:

• candidate_table – pandas table

• results – results from db query

Returns:

updated dataframe

class mirar.processors.database.database_selector.SelectSourcesWithMetadata(*args, base_output_column: str = 'prv_candidates', **kwargs)

Bases: DatabaseMultimatchSelector, BaseValuesCrossmatch

Processor to import sources from a database using metadata values

description() → str

Return a description of the processor

Returns:

A description of the processor

class mirar.processors.database.database_selector.SingleSpatialCrossmatchSource(crossmatch_radius_arcsec: float, ra_field_name: str = 'ra', dec_field_name: str = 'dec', order_field_name: str | None = None, order_ascending: bool = False, query_dist: bool = False, **kwargs)

Bases: BaseSpatialCrossmatchSource, DatabaseSingleMatchSelector

Processor to import a single source from a database using spatial crossmatch

description() → str

Return a description of the processor

Returns:

A description of the processor

class mirar.processors.database.database_selector.SpatialCrossmatchSourceWithDatabase(crossmatch_radius_arcsec: float, ra_field_name: str = 'ra', dec_field_name: str = 'dec', order_field_name: str | None = None, order_ascending: bool = False, query_dist: bool = False, **kwargs)

Bases: BaseSpatialCrossmatchSource, DatabaseMultimatchSelector

Processor to import multiple sources from a database using spatial crossmatch

description() → str

Return a description of the processor

Returns:

A description of the processor

mirar.processors.database.database_selector.update_header_with_single_match(data: DataBlock, res: list[dict]) → DataBlock

Update a datablock with a single db query result

Parameters:

• data – datablock to update

• res – corresponding db query

Returns:

updated datablock

mirar.processors.database.database_updater module

Module for processors to modify database entries

class mirar.processors.database.database_updater.BaseDatabaseUpdater(db_alter_columns: str | list[str], **kwargs)

Bases: BaseDatabaseSelector, ABC

Base class for database updaters

base_key = 'dbupdater'

class mirar.processors.database.database_updater.ImageDatabaseMultiEntryUpdater(sequence_key: str, **kwargs)

Bases: ImageSequenceDatabaseUpdater

Processor to modify multiple entries specified by a list of sequences in an image database

class mirar.processors.database.database_updater.ImageDatabaseUpdater(db_alter_columns: str | list[str], **kwargs)

Bases: BaseDatabaseUpdater, BaseImageDatabaseSelector, ABC

Base Class for updating image entries in a database

static generate_value_dict(image)

Get the value dictionary for an image

Parameters:

image – Image

Returns:

Value dictionary

class mirar.processors.database.database_updater.ImageSequenceDatabaseUpdater(sequence_key: str | list[str] | None = None, **kwargs)

Bases: ImageDatabaseUpdater

Processor to modify images in a database with a sequence

description() → str

Return a description of the processor

Returns:

A description of the processor

get_constraints(data) → DBQueryConstraints

Function to get the constraints for a database query

Parameters:

data – Image

Returns:

Constraints for a database query