agglovar.merge.cumulative

A merging strategy that adds callsets cumulatively to the merge.

This strategy uses a table of variants that accumulates as callsets are added (the cumulative table). The cumulative table is initially empty. As callsets are added, variants intersecting the cumulative table are added to existing entries, and variants not intersecting the cumulative table are appended as new variants. This process is repeated for each callset added.

After all callsets are processed, the cumulative table represents a nonredundant callset where each entry is one variant that was found in one or more of the original callsets. Columns tracking the sources and the variant within each source.

This strategy is fast and uses minimal memory, but is not necessarily optimal. The order variants are input into the callset may alter the merged results in nontrivial ways, especially in loci where multiple join choices are possible.

Internally, source tracking is held in a long-form sidecar table (df_sources) that grows by row-append per iteration. The public list columns (mg_src, mg_stat) and the mg_src_lead index are materialized once at finalization via a single group_by(_mg_index).agg(...). Each mg_src entry carries src_index, src_name, src_meta, var_index, and var_id; mg_src_lead is the position within mg_src of the lead (representative) source entry. The cumulative variant table (df_cumulative) keeps (chrom, pos) ordering across iterations via merge_sorted on a composite _mg_sort key (rank(chrom) * pos_mult + pos) to avoid the per-iteration O(N log N) global sort. Position-sorted ordering within each chromosome keeps the pairwise join’s row-index chunks position-local so its per-stage df_b filter can prune on both sides.

Classes

LeadStrategy

Strategy for choosing the lead variant.

MergeCumulative

Iterative intersection.

Module Contents

class agglovar.merge.cumulative.LeadStrategy(*args, **kwds)

Bases: enum.Enum

Strategy for choosing the lead variant.

When variants from multiple sources join into one record, this strategy determines which variant is chosen as the lead variant. The lead variant represents the merged records in the merged callset.

FIRST = 'right'
LEFT = 'left'
class agglovar.merge.cumulative.MergeCumulative(pairwise_join: agglovar.pairwise.base.PairwiseJoin, lead_strategy: LeadStrategy = LeadStrategy.LEFT)

Bases: agglovar.merge.base.MergeBase

Iterative intersection.

Variables:

join – Pairwise join strategy for intersects.

__call__(callsets: collections.abc.Iterable[agglovar.merge.base.CallsetDefInputType], retain_index: bool = False, pre_filter: collections.abc.Iterable[polars.Expr] | polars.Expr | None = None, sort: bool = True, add_id: bool = True, temp_dir: bool | str | pathlib.Path = False, match_threads: int | None = 4) → polars.LazyFrame

Intersect callsets.

Parameters:

  • callsets – Callsets to intersect.

  • retain_index – If True, do not drop an existing “_index” column in callset tables if they exist.

  • pre_filter – If set, filter each table with these expressions. Filter is applied last (after “_index” is set).

  • temp_dir – Forwarded to the pairwise intersect for each cumulative step. See agglovar.pairwise.base.PairwiseJoin.join_iter().

  • match_threads – Number of worker processes used to score sequence matches (match_prop) during the merge. A single spawn-context process pool is created once and reused across all sources. Defaults to 4. If None, no pool is created and match scoring runs serially (turns multiprocessing off entirely). Ignored when the pairwise join does no sequence matching. In a spawn-restricted or resource-limited environment the pool degrades to serial scoring with a logged warning (see agglovar.pairwise.overlap.PairwiseOverlap.make_match_pool()).

Returns:

A merged callset table.

join: agglovar.pairwise.base.PairwiseJoin
lead_strategy
pairwise_join