Preprocessing: choosing filters and transforms

MicrobiomeClassifier’s preprocessing="auto" applies three steps in order:

  1. AbundanceFilter — drop very-low-abundance taxa (min_abundance=1e-6).

  2. PrevalenceFilter — drop taxa present in too few samples (min_prevalence=0.01).

  3. CLRTransform — centered log-ratio transform (pseudocount=1e-6).

Those are the values "auto" uses. Note the standalone PrevalenceFilter class defaults to min_prevalence=0.05; "auto" deliberately sets a looser 0.01. Use explore() to pick these from your own data rather than trusting any default blindly.

Why filter at all?

Microbiome feature tables are wide and sparse (often 80–95% zeros). Most taxa are low-abundance, rarely observed, or sequencing noise / contaminants. Filtering:

  • removes features that carry little signal and inflate the multiple-comparison and overfitting burden;

  • stabilises the CLR transform, whose geometric mean is sensitive to a long tail of near-zero values.

Abundance cutoff

The log10 histogram of per-taxon mean abundance is typically bimodal — a low-abundance “noise” mode and a higher “signal” mode. The trough between them is the natural cutoff. mf.explore(dataset) draws this histogram with the proposed cutoff overlaid, so you can see whether 1e-6 sits in the trough for your data (it may not — relative-abundance vs. count tables differ by orders of magnitude).

  • Use a higher cutoff when you only trust well-quantified taxa, or the table is relative abundance already normalised to 1.

  • Use a lower cutoff for deeply-sequenced count data where rare taxa are still reliable.

Prevalence cutoff

Prevalence is the fraction of samples in which a taxon appears. The distribution is typically U-shaped (taxa present almost never or almost always). "auto" uses 0.01 (removes taxa seen in <1% of samples); the standalone class defaults to 0.05. Raise it (e.g. 0.1) when you want features informative across the cohort rather than a handful of samples — often the bigger lever on model stability than the abundance cutoff.

CLR transform

Microbiome abundances are compositional: they are relative (constrained to a constant sum), so only ratios between taxa carry information, not absolute values. The centered log-ratio (CLR) maps compositions to a real-valued space where standard classifiers and distances behave sensibly, by taking the log of each part divided by the sample’s geometric mean.

  • The pseudocount replaces zeros before the log. Its value affects how strongly zeros are pulled toward the low end; report it alongside results.

  • CLR keeps real taxon names, so counterfactuals and importances stay interpretable (see Counterfactual Explanations).

When not to CLR: if you plan to interpret absolute abundances directly, or your downstream model already assumes compositional structure, CLR may not be appropriate.

Known limitations

  • Zero handling is pseudocount-based, not model-based; results can be sensitive to the pseudocount on very sparse tables.

  • CLR only, no ILR/ALR. CLR components are not fully independent (they sum to zero by construction). An isometric log-ratio (ILR) basis avoids this but needs a chosen partition; it is on the roadmap.

  • Filtering thresholds are global, not per-class; extremely class-imbalanced taxa may be dropped. Inspect with mf.explore before committing.

Doing it manually

The transforms are plain scikit-learn transformers, so you can compose them yourself (and preserve real taxon names):

from microfactual import AbundanceFilter, PrevalenceFilter, CLRTransform

X = CLRTransform().fit_transform(
    PrevalenceFilter(min_prevalence=0.1).fit_transform(
        AbundanceFilter(min_abundance=1e-5).fit_transform(dataset.X)))