Conformal Module#

The ex_fuzzy.conformal module provides conformal prediction utilities for fuzzy classifiers.

Overview#

This module wraps fuzzy classifiers to output prediction sets with coverage guarantees and supports evaluation of empirical coverage.

Classes#

ConformalFuzzyClassifier#

class ex_fuzzy.conformal.ConformalFuzzyClassifier(clf_or_nRules=None, score_type='membership', **kwargs)[source]#

Bases: ClassifierMixin, BaseEstimator

Conformal prediction wrapper for fuzzy rule-based classifiers.

Can wrap an existing BaseFuzzyRulesClassifier or create one internally using the same parameter signature. Provides prediction sets with statistical coverage guarantees.

Parameters:

  • clf_or_nRules (BaseFuzzyRulesClassifier or int, optional) – Either a trained classifier to wrap, or nRules parameter to create new one. If None, uses default nRules=30.

  • score_type (str, default='membership') – Nonconformity score type: - ‘membership’: 1 - membership degree of true class (default) - ‘association’: 1 - max association degree for rules of true class - ‘entropy’: Entropy of class probability distribution

  • **kwargs (dict) – Additional parameters passed to BaseFuzzyRulesClassifier if creating new. Common parameters include: nAnts, fuzzy_type, n_linguistic_variables, tolerance, verbose, backend, etc.

clf#

The underlying fuzzy classifier

Type:

BaseFuzzyRulesClassifier

score_type#

The type of nonconformity score being used

Type:

str

rule_base#

The rule base from the underlying classifier (property)

Type:

MasterRuleBase

nclasses_#

Number of classes (property)

Type:

int

Examples

Wrap an existing trained classifier:

>>> from ex_fuzzy.evolutionary_fit import BaseFuzzyRulesClassifier
>>> from ex_fuzzy.conformal import ConformalFuzzyClassifier
>>>
>>> clf = BaseFuzzyRulesClassifier(nRules=20)
>>> clf.fit(X_train, y_train, n_gen=50)
>>>
>>> conf_clf = ConformalFuzzyClassifier(clf)
>>> conf_clf.calibrate(X_cal, y_cal)
>>> pred_sets = conf_clf.predict_set(X_test, alpha=0.1)

Create classifier from scratch:

>>> conf_clf = ConformalFuzzyClassifier(nRules=20, nAnts=4)
>>> conf_clf.fit(X_train, y_train, X_cal, y_cal, n_gen=50)
>>> pred_sets = conf_clf.predict_set(X_test, alpha=0.1)

Get rule-wise explanations:

>>> results = conf_clf.predict_set_with_rules(X_test, alpha=0.1)
>>> for r in results:
...     print(f"Prediction set: {r['prediction_set']}")
...     print(f"Rule contributions: {r['rule_contributions']}")

A conformal wrapper that provides prediction sets and calibration utilities.

__init__(clf_or_nRules=None, score_type='membership', **kwargs)[source]#
fit(X, y, X_cal=None, y_cal=None, cal_size=0.2, **fit_kwargs)[source]#

Fit the classifier and calibrate for conformal prediction.

If X_cal/y_cal not provided, automatically splits X/y using cal_size.

Parameters:

  • X (array-like of shape (n_samples, n_features)) – Training data

  • y (array-like of shape (n_samples,)) – Training labels

  • X_cal (array-like of shape (n_cal_samples, n_features), optional) – Calibration data. If None, split from X using cal_size.

  • y_cal (array-like of shape (n_cal_samples,), optional) – Calibration labels. If None, split from y using cal_size.

  • cal_size (float, default=0.2) – Fraction of data to use for calibration if X_cal not provided

  • **fit_kwargs (dict) – Parameters passed to clf.fit() (n_gen, pop_size, etc.)

Returns:

self – The fitted conformal classifier

Return type:

ConformalFuzzyClassifier

Examples

>>> conf_clf = ConformalFuzzyClassifier(nRules=20)
>>> # Auto-split for calibration
>>> conf_clf.fit(X, y, cal_size=0.2, n_gen=50)

>>> # Or provide explicit calibration set
>>> conf_clf.fit(X_train, y_train, X_cal, y_cal, n_gen=50)
calibrate(X_cal, y_cal)[source]#

Calibrate conformal prediction thresholds using calibration set.

Computes nonconformity scores on the calibration set and stores them for computing p-values during prediction.

Parameters:

  • X_cal (array-like of shape (n_cal_samples, n_features)) – Calibration samples

  • y_cal (array-like of shape (n_cal_samples,)) – Calibration labels

Returns:

self – The calibrated conformal classifier

Return type:

ConformalFuzzyClassifier

Raises:

ValueError – If the underlying classifier has not been fitted yet

Examples

>>> # After fitting the base classifier separately
>>> conf_clf = ConformalFuzzyClassifier(trained_clf)
>>> conf_clf.calibrate(X_cal, y_cal)
predict(X)[source]#

Standard point prediction (delegates to wrapped classifier).

Parameters:

X (array-like of shape (n_samples, n_features)) – Test samples

Returns:

predictions – Predicted class labels

Return type:

ndarray of shape (n_samples,)

predict_set(X, alpha=0.1)[source]#

Predict conformal sets with coverage guarantee 1-alpha.

Returns prediction sets that, under exchangeability assumptions, contain the true label with probability at least 1-alpha.

Parameters:

  • X (array-like of shape (n_samples, n_features)) – Test samples

  • alpha (float, default=0.1) – Significance level. The coverage guarantee is 1-alpha. For example, alpha=0.1 gives 90% coverage.

Returns:

prediction_sets – For each sample, a set of class indices in the prediction set. Empty sets indicate high uncertainty for all classes. Sets with multiple classes indicate ambiguous predictions.

Return type:

list of sets

Raises:

ValueError – If the classifier is not calibrated

Examples

>>> pred_sets = conf_clf.predict_set(X_test, alpha=0.1)
>>> for i, pred_set in enumerate(pred_sets):
...     if len(pred_set) == 1:
...         print(f"Sample {i}: confident prediction {pred_set}")
...     elif len(pred_set) > 1:
...         print(f"Sample {i}: ambiguous between {pred_set}")
...     else:
...         print(f"Sample {i}: high uncertainty (empty set)")
predict_set_with_rules(X, alpha=0.1)[source]#

Predict conformal sets with rule-level explanations.

Returns prediction sets along with which rules contribute and their individual confidence levels. This provides explainable uncertainty quantification.

Parameters:

  • X (array-like of shape (n_samples, n_features)) – Test samples

  • alpha (float, default=0.1) – Significance level (0.1 = 90% coverage)

Returns:

results – Each result dict includes keys: ‘prediction_set’ (set of class indices), ‘rule_contributions’ (list of per-rule contribution dicts with ‘rule_index’, ‘class’, ‘firing_strength’, and ‘rule_confidence’), and ‘class_p_values’ (dict mapping class index to p-value).

Return type:

list of dicts

Raises:

ValueError – If the classifier is not calibrated

Examples

>>> results = conf_clf.predict_set_with_rules(X_test[:5], alpha=0.1)
>>> for i, r in enumerate(results):
...     print(f"Sample {i}:")
...     print(f"  Prediction set: {r['prediction_set']}")
...     print(f"  P-values: {r['class_p_values']}")
...     for contrib in r['rule_contributions'][:3]:
...         print(f"  Rule {contrib['rule_index']}: "
...               f"class={contrib['class']}, "
...               f"strength={contrib['firing_strength']:.3f}")
get_calibration_info()[source]#

Return calibration statistics for inspection.

Returns:

info – Dictionary containing: - ‘n_calibration_samples’: total calibration samples - ‘samples_per_class’: dict of samples per class - ‘score_type’: the nonconformity score type used - ‘n_rules_calibrated’: number of rules with calibration data

Return type:

dict

Examples

>>> info = conf_clf.get_calibration_info()
>>> print(f"Calibrated with {info['n_calibration_samples']} samples")
>>> print(f"Score type: {info['score_type']}")
print_rules(return_rules=False, bootstrap_results=False)[source]#

Print or return the rules from the underlying classifier.

set_fit_request(*, X_cal='$UNCHANGED$', cal_size='$UNCHANGED$', y_cal='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

  • X_cal (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for X_cal parameter in fit.

  • cal_size (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for cal_size parameter in fit.

  • y_cal (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for y_cal parameter in fit.

Returns:

self – The updated object.

Return type:

object

set_score_request(*, sample_weight='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.

Returns:

self – The updated object.

Return type:

object

evaluate_conformal_coverage#

ex_fuzzy.conformal.evaluate_conformal_coverage(conf_clf, X_test, y_test, alpha=0.1)[source]#

Evaluate conformal prediction coverage and efficiency.

Computes metrics to assess the quality of conformal predictions, including whether the coverage guarantee is satisfied and how informative the prediction sets are.

Parameters:

  • conf_clf (ConformalFuzzyClassifier) – A calibrated conformal classifier

  • X_test (array-like of shape (n_samples, n_features)) – Test samples

  • y_test (array-like of shape (n_samples,)) – True labels for test samples

  • alpha (float, default=0.1) – Significance level used for prediction sets

Returns:

metrics – Dictionary containing: - ‘coverage’: Empirical coverage (should be >= 1-alpha) - ‘expected_coverage’: The target coverage (1-alpha) - ‘avg_set_size’: Average prediction set size - ‘efficiency’: 1 / avg_set_size (higher is better) - ‘empty_sets’: Proportion of empty prediction sets - ‘singleton_sets’: Proportion of single-class sets - ‘coverage_by_class’: Per-class coverage rates

Return type:

dict

Examples

>>> metrics = evaluate_conformal_coverage(conf_clf, X_test, y_test, alpha=0.1)
>>> print(f"Coverage: {metrics['coverage']:.3f} (expected: {metrics['expected_coverage']:.3f})")
>>> print(f"Average set size: {metrics['avg_set_size']:.2f}")
>>> print(f"Singleton rate: {metrics['singleton_sets']:.2%}")

>>> # Check coverage guarantee
>>> if metrics['coverage'] >= metrics['expected_coverage'] - 0.05:
...     print("Coverage guarantee approximately satisfied")

Examples#

from ex_fuzzy.conformal import ConformalFuzzyClassifier, evaluate_conformal_coverage
from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=0)
X_cal, X_test, y_cal, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=0)

conf_clf = ConformalFuzzyClassifier(nRules=20, nAnts=4)
conf_clf.fit(X_train, y_train, X_cal, y_cal, n_gen=50, pop_size=50)

pred_sets = conf_clf.predict_set(X_test, alpha=0.1)
metrics = evaluate_conformal_coverage(conf_clf, X_test, y_test, alpha=0.1)