SelectArena
A modern benchmark for feature selection on real-world tabular data
Each feature selection method keeps a fixed budget of a dataset's features; a downstream model is then trained on the selected features only. Methods compete on the same dataset split, downstream model and budget, and the leaderboard ranks them by Elo. Pick a task type, downstream model or budget to see the leaderboard of that slice.
Elo rating of each feature selection method, from worst to best. Elo summarizes how often a method beats another on the same dataset, budget and downstream model (lower downstream error wins), with Random anchored at 1000 (dashed line). Error bars are 95% intervals from cluster-bootstrap resamples of the datasets. The left bar of each method is shaded by stability (how little the selected subset changes across bootstrap resamples of the data), the right bar by validity (how well injected noise features are rejected); darker is better for both. Hatched bars have no stability or validity result yet.
Mean rank of each method on each dataset, over splits, downstream models and budgets (1 is best).
These metrics come from runs of the selectors alone, without a downstream model.
- Stability (Nogueira, Sechidis & Brown, JMLR 2018): agreement of the selected features across bootstrap resamples of the data, corrected for chance; 1 means identical selections, 0 the agreement of random selections.
- Validity: share of the budget filled with original features after synthetic noise features are added (the noise level is the number of noise features per original one). The leaderboard and the figure use the validity at noise level 1.0.
Stability
Validity
How the leaderboard is computed
Methods are compared within a comparison group: one dataset split, one downstream model configuration and one feature budget.
- Elo: Bradley-Terry rating fit on all pairwise matches. Within each dataset, every combination of budget and downstream model (errors averaged over splits) is a setting in which every pair of methods plays a match, won by the lower error; ties are skipped. Random is anchored at 1000; a gap of 150 points means about 70% win probability. The 95% interval comes from cluster-bootstrap resamples of the datasets.
- Avg. rank: mean rank of the method's downstream error within the groups (1 is best).
- Win rate: share of pairwise comparisons won against the other methods (ties count half).
- Improvability: mean relative gap
(error − best error) / errorto the best method of the group (0 is best). - Error vs. all features: mean ratio of the error to the same model trained on all features. Below 1 means the selection improved the model.
- Stability and Validity: see the Stability & validity tab.
- Time: median wall-clock time of the selection. Fallback: share of runs in which the method ran out of time and random features filled the budget.
Budgets. Budget i of b keeps ceil(d · i / (b + 1))
of the d features when d ≤ 100, and ceil(d ^ (i / (b + 1))) otherwise.
Method types. The Type column follows the paper's functional taxonomy (Appendix A). Search methods evaluate many feature subsets with a model-dependent utility: forward search (SFS), backward search (SBE, MarkovBlanket) and leave-one-out (LOCO). Non-search methods avoid repeated subset evaluations and rely on a proxy instead: correlation (F-Test), distance ((R)ReliefF, LaplacianScore), information theory (MI, mRMR, GainRatio), trees (CART, RFImportance) and regularization (Lasso, ElasticNet). Random is the baseline.
Adding a method
Implement the method as described in the
SelectArena repository
(one AbstractFeatureSelector subclass) and open a pull request or an issue there.
Data
Everything shown here is plain files in this Space's data/ folder:
meta.json lists every published leaderboard, and each
one is a CSV, e.g. the
overall leaderboard.
Every table and figure also has its own CSV / SVG / PNG download button.