Automated Machine Learning (AutoML) systems are an important part of the machine learning engineering toolbox. However, existing solutions often require substantial amounts of compute, and therefore extended running times, to achieve competitive performance.

In this post, we evaluate Plexe, an AutoML framework designed to deliver competitive or superior predictive performance while demanding fewer computational resources compared to other popular frameworks. We use as our baseline performance results published in the 2024 paper by Gijsbers et al. In this post, we frequently refer to their methodology and findings, so we encourage you to check out their work.

Across a set of 20 benchmark datasets, relative to the AutoML tools we are evaluating against, Plexe consistently achieves superior or competitive prediction accuracy, while converging to a solution in a shorter wall time on comparable hardware.

While we are not yet ready to release Plexe itself to the world, the benchmark experiments themselves are fully reproducible, with code and data publicly accessible at GitHub – plexe-ai/plexe-results. We plan to release Plexe in the coming months.

Experimental Design

Baseline Selection

We compare against leading AutoML frameworks:

• AutoGluon(B)
• H2O AutoML
• LightAutoML
• FLAML
• AUTO-SKLEARN
• MLJAR
• TPOT

Evaluation Protocol

We evaluate Plexe following the same protocol described in the paper by Gijsbers et al:

• We selected 20 datasets from the OpenML benchmark suite
• We apply a 1-hour maximum runtime cutoff per dataset
• Standard metrics (AUC for classification, RMSE for regression)

Hardware Configuration

All experiments were conducted on standardized cloud infrastructure:

• 8 vCPUs
• 30GB RAM
• Standard cloud instance (Digital Ocean)

This is comparable to the hardware setup used by Gijsbers et al in their paper.

Results

Benchmark Performance

Results across the OpenML benchmark suite show consistent performance improvements:

Note: *Steel plates uses logloss metric (lower is better)’-‘ indicates framework failed to complete or produce valid results

Computational Efficiency

In the AutoML benchmark experiment, while a one-hour budget was set for each framework’s training, not all frameworks consistently used the full time. In fact, there were instances where frameworks finished well before the allotted hour, indicating variability in processing time requirements across frameworks and datasets. Additionally, certain frameworks occasionally exceeded the time budget by a few minutes, though significant overruns were rare and generally controlled within an acceptable range of leniency. In Plexe’s case, all best solutions were found well within the 1 hour limit, with 75% finishing in less than 30 minutes.

External Validation: Kaggle Competitions

Performance on Kaggle competitions provides independent validation and an example for a situation where Plexe can be immediately useful:

Note: * represents competition ongoing at the time of submission

Side note: Test set accuracy for the Titanic Survival dataset reported by MLJar (running 4 hours job on m5.24xlarge machine (96CPU, 384GB RAM)): 77.99% whereas Plexe reported an accuracy of 84.51%.(with a 1 hour limit running on the same hardware as defined on the top of this blog post)

Discussion

While these early results are only indicative, we do find them encouraging. In particular, we observed:

• Superior performance relative to established tools on 12/20 benchmark datasets
• Competitive performance (within 0.005) on remaining datasets
• Consistent results across diverse problem types

We acknowledge that the experiment has a number of limitations, which constrains the conclusions we can draw:

• Limited to tabular data
• Limited to 20 (OpenML) + 11 (Kaggle) tasks, all of which have “small” datasets
• Evaluation restricted to 1-hour time budgets
• Benchmark suite could be expanded to include more recent datasets

Nonetheless, we feel these initial results warrant further work on the tool. There are several directions that we want to explore:

• Extension to unstructured data types
• Support for deep learning models
• Expansion of benchmark suite with additional datasets

Conclusion

Our empirical evaluation of Plexe’s early prototype suggests that the tool is capable of achieving competitive or superior performance compared to current AutoML frameworks, while reducing computational requirements. This provides an encouraging foundation for us to continue building upon, and we plan to release a fully fledged beta to the public in the coming months. Stay tuned for updates!

References

Gijsbers, P., Bueno, J., Coors, S., LeDell, E., Poirier, S., Thomas, J., Bischl, B., & Vanschoren, J. (2023). AMLB: An AutoML Benchmark. 22-0493.pdf, benchmarking results and framework evaluation.

Source: Plexe: Production-ready custom AI from natural language