Testing Calibration in Nearly-Linear Time
Gorde:
| Argitaratua izan da: | arXiv.org (Jun 21, 2024), p. n/a |
|---|---|
| Egile nagusia: | |
| Beste egile batzuk: | , , |
| Argitaratua: |
Cornell University Library, arXiv.org
|
| Gaiak: | |
| Sarrera elektronikoa: | Citation/Abstract Full text outside of ProQuest |
| Etiketak: |
Etiketarik gabe, Izan zaitez lehena erregistro honi etiketa jartzen!
|
MARC
| LEADER | 00000nab a2200000uu 4500 | ||
|---|---|---|---|
| 001 | 2929273483 | ||
| 003 | UK-CbPIL | ||
| 022 | |a 2331-8422 | ||
| 035 | |a 2929273483 | ||
| 045 | 0 | |b d20240621 | |
| 100 | 1 | |a Hu, Lunjia | |
| 245 | 1 | |a Testing Calibration in Nearly-Linear Time | |
| 260 | |b Cornell University Library, arXiv.org |c Jun 21, 2024 | ||
| 513 | |a Working Paper | ||
| 520 | 3 | |a In the recent literature on machine learning and decision making, calibration has emerged as a desirable and widely-studied statistical property of the outputs of binary prediction models. However, the algorithmic aspects of measuring model calibration have remained relatively less well-explored. Motivated by [BGHN23], which proposed a rigorous framework for measuring distances to calibration, we initiate the algorithmic study of calibration through the lens of property testing. We define the problem of calibration testing from samples where given \(n\) draws from a distribution \(\mathcal{D}\) on \((predictions, binary outcomes)\), our goal is to distinguish between the case where \(\mathcal{D}\) is perfectly calibrated, and the case where \(\mathcal{D}\) is \(\varepsilon\)-far from calibration. We make the simple observation that the empirical smooth calibration linear program can be reformulated as an instance of minimum-cost flow on a highly-structured graph, and design an exact dynamic programming-based solver for it which runs in time \(O(n\log^2(n))\), and solves the calibration testing problem information-theoretically optimally in the same time. This improves upon state-of-the-art black-box linear program solvers requiring \(\Omega(n^\omega)\) time, where \(\omega > 2\) is the exponent of matrix multiplication. We also develop algorithms for tolerant variants of our testing problem improving upon black-box linear program solvers, and give sample complexity lower bounds for alternative calibration measures to the one considered in this work. Finally, we present experiments showing the testing problem we define faithfully captures standard notions of calibration, and that our algorithms scale efficiently to accommodate large sample sizes. | |
| 653 | |a Lower bounds | ||
| 653 | |a Calibration | ||
| 653 | |a Algorithms | ||
| 653 | |a Linear programming | ||
| 653 | |a Machine learning | ||
| 653 | |a Prediction models | ||
| 700 | 1 | |a Jambulapati, Arun | |
| 700 | 1 | |a Tian, Kevin | |
| 700 | 1 | |a Yang, Chutong | |
| 773 | 0 | |t arXiv.org |g (Jun 21, 2024), p. n/a | |
| 786 | 0 | |d ProQuest |t Engineering Database | |
| 856 | 4 | 1 | |3 Citation/Abstract |u https://www.proquest.com/docview/2929273483/abstract/embedded/7BTGNMKEMPT1V9Z2?source=fedsrch |
| 856 | 4 | 0 | |3 Full text outside of ProQuest |u http://arxiv.org/abs/2402.13187 |