Michael Kattan Research Program

Research

Ever since my dissertation, “A Comparison of Machine Learning with Traditional Statistical Techniques,” I’ve had a long-standing interest in machine learning (ML) and artificial intelligence (AI).  At first, I compared AI with human experts [1] to better understand when one should outperform the other [2].  I then compared ML with traditional statistical techniques, similarly, trying to understand when one would prove superior.  I first developed a theoretical framework to describe the factors that should drive the performance in favor of or against ML in any given situation [3].  With the framework in place, I simulated data to illustrate the validity of this framework [4].  I later published a condensed illustration of the framework [5].   In wanting to apply ML in more varied applications, I noticed they were not well suited to handle time-until-event data and built these extensions [6,7].  With that in place, I was able to compare a variety of ML techniques with the standard statistical approach for time-until-event data, Cox proportional hazards regression [8].  What matters most is how well these ML and AI techniques fare in real-world data; to this end, I’ve studied their performances when predicting prostate cancer recurrence [9], clinical deterioration in the ward [10], and pelvic floor disorders after delivery [11].  In a recent comparative effectiveness study of bariatric surgery, we found that random forests were best for 2 of the models, but regression was superior for the remaining 6 models [12].  This disappointment for random forests led us to pursue enhancement, specifically, adding multi-objective particle swarm optimization (MOPSO).  We found the combination to outperform random forests alone [13], though random forests did well for us predicting progression of diabetic kidney disease [14].  More recently, when predicting good postoperative depth of focus after cataract surgery, extreme gradient boost outperformed logistic regression [15].  All of these complex issues regarding machine learning vs. statistical methods are discussed in our book [16].      

1. Kattan, M.W., Inductive expert systems vs. human experts. AI Expert, 1994: p. 32-38.
   
   
2. Kattan, M.W., D.A. Adams, and M.S. Parks, A Comparison of Machine Learning with Human Judgment. J Management Inf Sys, 1993. 9(4): p. 37-57.
   
   
3. Kattan, M.W. and R.B. Cooper, The predictive accuracy of computer-based classification decision techniques.  A review and research directions. Omega Int J Mgmt Sci, 1998. 26(4): p. 467-482.
   
   
4. Kattan, M.W. and R.B. Cooper, A simulation of factors affecting machine learning techniques: an examination of partitioning and class proportions. Omega Int J Mgmt Sci, 2000. 28: p. 501-512.
   
   
5. Kattan, M.W., Statistical prediction models, artificial neural networks, and the sophism "I am a patient, not a statistic". J Clin Oncol, 2002. 20(4): p. 885-887.
   
   
6. Zupan, B., (Kattan, M.W.)et al., Machine learning for survival analysis: a case study on recurrence of prostate cancer. Artif Intell Med, 2000. 20(1): p. 59-75.
   
   
7. Kattan, M.W., K.R. Hess, and J.R. Beck, Experiments to determine whether recursive partitioning (CART) or an artificial neural network overcomes theoretical limitations of Cox proportional hazards regression. Comput Biomed Res, 1998. 31: p. 363-373.
   
   
8. Kattan, M.W., Comparison of Cox regression with other methods for determining prediction models and nomograms. J Urol, 2003. 170(Supplement): p. S6-S10.
   
   
9. Cordon-Cardo, C., (Kattan, M.W.) et al., Improved prediction of prostate cancer recurrence through systems pathology. J Clin Invest, 2007. 117(7): p. 1876-1883.
   
   
10. Churpek, M.M., (Kattan, M. W.)et al., Multicenter Comparison of Machine Learning Methods and Conventional Regression for Predicting Clinical Deterioration on the Wards. Crit Care Med, 2016. 44(2): p. 368-74.
    
    
11. Jelovsek, J., (Kattan, M. W.)et al., Predicting risk of pelvic floor disorders 12 and 20 years after delivery. Am J Obstet Gynecol, 2018. 218(2): p. 222.e1-222.e19.
    
    
12. Aminian A, Zajichek A, Arterburn DE, Wolski KE, Brethauer SA, Schauer PR, Nissen SE, Kattan MW. Predicting 10-Year Risk of End-Organ Complications of Type 2 Diabetes With and Without Metabolic Surgery: A Machine Learning Approach. Diabetes Care. 2020 Apr;43(4):852-859. doi: 10.2337/dc19-2057. Epub 2020 Feb 6. Erratum in: Diabetes Care. 2020 Jun;43(6):1367. PMID: 32029638; PMCID: PMC7646205.
    
    
13. Asadi S, Roshan S, Kattan MW. Random forest swarm optimization-based for heart diseases diagnosis. J Biomed Inform. 2021 Feb 1;115:103690. doi: 10.1016/j.jbi.2021.103690. Epub ahead of print. PMID: 33540075.
    
    
14. Chan L, Nadkarni GN, Fleming F, McCullough JR, Connolly P, Mosoyan G, El Salem F, Kattan MW, Vassalotti JA, Murphy B, Donovan MJ, Coca SG, Damrauer SM. Derivation and validation of a machine learning risk score using biomarker and electronic patient data to predict progression of diabetic kidney disease. Diabetologia. 2021 Apr 2. doi: 10.1007/s00125-021-05444-0. Epub ahead of print. PMID: 33797560.
    
    
15. Liu Y, Wei D, Bai T, Luo J, Wood J, Vashisht A, Zhang S, Xuan J, Kattan M, Coplan P. Using machine learning to predict post-operative depth of focus after cataract surgery with implantation of Tecnis Symfony. Eur J Ophthalmol. 2021 Feb 2:1120672121991777. doi: 10.1177/1120672121991777. Epub ahead of print. PMID: 33530727.
    
    
16. Gerds TA and Kattan MW. (2021).  Medical Risk Prediction Models With Ties to Machine Learning.  CRC Press.

Selected Publications

Most of my statistical prediction modeling preferences are outlined in a recent book. Get it here.

RESEARCH INTERESTS

Inspired by personal frustrations with medical uncertainty, I am particularly interested in statistical prediction models and medical decision making.  The most common mistakes made, with ways to fix them, are discussed here.

A. Prediction Model Development

1. Here are the requirements for having a statistical prediction model endorsed by the American Joint Commission on Cancer.
2. Here is how we process data from Epic to make it research ready.
3. In the TRIPOD group, we came up with a checklist of what should be reported in a paper presenting a prediction model.
4. Making a prediction model when there is a time-varying covariate.
5. Propensity scores do not improve the accuracy of statistical prediction models.
6. Here's the code to make binary, ordinal, and survival outcome nomograms.
7. Here's how to make a competing risks regression nomogram. Detailed R code is here.
8. Machine learning approaches usually lose
   • to logistic regression
   • to Cox regression
   • again to Cox regression
   • and even when ML is RSF
   • and again

B. Prediction Model Assessment

1. Here's a decent way to compare two rival prediction tools that both predict on an ordinal scale.
2. How to make a calibration plot for a prediction model in the presence of competing risks.
3. How to estimate a time-dependent concordance index.
4. This is why you can't compare two prediction models tested on separate datasets. The figure is updated here.
5. A guide to the many metrics for assessing prediction models.
6. How to determine the area under the ROC curve for a binary diagnostic test.
7. The concordance index is not proper.  Use the Index of Predictive Accuracy (IPA) instead.
8. This is a framework for reviewers when evaluating statistical prediction modeling manuscripts.
9. Here are 8 common statistical prediction modeling mistakes, and how to fix them.

C. Prediction Communication and Interpretation

1. As cancer survivors, we like to think we both needed the treatment we received and were cured by it, but that is hard to prove.
2. Here's an example of how patients should be counseled: a table of tailored predictions of benefits and harms crossed by treatment options.
3. It is useless and confusing to put confidence intervals around a predicted probability.
4. You must apply a statistical prediction model to achieve informed consent.
5. What is a real nomogram anyway?
6. My definition of comparative effectiveness.
7. Too often we diagnose patients based on some arbitrary cutoff.  Let's stop doing that and recognize risk is on a continuum.
8. Don't just look at the p-value when judging a new marker.
9. Cancer staging systems need to go away.
10. "I'm a patient, not a statistic" is bogus.
11. Here is how we make our online risk calculators.
12. Patients found our risk calculator decision aid easy to use and useful.
13. Simplifying a regression model with friendly integers loses accuracy.
14. With biomarker studies, you should do this rather than report a p-value.

D. Predictions Doctors Make

1. The wisdom of crowds of doctors: averaging their individual predictions improves accuracy over the individuals themselves.
2. Probably due to cognitive biases, predicted probabilities coming from doctors are
   • not terribly accurate
   • highly variable
   • occasionally worse than a coin toss, also here.
   • and get worse when doctors choose to ignore what a prediction model says

E. Decision Analysis and Utility Assessment

1. The method used to measure utilities affects the decision analytic recommendation.
2. Unfortunately, you probably have to measure individual patient utilities to run a decision analysis on someone.
3. Stop multiplying health state utilities to get the utility of the combined health state.
4. Why utilities are more helpful than traditional health-related quality of life measures with respect to medical decision making.
5. The layout of the time trade-off is problematic.
6. How to measure standard gamble on paper.

F. Novel Uses of Prediction Models

1. Here's an example of how to make a synthetic control arm for a single arm study, using a prediction model.  Here's how to calculate the p-value.
2. Rather than running a decision analysis at the bedside, apply a nomogram instead -- much easier and same answer.

G. General Biostatistical Issues

1. Statistical reporting guidelines.
2. Guidelines for figures and tables.
3. Problems with p-values and ways to avoid using them.
4. The problem of causal language in observational studies.
5. Common problems with statistical prediction modeling papers.