Machine　learning　(ML)　has　transformed　neuroimaging　research　by　enabling　accurate　predictions　and　feature　extraction　from　large　datasets.　In　this　study,　we　investigate　the　application　of　six　ML　algorithms　(Lasso,　relevance　vector　regression,　support　vector　regression,　extreme　gradient　boosting,　category　boost,　and　multilayer　perceptron)　to　predict　brain　age　for　middle-aged　and　older　adults,　which　is　a　crucial　area　of　research　in　neuroimaging.　Despite　the　plethora　of　proposed　ML　models,　there　is　no　clear　consensus　on　how　to　achieve　better　performance　in　brain　age　prediction　for　this　population.　Our　study　stands　out　by　evaluating　the　impact　of　both　ML　algorithms　and　image　modalities　on　brain　age　prediction　performance　using　a　large　cohort　of　cognitively　normal　adults　aged　44.6　to　82.3　years　old　(N　=　27,842)　with　six　image　modalities.　We　found　that　the　predictive　performance　of　brain　age　is　more　reliant　on　the　image　modalities　used　than　the　ML　algorithms　employed.　Specifically,　our　study　highlights　the　superior　performance　of　T1-weighted　MRI　and　diffusion-weighted　imaging　and　demonstrates　that　multi-modality-based　brain　age　prediction　significantly　enhances　performance　compared　to　unimodality.　Moreover,　we　identified　Lasso　as　the　most　accurate　ML　algorithm　for　predicting　brain　age,　achieving　the　lowest　mean　absolute　error　in　both　single-modality　and　multi-modality　predictions.　Additionally,　Lasso　also　ranked　highest　in　a　comprehensive　evaluation　of　the　relationship　between　BrainAGE　and　the　five　frequently　mentioned　BrainAGE-related　factors.　Notably,　our　study　also　shows　that　ensemble　learning　outperforms　Lasso　when　computational　efficiency　is　not　a　concern.　Overall,　our　study　provides　valuable　insights　into　the　development　of　accurate　and　reliable　brain　age　prediction　models　for　middle-aged　and　older　adults,　with　significant　implications　for　clinical　practice　and　neuroimaging　research.　Our　findings　highlight　the　importance　of　image　modality　selection　and　emphasize　Lasso　as　a　promising　ML　algorithm　for　brain　age　prediction.　©　2023　by　the　authors.