Purpose Immune checkpoint inhibitors (ICIs) are now one of the standards of care for patients with lung cancer and have greatly improved both progression-free and overall survival, although <20% of the patients respond to the treatment, and some face acute adverse events. Although a few predictive biomarkers have integrated the clinical workflow, they require additional modalities on top of whole-slide images and lack efficiency or robustness. In this work, we propose a biomarker of immunotherapy outcome derived solely from the analysis of histology slides. Approach We develop a three-step framework, combining contrastive learning and nonparametric clustering to distinguish tissue patterns within the slides, before exploiting the adjacencies of previously defined regions to derive features and train a proportional hazards model for survival analysis. We test our approach on an in-house dataset of 193 patients from 5 medical centers and compare it with the gold standard tumor proportion score (TPS) biomarker. Results On a fivefold cross-validation (CV) of the entire dataset, the whole-slide image-based survival analysis for patients treated with immunotherapy (WhARIO) features are able to separate a low- and a high-risk group of patients with a hazard ratio (HR) of 2.29 (CI95=1.48 to 3.56), whereas the TPS 1% reference threshold only reaches a HR of 1.81 (CI95=1.21 to 2.69). Combining the two yields a higher HR of 2.60 (CI95=1.72 to 3.94). Additional experiments on the same dataset, where one out of five centers is excluded from the CV and used as a test set, confirm these trends. Conclusions Our uniquely designed WhARIO features are an efficient predictor of survival for lung cancer patients who received ICI treatment. We achieve similar performance to the current gold standard biomarker, without the need to access other imaging modalities, and show that both can be used together to reach even better results.