Tumor cells constantly interact with their microenvironment, which comprises a variety of immune cells together with endothelial cells and fibroblasts. The composition of the tumor microenvironment (TME) has been shown to influence response to immune checkpoint blockade (ICB). ICB takes advantage of immune cell infiltration in the tumor to reinvigorate an efficacious antitumoral immune response. In addition to tumor cell intrinsic biomarkers, increasing data pinpoint the importance of the TME in guiding patient selection and combination therapies. Here, we review recent efforts in determining how various components of the TME can influence response and resistance to ICB. Although a large body of evidence points to the extent and functional orientation of the T cell infiltrate as important in therapy response, recent studies also confirm a role for other components of the TME, such as B cells, myeloid lineage cells, cancer-associated fibroblasts, and vasculature. If the ultimate goal of curative cancer therapies is to induce a long-term memory T cell response, the other components of the TME may positively or negatively modulate the induction of efficient antitumor immunity. The emergence of novel high-throughput methods for analyzing the TME, including transcriptomics, has allowed tremendous developments in the field, with the expansion of patient cohorts, and the identification of TME-based markers of therapy response. Together, these studies open the possibility of including TME-based markers for selecting patients that are likely to respond to specific therapies, and pave the way to personalized medicine in oncology.