Adipocytes and cancer-associated adipocytes (CAAs) are poorly investigated cells in the tumor microenvironment. Different image analysis software exist for identifying and measuring these cells using scanned hematoxylin and eosin (H&E)-stained slides. It is however unclear which one is the most appropriate for breast cancer (BC) samples. Here, we compared three software (AdipoCount, Adiposoft, and HALO®). HALO® outperformed the other methods with regard to adipocyte identification, (> 96% sensitivity and specificity). All software performed equally good with regard to area and diameter measurement (concordance correlation coefficients > 0.97 and > 0.96, respectively). We then analyzed a series of 10 BCE samples (n = 51 H&E slides) with HALO®. Distant adipocytes were defined >2 mm away from cancer cells or fibrotic region, whereas CAAs as the first three lines of adipocytes close to the invasive front. Intra-mammary heterogeneity was limited, implying that measuring a single region of ∼500 adipocytes provides a reliable estimation of the distribution of their size features. CAAs had smaller areas (median fold-change: 2.62) and diameters (median fold-change: 1.64) as compared to distant adipocytes in the same breast (both p = 0.002). The size of CAAs and distant adipocytes was associated with the body mass index (BMI) of the patient (area: rho = 0.89, p = 0.001; rho = 0.71, p = 0.027, diameter: rho = 0.87 p = 0.002; rho = 0.65 p = 0.049, respectively). To conclude, we demonstrate that quantifying adipocytes in BC sections is feasible by digital pathology using H&E sections, setting the basis for a standardized analysis of mammary adiposity in larger series of patients.