One of the most concerning aspects of the ongoing COVID-19 pandemic is that it disproportionately affects people from some specific ethnic and socio-economic minorities. In particular, since from the beginning of the pandemic it has been clear that people from Black and African American backgrounds seem to be hit especially hard by the virus, creating a substantial infection gap. The observed abnormal impact on these ethnic groups could probably be due to the co-occurrence of other known risk factors, including co-morbidity, poverty, level of education, access to healthcare, residential segregation and response to cures, although those factors do not seem able to explain fully and in depth the excess incidence of infections and deaths among African Americans. Here, we introduce the concept of diffusion segregation, that is the extent to which a given group of people is internally clustered or exposed to other groups, as a result of mobility and commuting habits. By analysing census and mobility data on major US cities, we found that the weekly excess COVID-19 incidence and mortality in African American communities at the beginning of the COVID-19 pandemic is significantly associated with their level of diffusion segregation. The results confirm that knowing where people commute to, rather than where they live, is potentially much more important to contain and curb the spreading of infectious diseases.