Both social and genetic factors contribute to cognitive impairment and decline, yet genetic factors identified through genome-wide association studies (GWAS) explain only a small portion of trait variability. This “missing heritability” may be due to rare, potentially functional, genetic variants not assessed by GWAS, as well as gene-by-social factor interactions not explicitly modeled. Gene-by-social factor interactions may also operate differently across race/ethnic groups. We selected 39 genes that had significant, replicated associations with cognition, dementia, and related traits in published GWAS. Using gene-based analysis (SKAT/iSKAT), we tested whether common and/or rare variants were associated with episodic memory performance and decline either alone or through interaction with education in >10,000 European ancestry (EA) and >2200 African ancestry (AA) respondents from the Health and Retirement Study (HRS). Nine genes in EA and five genes in AA were associated with memory performance or decline (p < 0.05), and these effects did not attenuate after adjusting for education. Interaction between education and CLPTM1 on memory performance was significant in AA (p = 0.003; FDR-adjusted p = 0.038) and nominally significant in EA (p = 0.026). In both ethnicities, low memory performance was associated with CLPTM1 genotype (rs10416261) only for those with less than high school education, and effects persisted after adjusting for APOE ε4. For over 70% of gene-by-education interactions across the genome that were at least nominally significant in either ethnic group (p < 0.05), genetic effects were only observed for those with less than high school education. These results suggest that genetic effects on memory identified in this study are not mediated by education, but there may be important gene-by-education interactions across the genome, including in the broader APOE genomic region, which operate independently of APOE ε4. This work illustrates the importance of developing theoretical frameworks and methodological approaches for integrating social and genomic data to study cognition across ethnic groups.