Previous research suggests that social relations are associated with age-related memory change. However, social relations are complex and multidimensional, and it is not yet clear which aspects (structure, quality) may be beneficial over time. Further, the strength and direction of associations may differ depending on relationship type (partner, children, other family, friends). Using longitudinal data from the Health and Retirement Study (n = 10,390; Mage = 69, SD = 9.53 at baseline), latent growth curve models tested which aspects of social relations predicted 6-year episodic memory trajectories. Both structure and quality of social relations were associated with initial memory level, such that being married/partnered, reporting more frequent contact with children and friends, reporting less support from family members other than partners and children, and reporting less strain across relationship types were each independently associated with better initial memory. In contrast, only structure was associated with subsequent memory decline. Specifically, being married/partnered and reporting more frequent contact with friends were each independently associated with slower memory decline. No evidence of bidirectionality was found, as baseline memory did not predict subsequent changes in social relations. This longitudinal study helps to clarify which aspects of social relations are most likely to influence late-life episodic memory trajectories. Future intervention studies may focus on increasing social interactions with nonfamily members to minimize memory decline with age.