Recent evidence indicates that thyroid hormones may be closely linked to cognition among adults. We investigated associations between thyroid hormones and longitudinal cognitive change, within and outside of reference ranges, stratifying by sex and race. This longitudinal study used data from the Healthy Aging in Neighborhoods of Diversity Across the Lifespan study, set in Baltimore City, MD, 2004-2013, on adults aged 30-64 years at baseline visit, with a length of follow-up between visits 1 and 2 ranging from <1 to 8 years; mean ± standard deviation: 4.64 ± 0.93. The final analytic sample sizes ranged from 1486 to 1602 participants with 1.6-1.7 visits per participant (total visits: 2496-2757), depending on the cognitive test. Eleven cognitive test scores spanning domains of learning or memory, language or verbal, attention, visuospatial and/or visuoconstruction, psychomotor speed, executive function, and mental status were used. Mixed-effects regression models were conducted, interacting time of follow-up with several thyroid exposures. Whites performed better than African Americans, with only 4 cognitive test scores of 11 declining significantly over time. Importantly, above reference range thyroid stimulating hormone (vs. reference range, thyroid stimulating hormone, above reference range [TSHarr]) was linked to faster rates of decline on the digits span backwards test, reflecting working memory (TSHarr × time γ ± standard error: −0.14 ± 0.05, p = 0.006) and clock-command, at test of visuospatial and/or visuoconstruction abilities (TSHarr × Time γ ± standard error: −0.10 ± 0.04, p = 0.004). The latter finding was replicated when comparing normal thyroid function to "subclinical hypothyroidism". Within-reference ranges, a higher thyroid stimulating hormone was related to faster decline on the clock-command test scores in women. In sum, higher baseline thyroid stimulating hormone was associated with faster cognitive decline over-time among urban US adults, specifically in domains of working memory and visuospatial and/or visuoconstruction abilities.