Identifying the relevant geography is an ongoing obstacle to effectively evaluating the influence of neighborhood built environment on physical activity. We characterized density and diversity of destinations that 77 older adults experienced within individually representative Global Positioning System (GPS) activity spaces and traditional residential buffers and assessed their associations with accelerometry-measured physical activity. Traditional residential buffers had lower destination density and diversity than activity spaces. Activity spaces based only on pedestrian and bicycling trips had higher destination densities than all-mode activity spaces. Regardless of neighborhood definition, adjusted associations between destinations and physical activity generally failed to reach statistical significance. However, within pedestrian and bicycling-based activity spaces, each additional destination type was associated with 243.3 more steps/day (95% confidence interval (CI) [36.0, 450.7]). Traditional buffers may not accurately portray the geographic space or neighborhood resources experienced by older adults. Pedestrian and bicycling activity spaces elucidate the importance of destinations for facilitating active transportation.