‘With Brexit hanging like a lead weight above the city’s new forest of glass towers, the memory of the Horse Hospital’s stone floor feels grounding’