Well, yeah, it’s feasible. A fair bit of “Dark Age” construction was done by re-purposing dressed stone – which is an expensive and difficult to manufacture and transport materiel – from Roman infrastructure.
But whether that’s enough to dodge the curse, I don’t know.