It depends a lot what period you’re talking about, because Roman tax policy changed a lot. The late Roman Republic did see a lot of tax exemptions and an insufficient bureaucracy which led to the rise of predatory tax farmers, but Augustus completely transformed the taxation system – establishing a population census and creating a corps of imperial bureaucrats to assess and collect taxes.
By the late Roman Empire, there was an increasing problem of increasing inequality leading to fewer free citizens paying taxes (and more and more slaves and serfs who were taxable property) and wealthy landowners bribing tax collectors, governors, and even emperors into letting them out of paying their taxes. (Incidentally, this contributed significantly to Rome’s increasing difficulties in keeping a standing army in the field.)