Historically, oats were often used for fodder, as opposed to wheat which is generally more expensive to produce and retained for exclusive human consumption.
Indeed, there was a connotation between oats and poverty (in part, because oats can be grown on poor soil which tended to go to the poorest people) – in Ben Johnson’s Dictionary, the definition of oats was “a grain, which in England is generally given to horses, but in Scotland supports the people,” suggesting the poverty and backwardness of the Scots in his mind.
A quick look through A Search of Ice and Fire shows oats primarily being eaten in porridge and often given to animals; the oatcakes are primarily there as hard rations, as wheat-based cakes would go off quicker.