I'm not really posting as my fantasy self here...
In my experience books are mostly purchases of the wealthier amongst us.
This situation will not improve if you artificially increase their price.
That's quite an elitist (no, I mean
actually elitist, not "elitist" in the sense it's often used in American political discourse post-1968 or so) statement, though you probably don't realise it. But read it again, and out of context, and you might.
The point is that books and reading ought not be restricted to certain social classes, that literacy is (or at least ought to be) a gateway to self-education and self-improvement and that a tax on books is (as the campaign against the Thatcher government's attempt to remove books from 0% VAT put it) a "tax on knowledge". Of course if you go even further back - one of the traditional calling-cards of a certain sort of reactionary regime was a tax on books and newspapers (guess why).
Making the situation even worse than it is (I remember reading an estimate on how many Americans read books on a regular basis and being pretty shocked) in order to fight the deficit is an unusual action for a liberal-ish government to take.
Besides, you'd get more money from taxing luxury goods.