Let's remember that Drumpf only 45 percent of the popular vote. Dubya in 2004, by contrast, fell just a bit short of 51 percent. Drumpf's approval ratings generally are in the high thirties, which is about where Dubya's were by 2006, but the decline in support was actual steeper and explains the swing that year. Also note that in the 2006 the Democratic Party had a majority favorability rating while the GOP's was low, whereas it's at forty percent now and about even with the GOP when last month. With a more modest decline of the Republican president's popularity and public equalization of the parties, Democratic gains will be modest and I strongly doubt they take back the House in 2018.
The problem with your point is you are comparing the dems right at the midterm vs the dems now only 6 months after the election. I wouldn't doubt the dems were not polling well in 05 after the Kerry loss