I am not that knowledgeable on this subject, but I do think there should be a distinction between Jefferson and Madison. Madison always had some more "Federalist tendencies" than Jefferson, and overall appeared less ideological, from what I have come across.
I’ve read very good historians compare Jefferson and Madison to Hamilton and Adams. A loud man, with intelligence and spirit, and a quiet man, with genius and wisdom. To me, however, Madison is much more distinct from the likes of Hamilton, Adams, Jefferson, and Burr, in that he was less concerned with his actions than with their results.
Madison had a grand mind and a deep understanding of government, but he lacked the understanding of partisan politics. His writings and speeches reflect a great deal of capability, but his frustration with his limited powers, both as Governor and as President, overwhelmed him. He was therefore quite willing to agree to “Federalist economics” if he perceived them as centralizing the government and therefore securing the future unity and power of America.