While that clause did cost DC an EV in the elections before 1984, it's meaningless now unless the size of the House were to be considerably expanded. The district is fully developed now and there's no reasonable prospect that Congress would ever allow changes to the building codes which would allow high-rises to obscure the Capitol or the Mall. Max population for the district is probably 1,000,000 which won't ever be enough for two representatives unless we start seeing major population declines and/or a massive increase in the size of the House. (Even with a cube-root House size, DC would still only get one.)
You're saying, till 1980 D.C. ought to have had 4 EV, and it basically "lost" one since then?