Maybe a way to allocate the representatives proportionally in a more direct way would be to have a multi-tiered PR system with a party preferential ballot.
Say that you have 1,000 Representatives.
Now, you could do the following:
a) Allocate 500 to all parties proportionally, no threshold
b) Allocate 250 to all parties passing 5% of the vote
c) Allocate 200 to all parties passing 20% of the vote
d) Allocate 50 to the party with the most votes
Now, voters would need to rank the parties in order of preference, like Australia does for their Senate elections. In the first tier, there's no problem: every party gets a seat if they can get 1/500 of the votes (maybe less, depending on whether you're using St. Laguë, d´Hondt or the Hare Quota). But from the second tier on, if a small party doesn't reach the threshold, all those votes could go to a viable party. However, I don't know whether it should be batch-style or single-elimination (the latter could help small parties to coalesce their votes around a party that would have enough votes to be viable). This would continue for the third and fourth round.
