The 2017 House GOP caucus was not nearly as pro-Trump as the 2021 one, so I'd say they'd try to go for Pence instead.
The problem is they can't. They are Constitutionally bound to pick one of the presidential candidates with the three most electoral votes. The Senate elects Vice President separately.
There's a nonzero chance of a McMullin/Pence administration in this scenario.
But probably they would still elect Trump to avoid pissing off their voters.
Pretty much this.
The only way the "solve" this issue with picking Pence would have been electing him vice president through the senate and the House not voting for a president. In this case, Pence would have assumed the role of acting president on January 20. Theoretically for the entire term, though the US formally would not have had a regular president then, just the vice president performing the powers and duties of the office. It would have been what happened in Season 5 in House of Cards, when Claire temporarily took over as acting president until Frank won the election by taking Ohio.
There would also be the risk of a Dem House coming in and electing Clinton in Jan 2019 if they picked up enough state delegations, which would bump Pence down to VP. Of course, nothing is stopping the lame duck GOP House from finally holding the vote and selecting Trump, but if they didn't have the votes in Jan 2017, they almost surely wouldn't have them in Dec 2018 after they just lost.
