I don't think the tory vote in the likes of Esher will go back to the tories, and lack of fear for Corbyn there may push some voters to the libdems even more so.
"The likes of Esher?" Sure. But Esher itself swung harder into the LD camp than anywhere else in the country (bar North Down) and has not historically been LibDem (not since the late 1800s, IIRC). A lot of our overperformance there was due to #candidatequality and I would expect some local unwind if Monica Harding chose not to stand again.