Let's say Pence just stopped to honor Payton Manning, which was what much of the day was about, but didn't attend the game and just went on to his next destination. Would there be a huge flap about the cost? I don't think so since prior to him walking out of the game, no articles were published about the OMGZ cost of honoring Payton.
When it came out that he made a cross-country trip solely for that purpose, I'd guess that there probably would have been something of a flap (*cough* party of fiscal responsibility *cough*), but likely not as big a flap as it is currently.
If Pence had been at all smart about this, he would have combined the trip with some official reason to visit Indiana.
ETA: it's also not just a matter of moving on to the next destination, which implies that it was on his way somewhere. If he'd been going from DC to CA, for example, and made a stop in IN, it would have been less of a big deal. But in this case, he flew cross country from Vegas to Indy, then back to the west coast. Without this, he would have just had one much shorter flight from NV to CA.