It's deference to the executive branch that seems to be what the majority is looking at. Both Gorsuch & Kavanaugh treated citizenship inquiries as unremarkable in their questions during the oral arguments.
This. Frankly I tend to agree. I've never thought it was a good argument here or in the immigration case to make the argument that "well yes any other President could do this but Trump is uniquely so bad that otherwise legitimate presidential powers can never, ever, ever be exercised by him but only by him because him am bad!"
To add to this, I did some research into the prior use of the citizenship question on the long form used through 2000. The question went to 1 out of 6 household so the statistics are highly relevant. There was a drop off in participation across all groups for the long form which isn't surprising given the time the long form took to complete. There was also a drop off in participation by minorities and immigrants on both forms which is independent of the citizenship question. I could not find that there was a steeper drop off by immigrants than what would be expected by the combined statistics of the two separate factors of long form and lower immigrant response on all forms.
It is especially interesting to look at CA comparing the 1990 response rate to that of 2000. Prop 187 (1994) denied services to illegal immigrants and might have been expected to further undermine counts in that state. However Gov Davis invested state funds to spend on city census committees and the result was an increase in minority and immigrant participation in 2000 compared to 1990 despite the onus of Prop 187.
The CA experience in 2000 suggests that reduced response rates by immigrants and minorities can be more than overcome through state and city investment in Census Complete Count efforts. Another way of saying it is that a public education campaign has more positive impact than the negative impact of a citizenship question.