Here are the annual domestic migration estimates for New York County:
DOMESTIC MIGRATION
2010: -24674
2011: -84179
2012: -116346
2013: -107730
2014: -153921
For comparison, here are international migration and natural increase estimates:
INTERNATIONAL MIGRATION
2010: +24130
2011: +113261
2012: +110105
2013: +118929
2014: +118799
NATURAL INCREASE
2010: +24510
2011: +93359
2012: +93032
2013: +88431
2014: +86353
Note that the 2010 numbers only cover three months (4/1/2010 - 6/30/2010).
Where are you getting these numbers? On the census website, the cumulative 2010-2014 net domestic migration for New York County (Manhattan) is -53,493, less than Brooklyn and far less than your total.