Yes, and only residents of the 50 states + D.C. can send electors to the college, so only they may hold elections. Another Constitutional amendment can add further electors. The Senate and the House votes are limited to residents of the States, as they are state representatives. There are US citizens on all continents (including Antarctica) who vote in federal, state, and local elections, and there are US citizens residing in the States who cannot vote in any.