Argentina, of course.
Since I'm the only person of Argentine heritage I can think of in the American LGBT blogosphere, it'd be a shame if I didn't mention this story. A judge in Beunos Aires last Friday ruled in favor of same-sex marriage in the case of HIV/AIDS activists Alex Freyre and José María Di Bello. The mayor has said he wouldn't challenge the ruling, and the couple set the date for December first, World AIDS Day.
It isn't surprising that Argentina's the first - it's the most secular country in Latin America. The AP says that 90% of the country is Catholic, which I'm sure is true if you asked people, but the CIA Wold Factbook clarifies that only 20% of people consider themselves "practicing," which is less religiosity than the US.
What I know less about is how Argentina handles federalist questions. The US is the only other country that allows same-sex marriage in some regions but not in others (Canada used to be like us until it legalized same-sex marriage throughout), so who knows how long this will stand.