The Alternative fuer Deutschland (AfD) party took 21.4% of the vote behind the centre-left SPD's 30.2%.
The German chancellor's CDU was supported by only 19.8% of those who voted, according to the exit polls.
The vote was seen as a key test before German parliamentary elections in 2017.
Before the vote in Mecklenburg-West Pomerania, all of Germany's other parties ruled out forming a governing coalition with the AfD.
However, the party's strong showing could weaken Mrs Merkel ahead of the national elections next year. ...