White, with his first goal with the Rangers, and Boyle scored 2:11 apart to bring New York to 5-4 with 9:53 remaining. Defenseman Matt Gilroy nearly tied it with 6:45 left when his shot from the left point hit Mason's glove and then the crossbar.
"We lost and we deserved to lose," Rangers coach John Tortorella said. "We shouldn't put ourselves in that type of spot. We didn't stay within ourselves and play the way we needed to. We shouldn't have let it get out of hand."
Ladd's empty-netter sealed it with 23.2 seconds to go.
"It was kind of stupid to let them back in the game," Bergfors said. "We stopped working a little bit there and they took care of it, but we got it done."
The Rangers came out flying and seemed determined to take the fight out of the Thrashers. New York grabbed a 1-0 lead just 38 seconds in when Callahan scored his second goal.
Despite pockets of empty seats throughout Madison Square Garden, the crowd was into the game as the Rangers pressured Mason. New York held a 7-1 shots advantage through the first 4:46 but then had its offense run dry.
Atlanta got even with 8:32 left in the first when Little scored off a pretty rush through the Rangers. Little surged into the middle of the zone, cut to his right and got around sprawling defenseman Marc Staal and then moved to the front of the net. Little beat Biron with a slick forehand-to-backhand move to tie it at 1.
"Wow, we are fun to watch," Ramsay said.
The Thrashers outshot the Rangers 11-3 over the final 15-plus minutes of the period. Atlanta earned the only power plays of the first, getting them back-to-back late in the frame for a five-on-three advantage that lasted 51 seconds.
New York killed off the first half before the period ended and finished off the power play in the opening minute of the second. The Rangers couldn't build off that momentum.
Kane gave the Thrashers their first lead when he snapped a shot from the outside edge of the right circle at 10:17 of the second. Kane, the No. 4 pick in the 2009 draft, is off to a sizzling start in his second NHL season, scoring six goals in nine games.
Tortorella burned his timeout as Kane retrieved the puck for 19-year-old rookie teammate Alex Burmistrov, who assisted for his first NHL point.
The breather seemed to help the Rangers as they tied the game just 2:31 later when Avery turned hard work behind the Atlanta net into a stuff-in goal at the right post.
But New York couldn't sustain it, even after receiving its only power play less than a minute after Avery's goal. The Thrashers soon went on top again.
After a turnover in the Rangers' end, Bergfors raced the puck up right wing on a two-on-one rush. With New York defenseman Michal Rozsival cutting off the passing lane, Bergfors fired a shot over Biron's shoulder to make it 3-2 with 3:53 left in the second period.
Atlanta's lead grew to two 49 seconds into the third when Boulton scored his first of the season with a backhander from the slot. Byfuglien made it 5-2 at 5:01.
"There were a couple of times where I could have made that big save that would have probably kept it tight," Biron said.
Rangers center Artem Anisimov injured his right ankle while blocking a shot in the second period. He will have an MRI on Thursday. Rozsival hyperextended a knee, but Tortorella doesn't believe that injury is serious. ... Ladd, who had a goal and an assist, tied his career high with a six-game point streak (three goals, seven assists). ... The Rangers haven't won four straight since Dec. 17-23, 2009.
Copyright by STATS LLC and The Associated Press
Arena: Madison Square Garden Location: New York, New York
Referees: Stephane Auger, Tim Peel Linesmen: David Brisebois, Brian Murphy
Attendance: 17,900 (99.4% full)
Team Stat Comparison
Power Play Goals
Power Play Opportunities
Shots on Goal
Ryan Callahan (2)
Assist: Artem Anisimov
Bryan Little (1)
Assists: Andrew Ladd, Ron Hainsey
Evander Kane (6)
Assists: Alex Burmistrov, Anthony Stewart
Sean Avery (1)
Assists: Dan Girardi, Ruslan Fedotenko
Niclas Bergfors (1)
Eric Boulton (1)
Assists: Anthony Stewart, Rich Peverley