NEW YORK -- When the ball went up, Brian McCann thought it was the second out of the inning. And when it came down, the Yankees had a three-game sweep.
"I'll take it, for sure," McCann said with a laugh. "I saw that everybody was looking at each other. So there was a chance, and luckily for us it fell."
Ellsbury and Derek Jeter each had an RBI single in the fifth inning for the Yankees. Hiroki Kuroda outpitched Cincinnati ace Johnny Cueto and left with a 2-1 lead before Todd Frazier's tying homer off Dellin Betances in the eighth.
Ellsbury also stole two bases and made a sliding catch in center field that saved a run.
"He can do anything on the baseball field. He showed all his tools today. He does this on a nightly basis," McCann said.
Ellsbury reached safely all five times up, including a leadoff single in the ninth against Aroldis Chapman (0-3). Before that, left-handed hitters had been 1 for 18 with 12 strikeouts against the hard-throwing Cuban this season.
Ellsbury got a great jump and swiped second without a throw. He reached third on a wild pitch before Chapman struck out Mark Teixeira with a 101 mph fastball.
McCann, another left-handed batter, lifted a popup that barely reached the outfield grass behind first base. But with the infield playing in, Frazier and second baseman Skip Schumaker scurried back trying to get under the ball as Jay Bruce rushed in from right field.
"It is anybody's ball," Frazier said. "If you can get to it, it is your ball."
All three appeared to have a hard time finding it in the air, however, and the ball landed untouched. Ellsbury alertly saw what was happening and dashed home easily as McCann stood on first base with his arms raised.
"You don't see that too often," Ellsbury said. "It's just a read that you have to make pretty quick."
Chapman stared out toward shallow right field, where Frazier, Schumaker and Bruce were practically frozen in disbelief.
"Just one of those things -- I thought I was under it and I was not," Frazier said. "By the time the ball was coming down it was already too late so, I made a mistake, turned my body the wrong way and, one of those things where I should have caught it."
Center fielder Billy Hamilton had trouble with the sun earlier, allowing Ellsbury's seventh-inning fly to fall for a double.
David Robertson (1-2) pitched a perfect ninth. It was New York's second walk-off win this season and the fifth game-ending hit for McCann -- his first since a home run for Atlanta against Houston on May 17, 2011.
The 39-year-old Kuroda gave up three hits and no earned runs over 6 2/3 innings in his 200th major league appearance.
"He was great," Yankees manager Joe Girardi said. "Did a wonderful job today."
Cueto threw 112 pitches in five innings, matching his shortest start this season.
"I felt really strong today," he said through a translator. "I would have kept going even with the amount of pitches I had."
Frazier won a matchup of All-Stars when he lofted Betances' full-count pitch into the left-field corner, just beyond the 318-foot sign.
It was the first home run at Yankee Stadium for Frazier, who grew up about 85 miles away in Toms River, New Jersey. He pumped his fists at home plate and received hearty high-fives in the dugout.
Betances has allowed two homers in 58 1/3 innings this season.
"I think we were all a little bit shocked when it happened," Girardi said.
Moments earlier, Schumaker made a costly baserunning mistake.
He singled to start the inning and, with Frazier at the plate, took off trying to steal second before Betances began his delivery. The big right-hander stepped off the rubber, and Schumaker was tagged out after a long rundown.
Schumaker, just off the seven-day concussion disabled list, had an RBI double with two outs in the fifth. It was Cincinnati's only hit in 22 at-bats with runners in scoring position during the series.
"They just outplayed us," Reds manager Bryan Price said. "Some of the things that happened in the series (were) unfortunate."
Frazier, an All-Star third baseman this season, made his eighth start of the year at first and robbed Teixeira of a first-inning hit with a diving stop at the bag. ... Cincinnati is 3-8 in interleague games. ... Yankees RF Ichiro Suzuki snapped an 0-for-18 slump with a sixth-inning single. ... New York rookie Shane Greene (2-0, 1.32 ERA) starts Monday night against Texas RHP Miles Mikolas (0-2, 10.05). ... Mariano Rivera's son, drafted by the Yankees in the 29th round last month, did not sign with the team before Friday's deadline. Mariano Rivera III was a sophomore starting pitcher at Iona last season.