Bryant scored 26 points and Pau Gasol had 25 for the Lakers, whose road skid is their longest in six years. Dwight Howard was ejected after drawing his second technical foul with 1:18 left in the first half.
The Lakers fell behind by 15 points midway through the first quarter and lost for the eighth time in 10 games.
"We've got a lot of issues," coach Mike D'Antoni said. "Offensively the ball sticks and we don't take very good shots. Defensively, ups and downs, we forget guys."
Jose Calderon had 22 points and nine assists while Ed Davis had 18 points and eight rebounds for Toronto. Alan Anderson added 14 points and eight assists as the Raptors ended a four-game losing streak against the Lakers.
"That was one of the most complete games we've had all year, top to bottom," Raptors coach Dwane Casey said.
Bryant made 10 of 32 shots, including 3 of 12 on 3-pointers, and complained the "heavy legs" had dulled his shooting effort.
"I've just got to rest my legs," he said. "My legs are a little tired. My shots are just short. That's on me. I'll take this loss on me, gladly. There were a lot of easy shots, a lot of them, that I should have put down."
"My offense was sub-par in terms of missing easy shots," he said. "I've got to do a better job of putting that ball in the hole when the opportunity presents itself."
"I didn't feel like we were ready to play," Nash said. "We cut too many corners at too many different times in the game and had too big a hole to climb out of."
"We're just not finding any consistent level out of our group," he said.
Howard got his first technical for arguing after a missed jump shot with 4:43 left in the first quarter. The star center was tossed after he and Anderson were called on a double technical when they got tangled up after a free throw. Howard finished with five points in 17 minutes.
Howard left without speaking to reporters, but Bryant said he was upset by the costly ejection.
"I really don't agree with that one at all," Bryant said. "What's a player supposed to do? A guy cuts in front when he's trying to walk back up the court. An official told me he should just walk away. I said, `Which direction? Should he just turn around a just walk to the bench?"
Clark said Los Angeles struggled to defend the paint without Howard's presence inside.
"When he's gone, it's just a layup special," Clark said.
Los Angeles responded with a 3 by Nash and, after a layup by Fields, a pair of 3-pointers by Bryant and a layup by Antawn Jamison made it 97-87 with 5:22 remaining.
Bryant hit a 3 with 1:35 left, cutting it to 103-97, but Amir Johnson rebounded Anderson's miss and scored a layup to make it an eight-point game with 1:02 left.
Gasol returned to the starting lineup after coming off the bench for the first time in his Lakers career in Thursday's loss to Miami. Gasol had sat out the previous five games with a concussion after being accidentally elbowed in the face by Denver's JaVale McGee.
It was a slow start for the Lakers, who missed 12 of their first 15 shots and were down 19-4 with 5:50 left in the first. Bryant missed his first six shots and made just one of eight in the opening quarter as Toronto led 29-20.
"We started off awful, without any energy," D'Antoni said.
Gasol scored 10 points and Bryant had nine in the third, but the Raptors got 10 points from Fields and six each from Calderon and Davis to take an 83-70 edge into the fourth.
Raptors C Andrea Bargnani (right elbow) has resumed working with a basketball, while rookie C Jonas Valanciunas (broken right ring finger) is lifting weights, Casey said. ... Nash made his first visit to Toronto since passing on a contract offer from the Raptors to sign with Los Angeles last summer. ... This was the first game of a three-game road trip for the Lakers, who will also visit Chicago and Memphis. ... Montreal Canadiens defenseman P.K. Subban attended the game.
After whipping the Warriors in Game 4, are the Thunder on the verge of completing an unprecedented run to the NBA Finals? Kevin Pelton has the spectacular facts.
How Oklahoma City took a commanding 3-1 series lead in the Western Conference finals with a dominant Game 4 win over Golden State.
After a Game 4 loss in which Draymond Green admitted he was "awful," Golden State's season is suddenly on the brink of elimination.