Former Georgia Tech starter Vad Lee completed 16 of 37 passes for 141 yards in his debut for the Dukes.
New James Madison coach Everett Withers saw his offense penetrate Maryland territory only once before halftime, on a late second-quarter drive halted by Johnson's interception in the end zone.
James Madison falls to 0-3 against Maryland. The Terps won the previous two meetings by a combined 11 points.
Madison's Jimmy Moreland intercepted a Rowe pass in the fourth quarter, setting up the Dukes' lone score as John Miller plunged from one yard out.
Brown scored on first-half carries of 11, 8 and 2 yards, scoring three rushing touchdowns for the second time in his career. He finished with 61 yards on seven carries and completed 11 of 24 passes for 111 yards, including a 41-yard touchdown pass to Deon Long that made it 38-0.
Ross ran for 86 yards on 16 carries. The Terps outgained the Dukes 285-161 on the ground, and averaged 5.7 yards per carry.
Maryland led 14-0 after 6 minutes in the opening quarter on drives of 58 and 46 yards.
On the first, Brown bruised his way in from 11 yards, one play after finding Stefon Diggs over the middle for 5 yards on third-and-4.
On the second, he ran 8 yards into the end zone untouched on a designed QB draw to finish off a five-play march.
Brad Craddock's 49-yard field goal pushed the lead to 17-0 later in the first.
In the second quarter, Brown finished off a 15-play, 80-yard drive with another punishing run on a fourth-and-goal from the 2.