Frazier also pulled down 10 rebounds in a successful return from an Achilles tendon injury that sidelined him for all but four games last season.
Wagner trimmed a 12-point halftime deficit to 33-30 when Penn State missed its first eight field-goal attempts of the second half.
The Nittany Lions responded with a 21-10 run capped by Frazier's 3-pointer to lead 54-41 with 8:21 to go.
The Lions didn't allow Wagner to within nine points after that.
Both teams shot poorly from the field -- Wagner at 29.9 percent and Penn State at 35.8. But Penn State converted 6 of 16 3-pointers as opposed to just 2 of 23 for Wagner.
Mario Moody's 11 rebounds enabled Wagner to outrebound Penn State 44-43, but the Seahawks missed a number of layups and other shots from the paint in their 20-for-67 shooting performance.
Nine players scored for Penn State as Miami, Ohio transfer Allen Roberts chipped in with seven rebounds and four points and forward Ross Travis added eight points.
It was a relatively clean game as Wagner committed just five turnovers to Penn State's six.
Wagner's Moody, who had eight points, was called for a technical foul in the first half for pointing at Penn State's student section after scoring a basket.
Ortiz, the Northeast Conference's defensive player of the year the past two seasons, scored 13 second-half points and Rivers had 10 but they were the Seahawks' only consistent scorers.
Mississippi State freshman guard Malik Newman will withdraw from the NBA draft, but sources told ESPN he might explore other opportunities besides returning to Mississippi State.
Seton Hall sophomore Isaiah Whitehead will stay in the NBA draft and not return for his junior season.
Washington State's trio of Josh Hawkinson, Conor Clifford and Ike Iroegbu have withdrawn from the NBA draft and will play their senior season with the Cougars.