Oliver finished with 19 points and 10 rebounds for the Flyers (16-12, 6-8 Atlantic 10 Conference), who have won two in a row.
Dayton led the majority of the first half and never trailed after halftime, but had a tough time holding off the Spiders. The Flyers had a seven-point advantage midway through the second half, then had their lead trimmed to 60-59 when Richmond's Cedrick Lindsay made a free throw with 6:11 left.
But the Spiders (17-12, 7-7) were unable to tie or pull ahead. Dayton's Josh Benson and Dyshawn Pierre each followed with a free throw, then Dillard converted a three-point play to push the Flyers lead to six and hold on.
Lindsay had 22 points and seven assists for Richmond.
Colorado State graduate transfer John Gillon has chosen Syracuse, giving the Orange a potential starter at point guard entering the 2016-17 season.
DeSean Murray, who averaged 20.2 points as a sophomore at Presbyterian, will transfer to Auburn to play for coach Bruce Pearl.
Former Kentucky guard Ed Davender, the only Wildcat with at least 1,500 points and 400 assists, has died. He was 49.