Oliver Pernell worked around Drummondville as a farm labourer, and as he accumulated wealth, for the first time in his life, he started buying up land in the area. Pernell was a well respected citizen, and a prominent landholder in the town of Drummondville. Years after his arrival in Canada, Pernell married Matilda Sutton (1822-1926), an escaped slave from Kentucky, and they had a child together named Marjorie (1871-1965). Pernell and his family lived on Stanley Avenue in a house that still stands today just south of the Sheraton Four Points Hotel on the same street. Pernell's house is one of the only remaining houses left in Niagara Falls that was once owned by a former slave.
Oliver Pernell was a spiritual man and a devoted supporter of the local BME Church built in 1836, near the Murray Street and Portage Road intersection. In the late 1870's, Oliver Pernell donated a piece of land to the BME church on the southwest corner of Peer and Gray Streets for the relocation of the church. The church was rolled to it's new site on a bed of logs pushed by members of the congregation, and it is still there to this day. Oliver lived out the remainder of his life in Drummondville, too frightened to return to America, and he died at his Stanley Avenue home on September 17th, 1906. Oliver Pernell is buried in Drummond Hill Cemetery in Niagara Falls.