Let's have another look at your past perfect
The widowed Reverend Ainsworth arrives in Halifax and, with encouragement from Mrs. Priestley, sets his sights on Ann Walker. Aware that Ainsworth's attentions are stirring up memories for her lover, Lister sets out to deal with the clergyman in her own inimitable manner. Meanwhile, Lister presses on with the transformation of her estate, but her ambitious plans threaten to lead her into trouble. At afternoon tea, Marian remains impressed by Mr. Abbott, though her aunt and father are less so. A discovery in the pigsty gives Mary Sowden reason to suspect that Thomas is lying about his father.