If the viewpoint of the speaker is between the mansion and the church, can mean "the wall **behind** the church", and can mean "the wall **in front of** the church" or "the wall between the mansion and me".
←West [] [] [speaker] [] [] East→
This happens because this means "the further place (in relation to the church)". The wall doesn't have to belong to the church itself. If the speaker thinks the church is somehow hiding the wall from the speaker's viewpoint, that's enough.
Please see this related question, too: What is the difference between vs ?