The paper "Completeness and properness of refinement operators in inductive logic programming" uses _properness_ , though the authors may not be English first-language speakers.
Wiktionary also has this meaning as the third offered for _properness_
_-ness_ and _-less_ are productive in English (for example as in _memorylessness_ ) so I see nothing wrong with _properness_.