Merit functions can be applied if you have a constrained optimization problem which simultaneously decreases the cost function and violates one of the constraints.
It allows to keep the iteration going while preserving a measure for the speed of convergence and does not require the cost function or the admissible set to be convex.
A simple example for a merit function associated with an equality-constraints problem is the penalty function
$$ p = f + \sigma \sum\limits_{j=1}^p |g_j|, $$
where $f$ is the cost function, $\sigma$ is some real positive value and $g_j = 0$ denote the equality constraints.
The penalty function then serves as a basis for evaluating the optimal step length $\alpha$, given a search direction $\mathbf{s}$:
$$ \alpha = \arg\min_{\alpha} p(\mathbf{x} + \alpha\mathbf{s}) $$
A merit function is a modified cost function to circumvent poor convergence behaviour in iteration regimes where constraints should be weakend.