I have never contemplated these issues before but _The Relationship Calculator_ by Stephen Morse seems to handle them without difficulty.
> stepfather of your father is your father's mother's spouse = step grandfather
>
> father of your stepfather is your mother's spouse's father = step grandfather
>
> step-father of your stepfather is your mother's spouse's mother's spouse = step grandparent
Stephen seems to share the view of ColeValleyGirl in her answer that there is no need for any further complexity once you reach a non-blood relationship.
If you have an incredibly complex example that is very well-documented, you might be able to invent a name that will be famous enough for everyone to adopt.