That behavior is required by POSIX, and you're safe to rely on it.
Another note that you want to set your locale to `C` to get consistent behavior. In locale with collation elements have the same sorting order, you will have strange result.
On GNU system with UTF-8 locale:
$ printf '%b\
' '\U2461' '\U2460' | sort
②
①
or:
$ printf '%s\
' A B a b | sort
a
A
b
B
Setting to `C` locale:
$ printf '%b\
' '\U2461' '\U2460' | LC_ALL=C sort
①
②
$ printf '%s\
' A B a b | LC_ALL=C sort
A
B
a
b
Some shells even do not support multibyte characters like `dash`, `mksh` or support but will choke on invalid sequences of bytes like `yash`.