You only ever have to escape the dot if it is part of a regular expression and should match a literal dot. To match a literal dot with a regular expression, you may use either `\.` or `[.]`.
The replacement part of an `s` expression in `sed` is however _not_ a regular expression.
The only characters that need to be escaped there (with backslash) are `\`, `&`, newline, and the `s` command delimiter (`/` if using `s/pattern/replacement/`).
This means that your command
sed "s/hostname=localhost/hostname=$machine.$domain/g" ambari-agent.ini
would work, as long as the values of the variables involved adhere to the above-mentioned rules. I would personally additionally anchor the expression to the start of the line and drop the `g` flag from the `s` command.