Analogous to convolution A convolution of two functions $f$ and $g$ is defined as $$[f*g] = \int_{-\infty}^{\infty} f(\tau) g(t-\tau) d\tau.$$ I am interested on an analogous transformation of the form $$[f\star g] = \int_{0}^{\infty} f(\tau) g\left(\dfrac{t}{\tau}\right) d\tau.$$ Does this transformation has a name?

This is a type of "multiplicative convolution", and will sometimes appear just as $\int_0^\infty f(\tau)g({t\over \tau})\;{d\tau\over \tau}$, since the positive reals already form a group, and ${d\tau\over \tau}$ is the multiplication-invariant measure there. In fact, you might find that you'll prefer ${d\tau\over |\tau|}$ in your situation, as well, so that the measure on the whole line (really with $0$ removed) is invariant under multiplication, so that this version of convolution has the expected properties.