It modifies the noun .
Here's a simpler example:
> - (I) read books
>
> - A child who reads books
>
> **** - A **male** child who reads books
In your example, being fairly literal, we have
> [[]][] - They were hooligans who invaded other stars
>
> [[]] [ **** ] - They were hooligans **of the universe** who invaded other stars
Edit per @Chocolate's suggestion: [] also directly modifies . The is the of and it is used to connect the clauses. It doesn't say that they **like** to invade other stars. It only says they like to fight/engage on battle, the invading is a separate action. You can think of it like this:
> [ **** ] [] - They were hooligans **who liked fighting**
>
> [][ **[]** ] [] - They were hooligans who liked fighting and **invaded other stars**. Note that the action of liking is separate from the action of invading.
>
> [][[]] [ **** ] - They were hooligans **of the universe** who liked fighting and invaded other stars