Yes, a vast majority of the people aboard first European ships to settle Australia were convicts.
The first European settlers arrived in Australia in the First Fleet.
Wikipedia:
> The total number of free people was 348 and the total number of prisoners was 696, coming to a total of 1044 persons.
This is supported by a quote from the Empire Newspaper of 1850:
> The founders of the colony therefore consisted of one free person to every two prisoners.
(Your childhood vision was inaccurate though. The same sources indicate over 200 marines were also sent. The convicts were still under the control of the British military.)
The Second Fleet was predominantly convicts. Over 1000 convicts left England, but there were many hundreds of deaths on the voyage or shortly afterwards. Ref, Ref
It should be noted that not all British Settlements were penal settlements. Example