" _Blame_ " is a strong word. The human Y chromosome has only a few dozen protein-coding genes. One of the most important, for males anyway, is SRY, the sex-determining region. This is a transcription factor that is responsible for modulating the expression of other genes; those genes need not be on the Y chromosome. SOX9 for example is itself a transcription factor, is upregulated by SRY, and is found on chromosome 11. _Most_ of the genes associated with maleness are found elsewhere. Moreover, once the testes form, they can be potent secretors of hormones that cause a number of physical changes. So is SRY to blame, or SOX9, or the testes? It depends on how direct you want your genes to be. If you're blaming SRY you're not wrong, but at that point you might as well blame fate.
Oh, and there are epigenetic differences between the sexes. You could also make some arguments about the Xist gene responsible for X-inactivation in females.