The sex chromosome pattern in the human females is $\mathrm{XX}$ and that of male is XY. Therefore, all the haploid female gametes (ova) have the sex chromosome X, however, the haploid male gametes have either X or Y. Thus $50 \%$ of sperms carry the X-chromosome while the other $50 \%$ carry the Y-chromosome. After fusion of the male and female gametes, the zygote carries either $\mathrm{XX}$ or $\mathrm{XY}$ depending upon whether the sperm carrying $X$ or $Y$ fertilizes the ovum. The zygote carrying $X X$ would be a female baby and $X Y$ would be a male baby. That is why it is correct to say that the sex of the baby is determined by the father.