According to mole concept 

Mass of metal $\mathrm{X}$ in alloy $= 12\times \frac{20}{100}$

Mass of metal $\mathrm{Y}$ in alloy $= 12-2.4 =9.6 \mathrm{g}$

Moles of $\mathrm{X}$ $= \frac{2.4}{40}$

Now, Number of atoms of $\mathrm{X} = \frac{2.4}{40}\times 6.023\times {10}^{23} = 3.61\times {10}^{22}$

Number of atoms of  $\mathrm{Y} = \frac{3.61\times {10}^{22}\times 5}{2} = 9.025\times {10}^{22}$

Thus, by the stoichiometry

$$\begin{gathered} 9.6\mathrm{g} \mathrm{Y} \to 9.025\times {10}^{22} \mathrm{atom} \\ 6.022\times {10}^{23}\to \frac{9.6}{9.025\times {10}^{22}}\times 6.023\times {10}^{23} \end{gathered}$$

Hence, the atomic mass of $\mathrm{Y} = 64 \mathrm{amu}$