We have,
$x_{1}, x_{2}$ be the roots of equation $x^{2}-3 x+a=0$
$\therefore \quad x_{1}+x_{2}=3$ and $x_{1} x_{2}=a$
Also, $x_{3}, x_{4}$ be the roots of equation $x^{2}-12 x+b=0$
$\therefore x_{3}+x_{4}=12$ and $x_{3} x_{4}=b$
Again, $x_{1}, x_{2}, x_{3}, x_{4}$ are in GP
$\therefore \quad x_{1}=A, x_{2}=A R, x_{3}=A R^{2}, x_{4}=A R^{3}$
Now, $x_{1}+x_{2}=3$
$\Rightarrow \quad A(1+R)=3$
and $\quad x_{3}+x_{4}=12$
$\Rightarrow \quad A R^{2}(1+R)=12$
From Eqs. (i) and (ii), we have
$R^{2}=4 \Rightarrow R=\pm 2$
When, $R=2, A=1$ and when, $R=-2,A=-3$
$\therefore$ Numbers are either 1,2,4,8 or -3,6,-12,24 But $x_{1} < x_{2} < x_{3} < x_{4}$
So, $x_{1}=1, x_{2}=2, x_{3}=4, x_{4}=8$
$\begin{aligned} \therefore a b &=x_{1} x_{2} x_{3} x_{4} \\ &=1 \times 2 \times 4 \times 8=64 \end{aligned}$