(D)
Let $y=m_{1} x$ and $y=m_{2} x$ be the lines represented by the equation. $a x^{2}+2 h x y+b y^{2}=0$
Then, $m_{1}+m_{2}=\frac{-2 h}{b}$ and $m_{1} m_{2}=\frac{a}{b}$
We have, $\frac{m_{1}}{m_{2}}=\frac{5}{3} \Rightarrow m_{1} \Rightarrow \frac{5 m_{2}}{3}$
$\therefore \frac{5 \mathrm{~m}_{2}}{3}+\mathrm{m}_{2}=\frac{-2 \mathrm{~h}}{\mathrm{~b}}$ and $\left(\frac{5 \mathrm{~m}_{2}}{3}\right) \mathrm{m}_{2}=\frac{\mathrm{a}}{\mathrm{b}}$
$\therefore \frac{8 m_{2}}{3}=\frac{-2 h}{b} \Rightarrow m_{2}=\frac{-3 h}{4 b}$ and $m_{2}^{2}=\frac{3 a}{5 b}$
$\left(\frac{-3 h}{4 b}\right)^{2}=\frac{3 a}{5 b} \Rightarrow \frac{9 h^{2}}{16 b^{2}}=\frac{3 a}{5 b}$
$\therefore \frac{\mathrm{h}^{2}}{\mathrm{ab}}=\frac{16}{15}$