Let $y=m_1 x$ and $y=m_2 x$ be the lines represented by the equation.
$$
a x^2+2 h x y+b y^2=0
$$
Then, $\mathrm{m}_1+\mathrm{m}_2=\frac{-2 \mathrm{~h}}{\mathrm{~b}}$ and $\mathrm{m}_1 \mathrm{~m}_2=\frac{\mathrm{a}}{\mathrm{b}}$
We have, $\frac{\mathrm{m}_1}{\mathrm{~m}_2}=\frac{5}{3} \Rightarrow \mathrm{m}_1=\frac{5 \mathrm{~m}_2}{3}$
$$
\begin{aligned}
& \therefore \frac{5 \mathrm{~m}_2}{3}+\mathrm{m}_2=\frac{-2 \mathrm{~h}}{\mathrm{~b}} \text { and }\left(\frac{5 \mathrm{~m}_2}{3}\right) \mathrm{m}_2=\frac{\mathrm{a}}{\mathrm{b}} \\
& \therefore \frac{8 \mathrm{~m}_2}{3}=\frac{-2 \mathrm{~h}}{\mathrm{~b}} \Rightarrow \mathrm{m}_2=\frac{-3 \mathrm{~h}}{4 \mathrm{~b}} \text { and } \mathrm{m}_2^2=\frac{3 \mathrm{a}}{5 \mathrm{~b}} \\
& \left(\frac{-3 \mathrm{~h}}{4 \mathrm{~b}}\right)^2=\frac{3 \mathrm{a}}{5 \mathrm{~b}} \Rightarrow \frac{9 \mathrm{~h}^2}{16 \mathrm{~b}^2}=\frac{3 \mathrm{a}}{5 \mathrm{~b}} \\
& \therefore \frac{\mathrm{h}^2}{\mathrm{ab}}=\frac{16}{15}
\end{aligned}
$$