The given equation is
$x^2+6 x y+8 y^2=10$ $\ldots$ (i)
Since axes are rotated through an angle $\frac{\pi}{4}$.
$\therefore \quad x=x_1 \cos \frac{\pi}{4}-y_1 \sin \frac{\pi}{4}=\frac{x_1-y_1}{\sqrt{2}}$
and $\quad y=x_1 \sin \frac{\pi}{4}+y_1 \sin \frac{\pi}{4}=\frac{x_1+y_1}{\sqrt{2}}$
On putting the value of $x$ and $y$ in Eq. (i)
$\left(\frac{x_1-y_1}{\sqrt{2}}\right)^2+6\left(\frac{x_1-y_1}{\sqrt{2}}\right)\left(\frac{x_1+y_1}{\sqrt{2}}\right)$ $+8\left(\frac{x_1+y_1}{\sqrt{2}}\right)^2=10$
$\Rightarrow \quad \frac{x_1^2+y_1^2-2 x_1 y_1}{2}+\frac{6 x_1^2-6 y_1^2}{2}$ $+\frac{8\left(x_1^2+y_1^2+2 x_1 y_1\right)}{2}=10$
$\Rightarrow \quad x_1^2+y_1^2-2 x_1 y_1+6 x_1^2-6 y_1^2+8 x_1^2$ $+8 y_1^2+16 x_1 y_1=20$
$\Rightarrow \quad 15 x_1^2+3 y_1^2+14 x_1 y_1=20$
$\therefore$ Required equation is
$15 x^2+14 x y+3 y^2=20$