We have,
$\begin{array}{r}
\tan ^{-1} \frac{1}{5}+\frac{1}{2} \sec ^{-1} x+\tan ^{-1} \frac{1}{8}=\frac{\pi}{8} \\
\Rightarrow \quad\left(\tan ^{-1} \frac{1}{5}+\tan ^{-1} \frac{1}{8}\right)+\frac{1}{2} \sec ^{-1} x=\frac{\pi}{8}
\end{array}$
$\Rightarrow \quad \tan ^{-1}\left[\frac{\frac{1}{5}+\frac{1}{8}}{1-\frac{1}{5} \times \frac{1}{8}}\right]+\frac{1}{2} \sec ^{-1} x=\frac{\pi}{8}$
$\begin{array}{ll}\Rightarrow & \tan ^{-1} \frac{13}{39}+\frac{1}{2} \sec ^{-1} x=\frac{\pi}{8} \\ \Rightarrow & 2 \tan ^{-1} \frac{13}{39}+\sec ^{-1} x=\frac{\pi}{4} \\ \Rightarrow & 2 \tan ^{-1} \frac{1}{3}+\sec ^{-1} x=\frac{\pi}{4} \\ \Rightarrow & \tan ^{-1} \frac{\frac{2}{3}}{1-\frac{1}{9}}+\sec ^{-1} x=\frac{\pi}{4} \\ \Rightarrow & \tan ^{-1} \frac{3}{4}+\sec ^{-1} x=\frac{\pi}{4}\end{array}$
$\begin{array}{rr}\Rightarrow \quad & \tan ^{-1} \frac{3}{4}+\tan ^{-1} \sqrt{x^2-1}=\frac{\pi}{4} \\ \Rightarrow \quad & \tan ^{-1}\left[\frac{\frac{3}{4}+\sqrt{x^2-1}}{1-\frac{3}{4} \sqrt{x^2-1}}\right]=\frac{\pi}{4}\end{array}$
$\begin{array}{cc}\Rightarrow & \frac{3+4 \sqrt{x^2-1}}{4-3 \sqrt{x^2-1}}=1 \\ \Rightarrow & 3+4 \sqrt{x^2-1}=4-3 \sqrt{x^2-1} \Rightarrow 7 \sqrt{x^2-1}=1 \\ \Rightarrow & \sqrt{x^2-1}=\frac{1}{7} \Rightarrow x^2-1=\frac{1}{49} \Rightarrow x^2=\frac{50}{49}\end{array}$