Given that,

$\tan \frac{A-B}{2}=\frac{1}{4}\tan \frac{A+B}{2}$

$\Rightarrow \frac{\sin \left({\displaystyle \frac{A-B}{2}}\right)}{\cos \left({\displaystyle \frac{A-B}{2}}\right)}=\frac{1}{4}\frac{\sin \left({\displaystyle \frac{A+B}{2}}\right)}{\cos \left({\displaystyle \frac{A+B}{2}}\right)}$

$\Rightarrow 4\sin \left(\frac{A-B}{2}\right)\cos \left(\frac{A+B}{2}\right)=\sin \left(\frac{A+B}{2}\right)\cos \left(\frac{A-B}{2}\right)$

$\Rightarrow 2\left(\sin A-\sin B\right)=\frac{1}{2}\left(\sin A-\sin \left(-B\right)\right)$

$\Rightarrow 4\sin A-4\sin B=\sin A+\sin B$

$\Rightarrow 3\sin A=5\sin B$

$\Rightarrow \frac{5}{\sin A}=\frac{3}{\sin B}$ [From sine rule]

$\Rightarrow \frac{a}{\sin A}=\frac{b}{\sin B}$

$\therefore b=3$

Now, $\sqrt{a^2-b^2}=\sqrt{5^2-3^2}$

$=\sqrt{25-9}$

$=\sqrt{16}$

$=4$