Given system of equations

$D=\left|\begin{array}{ccc}2& 1& 1\\ 1& -1& 1\\ 1& 1& a\end{array}\right|=0\Rightarrow -2a-2-a+1+2=0\Rightarrow a=\frac{1}{3}$

$D_1=\left|\begin{array}{ccc}5& 1& 1\\ 3& -1& 1\\ b& 1& a\end{array}\right|=5\left(-a-1\right) + -1\left(3a-b\right) +1\left(3+b\right) = -8a+2b-2$

$D_2=\left|\begin{array}{lll}2& 5& 1\\ 1& 3& 1\\ 1& b& a\end{array}\right|=2\left(3a-b\right) -5 \left(a-1\right)+1\left(b-3\right) = a-b+3$

$D_3=\left|\begin{array}{ccc}2& 1& 5\\ 1& -1& 3\\ 1& 1& b\end{array}\right|=2\left(-b-3\right) - 1 \left(b-3\right) + 5\left(2\right) = 7-3b$

For the system to have no solution $\Rightarrow D = 0 , D_3 \ne 0$

$\therefore a = \frac{1}{3} , b \ne \frac{7}{3}$.