For no solution $D=\left|\begin{array}{ccc}1& 1& 1\\ 1& a& 1\\ a& b& 1\end{array}\right|=0$

Expanding the determinant along the first row, we get $D=-{\left(a-1\right)}^2=0$

$\Rightarrow a=1$

For $a=1$, first two planes are coincident.

Now, we have $x+y+z=1 \& ax+by+z=0$

For no solution, these two planes are parallel.

$\Rightarrow \frac{1}{a}=\frac{1}{b}=\frac{1}{1}$

$\Rightarrow a=1 \& b=1$.

$\therefore b=\left\{1\right\} \Rightarrow$it is a singleton set