From the nth state, the atom may go to (n - 1)th state, ...., 2nd state of 1st state, So there are (n - 1) possible transitions starting from the nth state. The atoms reaching (n - 1)th state may make (n - 2) different transitions. Similarly for other lower sates. The total number of possible transitions is

              (n - 1) + (n - 2) + (n - 3) + ... 2 + 1

              $=\frac{\text{n}\left(\text{n}-1\right)}{2}$

Alternative method:

Take any test value of $n$ for e.g.  case of $n=3$ and draw spectral lines that will be allowed for it. There will be $3$ spectral lines allowed for it. Only first option satisfies this when you put value of $n=3$.