The total probability of a sample space is 1. So if the probability of choosing a square is subtracted from 1 then we will find the probability of choosing a rectangle that is not a square. The cardinality of choosing a rectangle from N x N chessboard is n+1 choose 2 multiplied by n+1 choose 2. And the cardinality of choosing a square from the N x N chessboard is n(n+1)(2n+1) / 6. So the probability choosing a rectangle that is not a square will be