For a decreasing function, a point and its swap can both lie on the graph. Suppose f is decreasing and passes through (0,2) and (2,0); then f−1 passes through the swapped points, which are the same two points.
(0,2)→(2,0),(2,0)→(0,2)
So the graphs share (0,2) and (2,0), neither of which is on y=x. Increasing functions meet their inverse only on y=x.