Multiply top and bottom by the conjugate 5−3. The denominator becomes 5−3=2, and the numerator becomes a square.
(5−3)2=5−215+3=8−215
Every term now shares a factor of 2, so cancel it against the denominator:
28−215=4−15≈0.127
That matches the direct division 0.504÷3.968≈0.127. Two of the wrong options are the two ways the cancellation goes astray. Using 5+3=8 for the conjugate product, instead of 5−3, leaves 88−215, and cancelling the factor of 2 a second time leaves 24−15.