Data Sufficiency (DS) | Re: In the xy-plane, point O is located at the origin, point A has coordin

Ans should be B.
The most important clue that we have is AO>AB
This restricts how far r can go.
AO from coordinates will be 5 units in length. So maximum value of AB < 5.
And all points are in first quadrant only so maximum value of r can be 8 (excluding 8).
So maximum area of tringle could be
1/2 * 8 * 3 = 12 (excluding 12)
Let me know if its not clear or if I am wrong somewhere.