No, an external physical examination would be inconclusive. The reason is the TDF gene. To be more specific, if a person has XY and the gene is not active then the subject would have a female appearance. Also we cannot conclude that a person has a Y chromosome even if it has penis because that gene could be transferred on the X chromosome.
Here is a reference: <