The damage to your car was due to you hitting the tree and thus would normally fall under the collision portion of your physical damage coverages, like your insurance agent stated.

You are correct if you had hit the animal then it would be the comprehensive portion of your policy, and that associated deductible, that would apply here however since you did hit the animal but instead swerved to miss it and collided with an object, the tree, then your collision coverage would be used to claim the damages.

Collision covers damage to your car when your car hits, or is hit by, another vehicle, or other object so even though the chain of events including an animal nearly being hit, the animal did not do the damage to the car. The tree did the damage and thus collision coverages are used to cover the cost of repairs to your vehicle.