Managing Team Field Security Notes
If a user is the RecordManager for a contact, that user will always have access to that contact.