Drupal is a good choice
Have a look at the address book module. Also you need to setup a few bespoke modules which should really be included in the core (Some are in version 7)
Stick with version 6 for now as 7 is still in beta
I can consult on your project if you need a hand