Kanye West Scuffle Prompts Police Investigation

01/13/2014 at 08:20 PM EST

Kanye West is the subject of a Beverly Hills Police investigation after a man claimed on Monday that the rapper assaulted him.

'The named suspect was identified as Kanye West by the victim and several witnesses,' says a police statement. 'Mr. West had left the location prior to officers' arrival. As of this writing, BHPD detectives are conducting a follow-up investigation.'

A source close to Kim Kardashian says it was the bystander - who has not been identified - who instigated the event and threatened the reality star.

'Kim was walking into a furniture meeting for their new home,' the source tells PEOPLE. 'Kanye was there ahead of her for the meeting. So Kim gets there and walks into the building with a million paparazzi trailing her.'

The source says a Caucasian man began yelling racial obscenities to the paparazzi, then when Kardashian told him 'that wasn't cool,' then man began hurling epithets at her about her and West.

'At this point Kim was seriously afraid for her life,' the source continues. 'He was going nuts. Kanye came down to help her but then the guy ran into a chiropractic office so Kim and Kanye walked in after him.'

The man verbally confronted West with obscenities, and at one point Kanye hit him - 'barely,' the source insists.

A lawyer for West wasn't immediately available for comment.

* Additional reporting by JENNIFER GARCIA