Kanye Reveals He Called Jay Pharoah After the VMAs to 'Tell Him My Story'

09/02/2014 AT 09:35 AM EDT

Kanye West's improvised speeches during live performances receive nearly as much attention as his music.

This weekend, his set during the Budweiser Made in America festival proved no exception.

Jay Pharoah's not-really-a-hosting-stint at the MTV Video Music Awards saw him impersonating Jay Z and Kanye West ( to the at-least-politely-feigned amusement of West's wife Kim Kardashian, seated in the audience), and while West wasn't in attendance, he did tell a crowd on Saturday that he had some thoughts on the routine that he felt compelled to share with Pharoah.

'They want to make a joke out of how hard we work,' West said during his performance Saturday, according to The Patriot-News. 'It's fine and all funny and everything, but don't distract from our vision. It's not a joke what we do up here. This music that we do is not a joke. What we do culturally is not a joke.'

Continued West: 'I called Jay Pharoah right after the MTV Awards. I said, 'I appreciate your show, but let me tell you about my story. Let me tell you about what I went through to get to that position ...' So we ain't gonna have no black comedians going up stage spoofing the people that's working hard to open doors not only for black people, but [for] any creators, anybody that wants to add a contribution to the world.'

West did manage to end the speech on a positive note: 'When I leave tonight, I want y'all to leave with love.'

Pharoah has not commented on West's remarks.