photo credit:  Featureflash Photo Agency // Shutterstock

James Franco is the most recently actor to be added to the list of the sexual assault accused. Five women total have stepped forward to speak out on Franco, four being students, one other being a woman he mentored.

Franco appeared on several late night talk shows this week and gave his side after the accusation surfaced, but he still decided to skip the Critics Choice Awards on Thursday, at which he won Best Actor in a Comedy Film. Franco flew to L.A. for the awards, but changed his mind.

“I have my own side of this story, but I believe in these people that have been underrepresented getting their stories out enough that I will hold back things that I could say, just because I believe in it that much,” he said on Late Night with Seth Meyers. “So if I have to take a knock because I’m not going to try and actively refute things, then I will, because I believe in it that much.”

Sources tell PEOPLE that “he’s in a really bad place, so bad that he changed his phone number. His close friends are trying to be there for him but it’s been hard – he’s only talking to a select group of people. For now, he’s just hiding out.”

“Look, in my life I pride myself on taking responsibility for things that I have done,” Franco told Stepen Colbert on The Late Show With Stephen Colbert. “I have to do that to maintain my well being. The things that I heard that were on Twitter are not accurate. But I completely support people coming out and being able to have a voice because they didn’t have a voice for so long. So I don’t want to shut them down in any way.”

Written by Clarke Jones