One Direction Zayn Steps Away

No one could have expected One Direction to reach this level of worldwide fame – least of all Zayn Malik. 

Citing stress, the singer, 22, has taken a break from the On the Road Again world tour that is about to invade the Philippines for the weekend. 

Though Malik reached his boiling point when the media began accusing him of cheating on his fiancée, Perrie Edwards, a source who has been with the band since their X Factor days tells PEOPLE that he has always had a hard time dealing with the "intrusion" of fame. 

"Zayn was always the quiet one," the source says. "It's not like he is shy, as he could always talk to the other boys and he could talk to people his own age. It was more like he was under-confident. Almost like he didn't believe he should be there."

For the uninitiated, Malik and his fellow bandmates auditioned for The X Factor back in 2010 as solo artists before being eliminated from the competition. However, Simon Cowell thought that they would work better as a group and brought them back into the competition as 1D during the "boot camp" stage. 

At the time, Harry Styles and Liam Payne were getting the lion's share of lead vocals while Malik, Niall Horan and Louis Tomlinson took a backseat. (Tomlinson, who is closest to Malik, didn't get any solos whatsoever during his time on The X Factor, which he later admitted was a "tough pill to swallow.") 

"Pretty quickly after Zayn got into the X Factor machine he realized that he didn't actually want to be famous and he was quite happy to hide behind the other boys," the source tells PEOPLE. "He would let Liam do the talking and wouldn't really speak in their group interviews." 

When One Direction was voted out of X Factor, winning third place, Malik "was so devastated he couldn't look anyone in the eye and was crying his eyes out. He is very sensitive." 

On the inevitable scrutiny that follows success, "he finds the fame more difficult to deal with than the others," the source continues. "Zayn finds the fact that everyone is looking at him all the time a bit much – and he has for years. Even at the start of the band he was finding it really hard to deal with the constant intrusion. They would be at the studio all day and then they would have to go out to nightclubs, so they never had any time to themselves. 

"Zayn would be like, 'I can't go because I'm too tired,' but the other boys would be out the door and looking for a good time. He couldn't be bothered and just wanted to stay in." 

Adds the source: "Basically, everything ran away from him and it all happened a little too quickly for him." 

For their part, Malik's bandmates " are really worried about him but disappointed he's not there," another source close to the band previously told PEOPLE. "Their main concern is they want him to get better. They don't want him to quit."