According to The Mirror, the Stereo Kicks singer posted messages about cannabis.
The tweets were allegedly posted last year before he auditioned for the show.
One reportedly read: “Always smoke upon one before sleep,” whilst another said: “Come smoke herb with the boyz again soon pretty princess xxx (sic).”
The 19-year-old is said to be ‘not proud’ about his past behaviour.
A source told the paper: "Jake's not proud of how he's behaved in the past.
"He has done a lot of growing up over the past year and in no way wants to let his fans, or his band mates, down.”
"He holds his hands up and accepts how stupid his behaviour was, but wants to put this behind him and focus on making the most of the opportunity he has been given.
"The tweets about crack are actually references to song lyrics and he has never smoked or sold crack."
An X Factor spokesman said: "The X Factor does not condone the use of drugs in any way and appropriate action will be taken against any contestant found to be taking drugs while involved with the programme."
The X Factor continues tonight at 8pm on ITV. Watch it with us here on heatworld with heat’s Big Night In!