‘House of Villains’ star Jax Taylor.Photo:Matthias Clamer/E! Entertainment

HOUSE OF VILLAINS – Season: 1 – Pictured: Jax Taylor

Matthias Clamer/E! Entertainment

Vanderpumpmay rule, but not when it comes tobeing a villain, apparently.

“You think the No. 1 guy’s going to go that easily? I don’t think so,” Taylor tells PEOPLE, referencing one of his most memorableVPRquotes. “I’m not going anywhere. I’ll be back. I’m definitely making a comeback. Don’t worry.”

Taylor, 44, thinks his costars — who includeThe Challenge’sJohnny “Bananas” Devenanzio,Love Is Blind’sShake Chatterjee,Survivor’sJonny Fairplay,The Apprentice’sOmarosa Manigault NewmanandFlavor of Love’sTiffany “New York” Pollard— targeted him because of his athleticism.

“It was 100 percent, that was the reason,” he says. “I am a physical threat. I dominate them with everything I do. I’m athletic. I’m an alpha male. I dominate them with everything I do. It’s between me and Johnny Bananas as far as the dominant male in the house. And that’s totally fine.”

Manigault Newman, 49, put Taylor up for elimination alongside Chatterjee, 36, andThe BacheloralumCorinne Olympios.

“Omarosa wasn’t just trying to get me out,” Taylor claims. “Omarosa is a very big character; her and New York, they’re characters. They’re playing off script. They want to look like these ominous figures. It’s all for show. She’s a great person. they’re all great people, except for one.”

Taylor, of course, means hisHouse of Villainsnemesis: Chatterjee.

“I don’t like the kind of person he is,” the Bravo vet says. “He’s a very immature person. And I have to understand, I was that immature, too. He’s got a lot of growing up to do. He’s very entitled and I think he was very, very lucky to be on that show considering the people that he was on the show with. Everybody that was on theHouse of Villainsare true icons. I don’t consider him an icon.”

‘House of Villains’ contestants Shake Chatterjee (left) and Jax Taylor.Matthias Clamer/E! Entertainment (2)

HOUSE OF VILLAINS - Shake Chatterjee and Jax Taylor

Matthias Clamer/E! Entertainment (2)

OnHouse of Villains, Chatterjee stated that he didn’t think Taylor needed the $200,000 prize money because of his other businesses and partnerships. Taylor says the presumption came from a conversation he had with Chatterjee in which the Netflix star asked about his career path.

Taylor says he got a sense that Chatterjee “was struggling” and “doesn’t really like his job” as a veterinarian and a loan officer.

“He doesn’t make a lot of money, living in an apartment and he was venting to me a little bit,” the dad of2-year-old Cruzsays. “I sensed a little vulnerability there, so I thought, I’m going to be a little vulnerable too. I thought, ‘I’ll give you some advice, so you don’t make so many bad decisions.’ And he took it as, ‘He’s cocky, conceited, he doesn’t need this.’ And when I heard that, I was like, ‘You son of a bitch. You used that against me.’ I went into theHouse of Villainsbeing the nicest guy there and I screwed up.”

‘House of Villains’ contestants (from left) Jax Taylor, Shake Chatterjee, Johnny “Bananas” Devenanzio, Omarosa Manigault Newman, Anfisa Arkhipchenko.Casey Durkin/E! Entertainment

HOUSE OF VILLAINS Joel McHale, Jax Taylor, Shake Chatterjee, Johnny Bananas, OMAROSA, Anfisa Arkhipchenko

Casey Durkin/E! Entertainment

Taylor showed even more vulnerability when he teared up after Pollard, 41, called him a “demon.”

“I took that to heart,” he says. “That’s the first time I’ve ever gotten emotional on television. That was a real tear. I was like, ‘You don’t even know me. You’ve met me five minutes ago and you’re calling me a demon.’”

Taylor and Pollard came to an understanding atHouse of Villains’ premiere party earlier this month, but the situation made him realize the difference between his background and those of his castmates.

“I come from a reality show that’s actually following our lives, not these shows that try to be other people and stuff like that,” Taylor says. “I didn’t figure out how her shtick is. I shouldn’t have took offense to it, but I did. But I didn’t realize until after I left the show who these people are and what they do and their games. Because after the cameras are gone, they’re not the same people. I am the same person.”

Jax Taylor (left) and Johnny “Bananas” Devenanzio on ‘House of Villains.'.Casey Durkin/E! Entertainment

HOUSE OF VILLAINS – Episodic – Pictured: (l-r) Jax Taylor, Johnny Bananas

Taylor admits he “should have learned a little bit more” about how his castmates’ shows work, because he felt at somewhat of a disadvantage when it came to going against people such as Devenanzio, 41, and Fairplay, 49, who come from competition shows.

Jax Taylor and his wife Brittany Cartwright in N.Y.C. in June.Noam Galai/Bravo via Getty

Jax Taylor, Brittany Cartwright

Noam Galai/Bravo via Getty

These days, Taylor doesn’t consider his normal self a bad guy.“That villain guy, it just is not as there used to be,” he says. “I got a wife now, I have a kid, so I have to work hard to be that ‘asshole.’ I needed that guy for this, and he just wasn’t there. It was a blessing to be in that house with those legends, but I was definitely out of my element on that show.”

Never miss a story — sign up forPEOPLE’s free daily newsletterto stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

House of Villainsairs Thursdays at 10 p.m. ET on E! and can be streamed on the E! app.

source: people.com