Marlon Wayans is revealing why his family walked away from In Living Color at the height of its success.
The sketch comedy show, created by his brother Keenen Ivory Wayans, premiered on Fox in 1990 and aired for five seasons until 1994.
While the show served as a launching pad for various A-listers, including Jennifer Lopez, Jim Carrey and Jamie Foxx, it ultimately proved to be a family affair as Marlon, 52, and Keenen, 66, were joined by their siblings Damon Wayans, Kim Wayans and Shawn Wayans.
Marlon shared during an appearance on the Wednesday, Sept. 4, edition of the Club Shay Shay podcast that his family left the show after its fourth season because “it was hurting Keenen’s pockets.”
“I remember when we left In Living Color because Fox was syndicating the show and it was hurting Keenen’s pockets. Keenen was like, ‘Nah, you ain’t gonna do this to me,’ ” recalled Marlon. “My family said, ‘F— this money.’ And we all left In Living Color.”
The comedian recalled the 1992 Christmas episode “Driving Miss Schott,” in which he and his siblings decided to protest the show by wearing black sunglasses as Foxx, 56, sang “This Christmas.”
“You ain’t going to f— my brother. We a family. You touch one, you touch all,” said Marlon about their decision.
The PEOPLE Puzzler crossword is here! How quickly can you solve it? Play now!
While the actor states that they were offered “big checks” to stay on the series, he says that his family decided to “support” their brother and leave.
“That’s why season five, it got weird. It got Biz Markie. God bless Chris Rock, [but] he came at the wrong time. As brilliant as he is, he couldn’t save it. Because the eyes of the show is gone,” said Marlon.
Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.
After leaving the series, Marlon said he was “broke,” having “only $700 bank and $900,” which motivated him to work on The Wayans Bros., which aired for five seasons from 1995 to 1999.