Comedian Jeff Ross on Tom Brady, Justin Bieber and the Only Celeb Who Yelled at Him After a Roast

Roast master Jeff Ross, who has been hosting celebrity roasts for 30 years, is turning the tables on himself.

The comedian is now starring in Take a Banana for the Ride, a new one-man show on Broadway where he tackles his own life history, including tragedies like losing his mom young and recently surviving stage-3 colon cancer. He also dives into the deaths of pals Bob Saget, Norm McDonald and Gilbert Godfried.

"It's my way of honoring the people in my life," he says of sharing the touching stories. (The show was named for his grandfather, "Pop Jack," who helped raise him and always told him to take a banana when he left the house, as a way of looking out for him.)

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

Ross, of course, also touches on some of his more famous roasts.

"Tom Brady was a great sport about it," Ross tells PEOPLE on the infamous roast, which has seen more than 2 billion streams.

"I've got to credit Tom in that way because there hadn't been a roast in a long time, and it was meaner than anyone thought it would be. He took it really well. In a way, that really helped reinvent him and his career, helped him cross over into his new career of being an analyst and team owner and not just a quarterback."

He knows Brady regretted some of the jokes that were made about his ex-wife, Brazilian supermodel Gisele Bündchen.

"Well, he's never going to hear another Giselle joke as long as he lives," Ross says. "It was a total reboot for him, and everything with his coach and teammates kind of came together. They had some icy relationships that all melted. So I'm proud to say I think the roast was really great for him."

Speaking to PEOPLE, Ross recalls other fun roast subjects over the years.

"Justin Bieber was a great sport. It was actually his idea. He knew he needed a reset on his public persona. A roast is a way to put your past behind you. Donald Trump was a good sport too — he's got a pretty good sense of humor. Well, he thought it was funny unless you made fun of his finances. I told him, "I read your book, The Art of the Deal. Why did it have three Chapter 11s? But even he laughed at that."

He says Charlie Sheen could have been three full episodes; there was so much material to work with: "We could have done an episode every week to compete with Two and a Half Men."

In fact, only one celebrity ever yelled at him after a roast, and it was Penny Marshall, who had once been married to Rob Reiner. She didn't appreciate Ross's line at the Reiner roast about their wedding song being "How Much is That Doggy in the Window."

"She came up to me months later at an event in Las Vegas and screamed at me over it," he says. "She was right, it was mean. But I learned my lesson to roast people who volunteer, and to make it something affectionate — to make it an honor and not a pure insult. I don't want people to feel caught off guard."

While he loves his roasting gigs, Ross is perhaps enjoying doing his one-man show even more for the time being.

The comedian likes being vulnerable and letting people know a new side to him. He also wants to show his appreciation for everyone who showed up for him after his cancer diagnosis. (He says his scans are now clear.) After the 8 weeks are up on Broadway on Sept 28, he says he might even take it on the road to keep it going.

"Billy Crystal warned me about that," he says. "He ended up doing his one-man show for five years."