Fixer Upper's Chip and Joanna Gaines Feared Marriage Struggles If They Kept Filming 

Chip and Joanna Gains shocked fans when they announced that the upcoming fifth season of their popular HGTV show 

It was a decision motivated by the couple’s desire to spend more time with their four children (Drake, 12, Ella, 11, Duke, 9, and Emmie Kay, 7) and not by the unfounded rumors of marital problems and divorce that have followed — though during an appearance on 

 Tuesday, Chip admitted he was nervous their relationship could go that way if they show continued.

“I give them a little credence,” the 42-year-old reality star said of the doubters. “For us, the most important thing in the world is Jo and I’s relationship followed very quickly by these four beautiful kids. We didn’t want to push it, redline it for so long that we woke up and realized we are at a point of no return. We wanted to take a step back and focus on the thing that was absolutely the most important thing to us in the world.”

“People in our inner circle, I tell them we’re as healthy as we’ve ever been, I just didn’t want to keep pushing this envelope to the fullest extent,” he continued. “We’re really thankful that we had the opportunity to step back and take a break.”

WATCH: “For us, the most important thing in the world is Jo and I’s relationship.” –@chipgaines pic.twitter.com/uUTDQqQxZ5

Is Ending: What to Expect from Chip and Joanna’s Final Season

For the last five years, the Gaineses have been working nonstop to build their home improvement empire.

 premiered in 2013, it quickly became the highest-rated show on HGTV, turning the couple into household names. The series also helped to launch their booming Waco, Texas-based Magnolia businesses, which includes a real estate company, the Silos (a retail store that draws 40,000 visitors a week), wallpaper and furniture lines, a quarterly magazine, luxury vacation rentals, an upcoming restaurant and a product line for Target.

together and pursued solo book projects (Chip’s new book 

RELATED: Chip and Joanna Gaines’ Empire: Every Product Line, TV Show and Side Hustle That Has Made Them Wildly Successful

But with all that success comes more work, responsibility and stress. Filming 11 months out of the year while juggling Magnolia and its 500 employees began to wear on them.

“Nobody knows when you get into something like this,” Chip told

, looking back. “I’m an idiot, I thought this was like a get rich quick scheme. Unfortunately, it’s not quite that easy. The emotional and the amount of energy it takes to pour something like this together. Every day we showed up to work we wanted this to be the real deal. We wanted it to be authentic, sincere.”

This guy right here… I get to live life with him. I get to hear his thoughts and benefit from his wisdom and encouragement on a daily basis. The reason I am who I am today is because Chip pushed me to go for it. People think of him as a jokester, and he is. It’s a huge part of why I fell in love with him in the first place. But he is so much more than that. He is the ultimate cheerleader, coach and dream maker. I am so thrilled that his book releases today mostly because I believe it holds truth that the world is in desperate need of. I am always proud to be Chip’s wife but today more than ever. I know this book was a true labor of love. Well done, Chip Carter Gaines! #CapitalGaines @chipgaines

A post shared by Joanna Stevens Gaines (@joannagaines) on

RELATED VIDEO: Chip & Joanna Gaines Most Lovable Outtakes!

Joanna, 39, agreed and thanked the fans for standing by them throughout the years even during the news that the show would be ending.

“We’ve kind of all grown together,” she said. “With the audience, they’ve come alongside with us. On the other side, there’s so much encouragement. Even though it was sad, they get it. We have this young, growing family and I just think that was one of the biggest reasons why we wanted to end the show.”

As for whether they’d return to reality TV, the couple played coy.

“Who knows?” Chip said. “We’re so excited about this break. But would we go back and do this again? Absolutely. It was a joy it was a pleasure. All the things you know about. It introduced us to Kansas.”

\'s Shannon Beador Gives a Tour of Her Corona Del Mar Home: See Which Room \'Good Friend\' Jeff Lewis Designed in Just One Day!

\'I Don\'t Want to Leave!\' Tiffany Thornton Shares Sweet Honeymoon Photos with New Husband Josiah Capaci

Interior Designer Eric Hughes Reveals the Celeb Client Who Wanted an ‘Over-the-Top, Studio 54 Powder Room’

Ree Drummond \'Totally Gets\' Why Chip and Joanna Gaines Ended