Chip and Joanna Gaines Explain Why They've Decided To End Their Show 

Chip and Joanna Gaines are walking away from their hit HGTV show

after the upcoming fifth season, and now the home improvement stars are opening up about what went into making their big decision.

“We gave everything we had to this show — the beautiful homes and those sweet families — but this just felt like the right time to catch our breath for a bit,” Joanna, 39, says in the latest issue of PEOPLE.

For the last five years, the Gaineses (who are parents to children, Drake, 12, Ella, 11, Duke, 9, and Emmie Kay, 7) have been working nonstop to build their home improvement empire.

premiered in 2013, it quickly became the highest-rated show on HGTV, turning the couple into household names. The series also helped to launch their booming Waco, Texas-based Magnolia businesses, which includes a real estate company, the Silos (a retail store that draws 40,000 visitors a week), wallpaper and furniture lines, a quarterly magazine, luxury vacation rentals, an upcoming restaurant and a product line for Target.

RELATED VIDEO: It’s Official! Chip & Joanna Gaines Will End ‘Fixer Upper’ After 5th Season

is out Tuesday. Joanna also has a design book in the works.

But with all that success comes more work, responsibility and stress. Filming 11 months out of the year while juggling Magnolia and its 500 employees began to wear on them.

“Nobody told us that that was unique,” Chip, 42, says of the grueling production schedule. “Nobody told us that that was a little bit irrational.”

The couple also noticed their kids were tiring of the busy production schedule.

“They would ask, ‘Mom and Dad, do you guys have to film again today? When is this going to be over?” Chip recalls. “The kids were so young when we started,” Joanna adds. “Fast-forward to today and our oldest is almost a teenager — they’re all growing up so quickly.”

WATCH: Chip & Joanna Gaines’ Most Lovable Outtakes!

Despite tabloid rumors that marital strife played a role in their decision, the pair says that’s downright false. With their marriage stronger than ever, “We would rather stop here, where we still feel we’re in a really good place,” Chip says.

“We have the chance to regroup and refresh and find out in two years, thank God we stopped and gave each other time and energy and loved our family more than we would’ve had the resources to do otherwise. Hopefully that will pay dividends down the road.”

Now Chip and Joanna are looking forward to spending more time with family and working on their Magnolia business.

“We really want to focus on this break and take a step back to let this all soak in,” she says. “Regardless of what the future holds, we are hopeful for what God has for us and our family in this next season.”

For more on Chip and Joanna Gaines pick up the latest issue of PEOPLE on newsstands Friday.

\'s Shannon Beador Gives a Tour of Her Corona Del Mar Home: See Which Room \'Good Friend\' Jeff Lewis Designed in Just One Day!

\'I Don\'t Want to Leave!\' Tiffany Thornton Shares Sweet Honeymoon Photos with New Husband Josiah Capaci

Interior Designer Eric Hughes Reveals the Celeb Client Who Wanted an ‘Over-the-Top, Studio 54 Powder Room’

Ree Drummond \'Totally Gets\' Why Chip and Joanna Gaines Ended