I'm going through a re-read of the series as an adult now so I may change my mind. But growing up, I really liked Goblet of Fire. When I first got into the series, GoF was just released. So I kept re-reading them and it always ended on book 4 and Voldemort returning, which had such an impact. It wasn't until 3 years later that book 5 came out and I could finally continue the story. So book 4 always stuck with me. I also really like how it broadened the world by introducing us to other schools and the Tri-wizard tournament. Plus the whole mystery aspect of Crouch jr was fascinating.