I always think of Harry Potter and the Chamber of Secrets as my least favorite Potter book. Then I reread it and I can't remember why. I guess it is a bit like deciding my least favorite ice cream from cookies and cream, mint chocolate chip, and banana split. they are all fabulous; you just have to pick one as the "worst" and go with it.
CoS is missing Hermione the last half of the book. And it has Gilderoy Lockhart, one of the most annoying characters in the series to my mind and has my vote for the worst of the DADA teachers (yes, Umbridge is EVIL, but at least she does seem to give the students the theory while Lockhart appears to just read his autobiographies in class). I would also say that there are big sections where nothing seems to happen.
however, we get Dobby, and the Polyjuice potion, and the Malfoys, and so many things that end up being important later in the series. so it gets a 5.