Honestly, the first three Harry Potter games were so lame (except perhaps the second). The fourth was an improvement, but still it's more like an action game than anything. The fifth was great, mainly because of the environment reconstruction. But they could have improved the game play and the cutscenes (they suck). I mean, doing chores is fine but going around the castle like 100 times to do it is not cool
For me, the hardest part was in OOTP (PC). Finishing the story is simple, but discovering all locations and beating it 100% were like, hard!

It would have been more easier if they had more shortcuts or something like that to travel around. In the other games, i guess beating the basilisk in CoS and finshing 100% in Gof (both PC). The Voldemort level in the 1'st version of SS (PC) is hard, but once you get used to it, it's easy