Well, he went 36-44 and took them to the playoffs in four years. My guess is they gave up on him too soon.
Just imagine what the Browns would have been if they'd kept him. Nick Saban with the Dolphins was 15-17 and no playoffs in two years. Eventually he quit on the team and ran to Alabama...