A motherless teen boy has a neglectful father who puts him into boarding schools and camps. When he gets into trouble again, his headmaster challenges his father to spend the summer with him, on a small Maine island, way Down East, where they can finally learn a lot about each other and other people.
I read this book in sixth grade (a specific memory) and have looked for it since then, but Google searches turned up nothing but the quote from Caesar for years. It's a very interesting coming-of-age story in sort of a classic, trope-y style.