Prison?

There's no hint of him being in prison in the movie.
From his interaction with his sister we found out: that they are estranged but he cares about her, their father is in a nursing home and "not the same person as before"( probably abusive), and not even in this circumstance will he visit him. His simple life is his own, on his own terms, and lived with intention and peace of mind. Things he apparently didn't get from his family.

Wim Wenders:
"He was a businessman and he was rich and he was unhappy and he was drinking a lot and his life was going down the drain. One morning he wakes up in this crummy hotel room, doesn't even know how he got there, doesn't even know if he had sex or whatever happened. He thinks his life is shit, and he doesn't like it. He actually plays with the idea of ending it. Then, miraculously, early in the morning, there's this ray of sunlight appearing on this wall in front of him. And it falls through the little tree in front of the window. There is this play of leaves and sunlight and shadows moving, and he looks at it and stares at it and he starts crying, because he's never seen anything so beautiful. He probably has seen it, but he hasn't noticed. Then he realizes that's the answer to his existential crisis, to become somebody who notices that. He gives up his expensive car, his business job and becomes a gardener and eventually, the guardian of these toilets, because they're all in little parks. Somehow, they found Hirayama as the ideal character to take care of them. That was the backstory."