Bunji was not in good health during his early years. As he recalled his childhood experiences when writing the Konko Daijin On-Oboegaki, he remembers how he often suffered from stomach illness. However, he was an assiduous worker. Bunji would carry more pine branches than other village boys, and he voluntarily worked for the village construction. Through this way, he gained great trust from the villagers.

In 1831, when Bunji was 17, his adoptive mother gave birth to a son. They named him Tsurutaro. However, Tsurutaro died of an illness at the early age of five. To compound this tragedy, Bunji’s adoptive father Kumejiro, who was now the age of sixty-six, contracted a disease and a few weeks later followed his son in death.

With the death of his adoptive father, Bunji became the head of the household at age 23. He changed the family name to Akazawa, according to his adoptive father’s will. Bunji’s marriage was arranged at this time, and he married Furukawa Tose.