"It was crazy... There were some who were telling me how horrible I was, that I was psychologically damaging my kids."