You should attend hearings. Even if you can't afford representation.
If you don't, the judge will assume you are a pikey scumbag. No offence meant, and I'm sure you're not - doesn't sound as though you are from your story. I'm putting it in those terms so you can see what it might look like to a room of people who don't know you and only hear one side of the story. As you've found out; if you don't turn up the other side are also more likely to obtain whatever costs order they ask for.