Your ticket prices are too high. You can use a tool like arena management on buzzer manager to help you set more appropriate prices.
The difference at the start of the season was due to the fact you won your division last season, so to help with attendances in the higher division (which is supposed to be at a harder level), the game gives you an attendance bonus for about the first half of the season. That's why you've seen your crowds diminish, more so lately.