Yeah, like E.B.W. said, when you lose a lot, the fans respond by not showing up. This is especially the case with bleacher and lower tier sales; they are very sensitive to recent wins and losses. If priced correctly, the LB and courtside seats should be more stable, though losing every game (which means every TV game, every rivalry game), will erode their attendance.
One thing to note, is that this season you'll have an extra penalty to attendance for the first half to two-thirds of the season. Just as there is a bonus to attendance when you promote, there's a penalty when you relegate. That probably explains part of the poor attendance from your first game this season. You'll have to do your best to win away games, and TV/rivalry games, in order to keep your fan survey as high as possible.