The Taoyuan Airport Line isn't getting great ridership: 130,000/day or so, less than the Wen-Hu Line. Perhaps that's due to terrible transfers. More telling is that only 20% of the riders are taking it to the airport. In other words, people find it more useful to go to places other than the airport. Seen another way only 25% of the people using the airport take the train there.
It is extremely difficult for rail to airports to work. Airports simply aren't massive destinations compared to downtown business or commercial districts- just think how often you go to your job or even go somewhere central for shopping or entertainment, versus how often you go to the airport. Building a rail line around getting to the airport was a mistake on Taiwan's part, as it does little to decrease car usage and therefore does little to help the environment, and isn't useful for most people or financially self-supporting to boot.
Relevant article:
http://news.tvbs.com.tw/life/820010