In my experience the skill students have most difficulty with is listening.
When reading, there is a visual support and students can refer back to what they've read. When writing, students have time to organise their ideas and draft out their answers. Speaking can be difficult as a lot of skills are involved: pronunciation, grammar, register, vocabulary; not to mention thinking about what you want to say.
When listening, concentration is required to catch everything, or at least the gist of what is being said. If essential information is being given out, for example the time of a train, extra effort is required. If this is happening in "real life", i.e. listening to another person speaking, there is always the possibility of asking the person to repeat. You could say that there's a limit to how often you
can ask for repitition, especially in a busy train station!
In the classroom, if students are listening to a cassette, other factors are involved. The first would be the quality of the recording. Another would be the stress involved in getting the answers right. Students might feel they have to get the answers right the first time they listen. When answering questions, there are two options a) write it down (this affects concentration for the next section of the tape) or b) try to remember the answer and write it down at the end. This too is difficult. Consider if you were listening to the news with a friend and an accident is reported. You would retain some of the details and the fact that a large number of people suffered but you might not remember how many exactly. This is another issue, that people have different capacities for listening in their own language.
This is not to say that all is grim. Much can be done to help students approach a listening task. You've asked me which I think is the most difficult. This answer is my personal opinion based on my experience. Other teachers may give other answers.