If someone asks if you want to take the stairs or the elevator, saying you’ll take the ladder will probably confuse them.
The term “ladder” can also be used metaphorically–to climb the corporate ladder or the ladder to success, for example. Whereas a physical ladder allows one to climb up and down, most people don’t want to go down a metaphorical ladder.