Seems like quite a few people also posted the same issue as me and it was also solved by simply progressing in the story. Shadow of the Tomb Raider Full Walkthrough All Story Missions Septemby PowerPyx 6 Comments Here you can find a full Shadow of the Tomb Raider Walkthrough of all Story Missions (Chapters). The difference is that it's not clearly communicated anywhere when you can't do side missions so it's confusing and people (like me) think the ladder disappearing is a bug. So now i think the ladder disappearing isn't a bug, it's just that there are some parts in the game where you're not allowed to complete any side missions sort of like there are parts where you're not allowed to fast travel.
Sure enough after some story progression the ladder returned, the guy reappeared in the cave, and i finally completed the side mission. I thought maybe i just had to progress in the story to make him come back. So i noticed that all the chevron symbols that represented side missions had disappeared from the map and decided to check the cave where the mission giver was and surprise, surprise he was gone too.
