This is really a massive question with many answers, so forgive me for cutting out the fat from my response. It should first be established that the way missions have been undertaken, across thousands of years and a variety of religions, is just about as diverse as it could be. There is conversion by conquest, by hegemony, or by the explicit missions themselves, and each of these is its own field of factors.
An easy place to start with missionary work is whether or not the traditional beliefs of the unconverted peoples can be syncretized, or amalgamated with the missionary teachings into one new, cohesive body of beliefs. Craftier missionaries across time have integrated local beliefs or figures into their gospel through this technique of syncretism, but it is perhaps more common that the people themselves will syncretize the religion so as to reconcile their traditions with these new, often invasive, teachings. A good jumping off point for this is Louisiana Voodoo, a syncretic belief set which arose from the mixing of Catholic and West African teachings, wherein old-world rituals beseech Jesus or other Christian figures for blessing.1 For additional reading, consider the study of Haitian voodoo, a similar but distinct development of syncretic religion in the Caribbean. 2
Similarly, much of the surviving text from the Spanish conquest of Mesoamerica is from priests who were approaching their work anthropologically – first attempting to understand the peoples, and then attempting to preach to them with this cultural context. Certainly, some things were impermissible to the missionaries (cannibalism, ritual tortures, taboo sex practices) but you’d be surprised where and how a clever missionary can insert their own teachings into another peoples’. There is a silent body of anthropological work from missionaries the world over who approached things in this matter.
It may also be so that a foreign belief set and an indigenous belief set are sympathetic, and not difficult to integrate. Buddhism has blended with many belief sets in its lifetime (perhaps harder to call ‘mission work’ in our colloquial sense) though perhaps none more naturally than with Shintoism, the folk religion of Japan.
Furthermore, missionaries can be successful partially by incidence. Take the Islamic coversions in Southeast Asia in the mid-1000’s. Muslim missionaries arrived alongside traders, bringing unique goods and opportunities along with the word of their God. When your people are not doing great – possibly being oppressed – and a man arrives on a glorious ship and builds a glorious mosque, well, that has a lot of conversion power alone. Creating centralized places of worship and community interaction is a powerful tool of conversion for the missionaries, who are often providing some form of humanitarian aid on top of their gospel (though this is a good time to remind everybody that humanitarian missions can occur alongside conquests).
A lack of success bares less examination, at least in this forum, since the failures can be so multifaceted. The teachings may oppress the people in question, it may destablize the rulers, it may ask for unwelcome concessions – it may violate the hierarchy of society, disturb their ideals, impose unpopular changes (circumcision, anyone?) While I think there is a lot to say about unsuccessful missions, they are the default state of a mission, in my mind. That is why it is a conversion attempt.
So how do these replace an old belief set? Most often, they do not replace, but displace old belief sets. They remain, often in a compromised state, but they do often remain. Syncretism is the most likely outcome of a successful mission, though total conversion is entirely possible across the longitudinal scale, as the fluid entity of oral tradition is consumed by the edifice of gospel. Or, to cap it off with a sentence: you can convince an indigenous population to attend your sermons and observe your laws, but you can’t erase the cultural context which establishes their worldview. Not immediately, in any case, unless you intend to do things violently.
1 Hall (1995). Africans in Colonial Louisiana
2 Ray, Terry, and Karen Richman. “The Somatics of Syncretism: Tying Body and Soul in Haitian Religion.” Philosophy of the Social Sciences.
The Sociological Mail