Thomas Kuhn's The Structure of Scientific Revolutions provides a tentative outline for the process that governs the progress of science from one paradigm to another. A paradigm is a kind of theoretical framework or conceptual space which delimits the possibilities for scientific theorizing or marks the boundaries of scientific acceptability. A paradigm sets the stage of investigation, it provides all the basic tools for thinking and explanation. In other words, a paradigm is not just a theory or collection of theories, but is a logical substructure which determines the conditions of possibility for scientific theories at a given point in history.
The most important recent paradigms in physics have been Newtonian Mechanics, Relativity, and Quantum Mechanics. The Newtonian Mechanical paradigm framed physical research by conceiving space and time as uniform and infinite. Further, under the Newtonian Mechanical paradigm explanation consisted only in a mathematical formulation of how moving bodies make contact with and/or directly physically interact with other moving bodies. The transition from the Newtonian Mechanical paradigm to the Relativistic paradigm constituted a paradigm shift because some of the most basic assumptions of Newtonian Mechanics were overturned. For example, in Relativity space and time are not uniform. Rather, very massive bodies have such strong gravitational fields that they actually bend and warp the fabric of space-time itself. Similarly fundamental principles were overturned in the transition to Quantum Mechanics.
But how, precisely, do paradigm shifts occur? How is it possible to move from one paradigm to another which is conceptually incompatible with it at a fundamental level? According to Kuhn, there are contradictory tendencies/forces at work which produce the dialectic that eventually results in a paradigm shift. These opposed forces are 1. the attempt to fit anomalous observations/experimental results into existing theories (whether through reinterpretation of data or of theory) and 2. tension caused by an increasing number of observations/experimental results which don't fit comfortably or clearly into existing theory.
The reason why I have provided a rough outline of Kuhn's thoughts here is because I think a nearly identical process occurs with each individual's view of the world. A person holds a worldview at a certain point in time because it makes the best sense out of the information they have about the world. However, no one's view is able to satisfactorily account for all possible information they will encounter and many views will frequently encounter large amounts of information that just can't be properly made sense of. Therefore as a person learns more, as they encounter more and more information, more anomalies, inconsistencies, and unexplained phenomena will inevitably arise. After enough anomalies have accumulated the worldview breaks down and gives way to a new, more subtle and sophisticated one which is able to take more of what one knows about the world into account.
Of course, this is no smooth path to enlightenment composed of fluid transitions from one worldview to the next. Just as in the process of scientific paradigm shift, we all have a conservative tendency which leads us to reinterpret anomalous information to fit our existing worldview (or, if this can't be done, we try to slightly tweak our worldview in order to accommodate new information and thereby save the view from being displaced or discarded entirely). And this conservative tendency is not (only and always) motivated by xenophobia or complacency. There are legitimate epistemic as well as emotional and social reasons for clinging to an old view. The view any one person holds is the result (conscious or not) of the attempt to make sense of the world around them. It is clear from the fact that they remain alive that every person has a view of things which is at least practical/accurate enough for them to survive in the world. This track record of keeping us alive gives some level of credence to each person's worldview (especially for the person who holds the view). Further, most of us love being alive and want to continue to live so the efficacy of a worldview in aiding us in our continuous quest to stay alive or even thrive leads us to place emotional value on it. Last, none of us forms our worldview in a vacuum or alone in a cabin in the woods (although this can sometimes be helpful in forming a more sophisticated and critical view of things, c.f. Thoreau) but we form it in cooperation and interaction with those of our community. We learn an enormous amount of valuable information from those we interact with as well as depend on the other members of our social circle for many of our basic needs. We therefore often feel a level of social commitment to our worldview. For some, changing worldview can feel like betraying dear friends. A combination of these factors leads many of us to cling to whatever worldview we happen to have at the moment.
While some of our motivations for clinging to our current worldview are legitimate, there are a number of motivations for doing so which are not (or are considerably less so). Some of these motivations include personal weakness and wishful thinking. While it is certainly the case that these kinds of motivations can be equally compelling for an individual, they are more difficult to excuse from an outside perspective.
Further, and again as in the case of scientific paradigm shifts, there are also characteristics of worldviews themselves which inherently resist change. The feature of worldviews which generates the most resistance to change is the fact that worldviews determine fundamentals, delimit the scope of possibility, and generally force certain interpretations on us. As discussed before, this delimiting function of worldviews is not only a bad thing. Rather, it is a double-edged sword. These fundamentals are concepts and ideas that have been highly useful/successful in past experience and so have been codified (consciously or not) as basic, more or less unbreakable rules. It is a very good thing, for example, that we thoughtlessly/automatically avoid touching objects that have proven to be painfully hot in the past and do not, for the sake of intellectual rigor, subject this prejudice to frequent scrutiny by grabbing the flaming end of a burning branch every several months. But, as I've said, worldview fundamentals can also generate prejudices which are as violent and anti-social as they are persistent and resistant to change. One's religious beliefs may provide a set of fundamental concepts for thinking about e.g., homosexuality which forces one into the interpretation that this way of life is perverted and the work of evil spirits. The forced nature of this type of conclusion exempts the bigot from blame to some extent; it is unlikely that they are able to draw any other conclusion when looking at this issue in isolation. However, as the religious individual surveys the world and forms an increasingly complex network of automatic judgments which prevent her from interacting peacefully and productively with her culture and world, she becomes increasingly responsible for letting go of old emotional and social commitments and allowing her worldview to be torn apart by the growing tension of unresolved contradictions.