Einstein and "The Old Man"

Because Einstein was a highly respected figure, what he said about God and religion, are often quoted to support various views from atheism to all shades of religions or beliefs.

What is actually his relation to God, or "The Old Man", a term he sometimes used to refer to God?

A way of speaking.
To begin, it seems stupid to point out that when he says "The Old Man does not play dice", he does not in any way imply that God exists, any more than implying that God is male. It is just a way of speaking common with many. This much is obvious, yet there are people who uses such quotes in discussions to imply that Einstein believed in God, instead of reading it as meaning that the universe is deterministic, not random.
I used the word "I" quite often, although I believe in non-self, because it is convention in communication.

Being religious.
One of his often quoted statements is "Science without religion is lame. Religion without science is blind."
As a scientist, he expressed the importance of religion and beliefs in general.
But religion does not necessarily mean God, and God does not necessarily mean a personal God who has a child, gets angry, and generally interferes in human affairs.

Being religious for him means "To sense that behind anything that can be experienced there is something that our minds cannot grasp, whose beauty and sublimity reaches us only indirectly, this is religiousness."
And "Try and penetrate with our limited means the secrets of nature, and you will find that, behind all the discernible laws and connections, there remains something subtle, intangible, and inexplicable. Veneration for this force beyond anything we can comprehend is my religion. To that extent I am, in fact, religious."

In other words, being religious is some form of incompleteness (Dukkha) of science.
Religion is like a hypothesis to be verified - then it becomes science - or refuted and discarded, while humbly accepting that there are always things we don't understand.
It is not dogma. It is not absolute.
Einstein also rejected religious rituals, just as J. Krishnamurti did.

Rejects Personal God.
His rejection in a personal God began already when he was in school, and he repeated this rejection many times later.

"I cannot imagine a God who rewards and punishes the objects of his creation, whose purposes are modeled after our own -- a God, in short, who is but a reflection of human frailty."

"The idea of a personal God is an anthropological concept which I am unable to take seriously."

"I do not believe in a personal God and I have never denied this but have expressed it clearly. If something is in me which can be called religious then it is the unbounded admiration for the structure of the world so far as our science can reveal it."

"I have repeatedly said that in my opinion the idea of a personal God is a childlike one, but I do not share the crusading spirit of the professional atheist whose fervor is mostly due to a painful act of liberation from the fetters of religious indoctrination received in youth. I prefer an attitude of humility corresponding to the weakness of our intellectual understanding of nature and of our own being."
In this last quote, he preferred the agnostic attitude, and rejected atheism.

What kind of God?
If not a personal God, then in what kind of God did he believe in?
"I believe in Spinoza's God who reveals himself in the orderly harmony of what exists, not in a God who concerns himself with the fates and actions of human beings."

The closeness of Spinoza's God and Buddhism, both believing in a cosmic religious feeling, led him to say:

“The religion of the future should transcend a personal God and avoid dogma and theology. Covering both natural and spiritual, it should be based on a religious sense arising from the experience of all things natural and spiritual as a meaningful unity. Buddhism answers this description…If ever there is any religion that would cope with modern scientific needs it would be Buddhism.”

Science and religion.
The distinction between "what is" and "what should be" describes the all important difference between science and religion. The boundary of "what is" and "what should be" is not strict, there is interaction between the two:

"Yet it is equally clear that knowledge of what is does not open the door directly to what should be. One can have the clearest and most complete knowledge of what is, and yet not be able to deduct from that what should be the goal of our human aspirations. Objective knowledge provides us with powerful instruments for the achievements of certain ends, but the ultimate goal itself and the longing to reach it must come from another source. And it is hardly necessary to argue for the view that our existence and our activity acquire meaning only by the setting up of such a goal and of corresponding values. The knowledge of truth as such is wonderful, but it is so little capable of acting as a guide that it cannot prove even the justification and the value of the aspiration toward that very knowledge of truth. Here we face, therefore, the limits of the purely rational conception of our existence.
But it must not be assumed that intelligent thinking can play no part in the formation of the goal and of ethical judgments. When someone realizes that for the achievement of an end certain means would be useful, the means itself becomes thereby an end. Intelligence makes clear to us the interrelation of means and ends. But mere thinking cannot give us a sense of the ultimate and fundamental ends. To make clear these fundamental ends and valuations, and to set them fast in the emotional life of the individual, seems to me precisely the most important function which religion has to perform in the social life of man. And if one asks whence derives the authority of such fundamental ends, since they cannot be stated and justified merely by reason, one can only answer: they exist in a healthy society as powerful traditions, which act upon the conduct and aspirations and judgments of the individuals; they are there, that is, as something living, without its being necessary to find justification for their existence. They come into being not through demonstration but through revelation, through the medium of powerful personalities. One must not attempt to justify them, but rather to sense their nature simply and clearly. (The World as I See It)"

This is how conflict often arises: "For science can only ascertain what is, but not what should be, and outside of its domain value judgments of all kinds remain necessary. Religion, on the other hand, deals only with evaluations of human thought and action: it cannot justifiably speak of facts and relationships between facts. According to this interpretation the well-known conflicts between religion and science in the past must all be ascribed to a misapprehension of the situation which has been described."

The world would be more peaceful if we all adhere by the limitations of both science and religion.

Related:

• Some of my favorite quotes: Einstein
• Science and Buddhism
• The Hyperplanes of Science and Religion
• Einstein: The World as I see it