We deal with suffering in one of two ways, or with two different worldviews I should say. In the one view, suffering is meaningless. It makes no sense to us. It looks like pure cruelty. And so, when we come to the question of God and suffering, we view God as cruel. Some decide to no longer believe there is a God as a result. Some decide to believe God still exists, but maybe He is too transcendent to deal with us and our problems, or maybe we just get angry with Him and decide, in one way or another, to no longer follow Him (as though that will punish Him in some way more than it simply being a further punishment of ourselves). In any case, we view suffering in the temporal. We see it in terms of imminence and the emotion and pain that it immediately stirs in us. We view it as it occurs in space and time, not in terms of what is eternal (either because we have worldviews that don't believe in the eternal, or because the pain causes us to focus on the moment like a baby who cannot see that his parents only wish for him to be cured by the shots being administered, but only sees them as cruel and their actions as a betrayal of trust).
The other view is to see all things from a more transcendent perspective. This view acknowledges that no suffering is pleasant. All of it is to be lamented. But there is meaning in all of it, simply because God is involved, and God is good. Hence, come what may, God is working through it to bring about a greater good (like the parent who administers the shots to save his child).
It is interesting to find both of these perspectives in the Bible. Although the two perspectives can be found throughout, the two books of Genesis and Job deal with the issue head on. They each individually work to teach us something about God, His goodness, and suffering/evil in the world, but they also work to clarify by providing a canonical context to one another. In fact, both books share some rare archaic words, and along with that, and their use of divine names to portray God's closeness and distance, as well as the subject matter, make me believe that one is written in a response to the other, or by the same author or school. In any case, which came first is difficult to know, but what they say is brilliant and are God's words to us in times of pain and suffering.
In Job, we are presented with God from both the transcendent view and the imminent view of suffering. Here, unlike Genesis, however, the author reverses the use of the divine names. YHWH, which is used in Genesis to portray God's imminence and revelation of Himself to us in the day to day, is used here to show God's hidden love for Job. The name YHWH is relational, and it conveys God's love of Job as an upright man who God acknowledges as having His favor. But Elohim, unlike in Genesis, where it is used to convey God's transcendence and sovereignty which is largely hidden from us (as it resides in the eternal perspective), is used to convey how Job perceives God in his suffering (i.e., God is aloof. God does not care. God is hidden from Job's sight and is distant). We are left with God answering Job in the end, but only with a display of His sovereignty and rebuke of Job that he must trust in God's right to do what He has done, to bring about the suffering He has directed Satan to accomplish. We are never told why the suffering exists. There is no mention of a greater good here. Only that God knows what He is doing, and one must submit to that.
As said above, Genesis does the exact opposite. It presents YHWH from our perspective, the view from below. YHWH relates to us by entering into the conflict with us. It looks like He loses battles with evil. It looks like He struggles against evil and suffering, precisely because evil still overwhelms the earth, we still get sick, we still starve, we still die. But Elohim is used to show us the perspective from above. God is transcendent, and not under any threat of losing His sovereignty over evil. Instead, His entering into the struggle, from our perspective, is nothing more than His sovereign use of evil and suffering to accomplish His complete victory over it. In other words, it is through suffering that suffering will ultimately be defeated (e.g., Gen 50:20). Even evil itself, then is the means used for its very destruction. Suffering accomplishes the greater good.
Both of these theodicies (and I do believe both of them to be such despite the pull of other theories to go elsewhere) convey the idea that suffering is very real and it is very painful. They both portray God as sovereign. They both grapple with His goodness in light of suffering. But they both have different answers that I think are very complementary to one another, and provide us with a solid worldview to make sense of God's existence, His goodness, and the presence of evil and suffering.
First, Genesis teaches us that the view that sees evil as triumphant is only a finite perspective of what is going on. It's just the view from the ground. It's the view from below. It's the view that sees Joseph thrown in a pit by his brothers, unjustly put in prison to rot for years. But it stops there. It can only see the pain in the moment. The emotion of suffering clouds its long term judgment. It cannot evaluate reality as it is. It can only see the now. It sees suffering win. It sees God, for whatever reason (whether He is too weak or not good enough to care) fail us. It looks around but not up. It's the view from prison.
But Genesis teaches us that this isn't the whole story. There is God's perspective that is eternal. His view is actual reality in terms of what is going on and why it is going on. In this perspective, we see God overcoming evil through evil. We see Him destroy chaos with very acts of chaos. He sends a flood to destroy agents of suffering in order to save His children. He throws Joseph into a pit and then into prison through the hands of evil men in order to save many lives in the end. The suffering He used to bring about the good result of salvation is like the parent saving His child by harming him with the needle. But in our perspective, He's just being cruel, because we look out at Him like the baby who doesn't understand and only sees the parent as betraying him, harming him, being cruel to him, not caring for him. Yet, Genesis teaches us that this is the greatest of ironies, since it is God's very care of His children that has caused Him to act to harm us in the temporal sphere. These are acts of salvation.
Job, however, clarifies something important for us, though, and that is that we don't always get to see the good. We don't always get to understand why this or that occurred for such and such a purpose. We can read Genesis and feel that God is vindicated, but we read Job and have no sense that what He has done there was right. This is why the book compels us to listen to God's sovereignty in the end chapters. It wants us to know that we will never be able to see all of the tied up ends as God can, but we can know that God is in control. So even if the good that God was working toward in a specific act or life of suffering is not known, Job tells us to trust in Him anyway, and Genesis tells us to do so because, even if we don't know the specifics, we know that God is good and works toward the good of His children.
A great scene in Genesis displays this beautifully. Before God goes to destroy Sodom, he tells Abraham that He's going to destroy it. Abraham then questions God's goodness for possibly killing the righteous with the wicked. Abraham eventually pleads God down to destroying the city, only if He cannot find 10 righteous people in the city. His words are continually meant to call God's goodness into question.
But what is so great about the passage is that God shows Himself to be better than Abraham. Where Abraham was insinuating that God wasn't as concerned about doing what is right as Abraham was, God shows Himself to be completely just and good and far exceed Abraham's claim know what is right. God actually only finds one righteous man, Lot. That's it. His wife is wicked and turns back. His sons-in-law don't believe him. Even his daughters, who make it out, prove to be shady, as they get their father drunk and sleep with him later on. The point is that God is not unrighteous so as to let the wicked go and to punish the righteous for no reason. Instead, whereas Abraham would have saved the wicked over 10 people or let 9 people perish for the sake of destroying the wicked, God destroys the wicked and saves even a single righteous man.
The point, of course, is that God is better than our perceived judgments of Him, and He is in fact better than we are. If we could see from His perspective, we would see that, but we are often trapped in the pain in the moment, and it binds us to the view from prison. What Job and Genesis do, however, is encourage us, not to ignore that we are in prison, but to look out the window and up in order to make sense of it. Prison has a purpose. Sickness has a purpose. Loss has a purpose. Death has a purpose.
We live in pain, but this pain is not our end if we have become His children. He is good. He is sovereign. And He is using what is painful and what seems to us to be overwhelmingly meaningless, toward a most meaningful goal: to save our lives. Whether we can always see how that works out is irrelevant. The point is simply that if we believe God is good and sovereign, it always does work out.
And we know that God causes all things to work together for good to those who love God, to those who are called according to [His] purpose. For whom He foreknew, He also predestined [to become] conformed to the image of His Son, that He might be the first-born among many brethren; and whom He predestined, these He also called; and whom He called, these He also justified; and whom He justified, these He also glorified. What then shall we say to these things? If God [is] for us, who [is] against us? He who did not spare His own Son, but delivered Him up for us all, how will He not also with Him freely give us all things? Who will bring a charge against God's elect? God is the one who justifies; who is the one who condemns? Christ Jesus is He who died, yes, rather who was raised, who is at the right hand of God, who also intercedes for us. Who shall separate us from the love of Christ? Shall tribulation, or distress, or persecution, or famine, or nakedness, or peril, or sword? Just as it is written, "For Thy sake we are being put to death all day long; We were considered as sheep to be slaughtered." But in all these things we overwhelmingly conquer through Him who loved us. For I am convinced that neither death, nor life, nor angels, nor principalities, nor things present, nor things to come, nor powers, nor height, nor depth, nor any other created thing, shall be able to separate us from the love of God, which is in Christ Jesus our Lord. (Rom 8:28-39)