We know the overall story of Israel's failure in the wilderness. Of all the people who began the journey, only a handful actually entered the Promised Land. This reality is intended to illustrate how much the personal relationship with God means to our salvation. By and large, the Israelites of old had no personal relationship with Him.
This is the broad answer as to why this great mass of people failed. They did not believe God. The author really drives this point home by emphasizing it as an "evil heart of unbelief." It was evil because it would not permit them to faithfully submit to Him; it caused a struggle within them and led them to disobey. Israel's failure, despite God's many works in their behalf, illustrates that just believing that God exists is not sufficient for salvation.
This weakness of faith is not unique to them. The extensive coverage given in the Bible to the Israelites' release from Egyptian slavery and their subsequent failure in the wilderness is fully intended for our learning. Their failure followed their symbolic calling from the world, baptism in the Red Sea, receipt of the law, and acceptance of the covenant at Mount Sinai.
Their failure occurred at a time equivalent to our doing so after conversion. They failed to grow, overcome, and remain faithful during their testing in the wilderness. Are we not now making our spiritual wilderness journey? Have we not had people who once fellowshipped with us stop walking with us spiritually?
I firmly believe that the current scattered condition of the church was deliberately caused and executed by God, not Satan. Why? To test and to build our faith. Being in a church with large congregations is deceivingly comfortable. Such a circumstance tends to produce complacency, as Christ's message to the Laodiceans shows. But over the past 15-20 years, the practical day-to-day faith of those in the church of God has been seriously challenged.
Many have departed from fellowshipping with their brethren. Consider the use of the term "departing" in Hebrews 3:12. As seen from God's point of view, this is a strong warning, since He considers their hearts to have been "evil." The translators smoothed over the Greek term, aphistemi, underlying "departing from." Strong's Concordance says that it means "remove," "instigate," "revolt," "desert." The historical context indicates a stronger wording, "rebelled against," more clearly seen when compared with the faithfulness of both Christ and Moses, as extolled in verses 1-3.
In The Daily Study Bible Series: The Letter to theHebrews, commentator William Barclay translates this verse, "Have a care, brothers, lest an evil and disobedient heart be in any of you in a state of rebellion against the living God" (p. 32, emphasis ours). That is how God sees the destruction of their relationship with Him. When viewed within the context of the entire book of Hebrews, which extolls the greatness of Jesus Christ, our sovereign God, the rebellion that Paul is warning against is the turning away from a living, dynamic Person, not merely from some vague belief in a distant God.
It is very difficult to believe that the Israelites did not believe God exists after all the powerful witnesses they were given at the Red Sea and Mount Sinai. They had the same human nature that we do. Their problem was trusting Him, being faithful to Him in the daily activities of life, as one should be in a marriage. They had this problem because they really did not know Him, and they did not know Him because they did not seek Him.