"If his offering is a burnt sacrifice of the herd, let him offer a male without blemish; he shall offer it of his own free will at the door of the tabernacle of meeting before the Lord." Leviticus 1:3 (NKJV)
Many people think that the offering had to be perfect to honor God. This was not the case. The unblemished animal used for the burnt sacrifice had to be perfect because the flawlessness of the animal was transferred to the guilty, blemished people.
God is not out to destroy humankind for every wrong they commit, in fact, God gave us animal sacrifice as a way to open the door to His presence. Sin cannot exist before the face of God (Exodus 33:20). Since the fall of Adam, animal sacrifice enabled God to have a relationship with humankind.
When Adam's first-born son, Cain, brought his offering of first-fruits before God, there was nothing wrong with his offering. God simply had to look away from Cain because it did not provide redemption. This was not the case with Abel so it offended Cain.
The Young's Literal Translation of the Bible reveals it to us this way; "Abel, he hath brought, he also, from the female firstlings of his flock, even from their fat ones; and Jehovah looketh unto Abel and unto his present, and unto Cain and unto his present He hath not looked; and it is very displeasing to Cain" (Gen. 4:4-5).
God then says to Cain, "If you do well, will you not be accepted? And if you do not do well, sin lies at the door" (Gen. 4:7, NKJV). The word translated as sin in this verse, is the Hebrew word hatta't*. This word is more appropriately translated here as sin-offering. In other words,"the sin-offering lies at the door."
Any offering that had to do with the redemption of sin had to be, an animal without flaw, grain that was beaten into flour, or wine pressed from the vine (blood from grapes). This is why, Jesus was flawless, beaten, and drained of blood. God never has expected us to live up to His standards on our own.
There was not anything wrong with Cain's offering, it was actually a good offering. It was a portion of his increase. This would have permitted God to prosper his crops. This offering just did not provide redemption. It seems that Cain had a misconception; if he pleased God with his work, God would not turn away. God had to turn away for Cain's sake. God told Cain to provide a sin offering to enter the door. This whole reasoning is reiterated in our verse at the top of this post.
God wants to have a relationship with His children. The only perfect solution was to provide the perfect sacrifice---His Son Jesus. God is all about giving.
* Strong’s Greek and Hebrew Dictionary, copyright 1890 by JAMES STRONG, MADISON, NJ. Strong’s number, H2403.