The term "Sons of God" is a phrase that occurs throughout the Bible and seems to refer to a specific class of divine being. See especially Psalm 82:6-7.

I just don't see how Genesis 6:4 can refer to ordinary humans. I'd urge you to revisit that idea.