This is a really good question. I believe that the answer will not be a simple one. Jesus did say that “it is out of the heart that the mouth speaks,” (Mark 7:21 and Matthew 15:11) and it is what comes out of a man which makes him unclean (same verses). Clearly, it is when we speak angry words that we hurt other people. James calls the tongue, when used wrongly “a fire, a world of evil among the parts of the body. It corrupts the whole person, sets the whole course of his life on fire, and is itself set on fire by hell.” (James 3:6) We can make a biblical case that it is the evil thoughts which we actually express with our tongues which causes harm to others and is sin, not our unspoken thoughts.
Yet, a strong case can be made that God definitely cares about the thoughts in our hearts and that our heart-thoughts can be sinful as well. I believe that you are on to something in making a distinction between what we think and what we say, as the effect on others is greater when we speak. However, to say that no thought can be sinful unless it is spoken is proved to be wrong (or perhaps incomplete) by many scriptures in the Bible. In fact, if you go back to the Mark passage, what he is really saying is not that only what comes out is evil. What he is actually saying is that what goes in (for example food) is not what makes us unclean, but it is what is on the inside (which may be expressed with our tongue, but may not) which makes us unclean. For example he lists “evil thoughts” (Mark 7:21) as something which makes us unclean. In the Bible it is stated many times that God looks at the heart (for example 1 Sam 16:7). “I, the Lord search the heart” (Jeremiah 17:10) and “the heart is deceitful above all things” (Jeremiah 17:9) If we look at a list of sins in Galatians 5:19-21 we can see that Paul alternates from things we do or say (sexual immorality, debauchery, witchcraft, discord, etc.) and things we think or contemplate in our hearts (impurity, idolatry, hatred, jealousy, selfish ambition). A strong, indeed an iron-clad case, can be made that evil thoughts which are never actually expressed can be sinful, although when we speak those thoughts, we hurt others and we compound or add to the sin.
I will agree with you that what comes into our mind is not always completely under our control. There is an aspect of our thoughts that are not controlled by our volition. If so, such thoughts I would describe as temptation rather than sin. It is only when we take such a bad thought, cogitate on it, and dwell in it that it becomes sin. James 1:13-15 supports this idea. Such an evil thought comes from within–from our own evil desires, but only when conceived–only when it finds a place in our hearts or minds–does it give birth to sin. I would add that when this sinful thought is expressed in words to others, the sin is increased, as we then do harm to others, not just to ourself.
You use the example of a vow or promise. In this case, I will agree that, unless it is actually spoken, then we are not bound by such a vow to another human being and breaking of an unspoken vow cannot be sinful in the sense of hurting another of God’s creatures. Again, this is an example where your thought does come into play, but I believe it is not the whole story. An unspoken vow cannot be broken, and therefore sin is not involved, unless one makes a vow to God in one’s heart/mind. So, I think there is validity to what you are thinking but we need to get the whole picture.
I hope this helps.