Catholic guilt is a thing. Many people experience it. But I guess that people of other faiths could experience religious guilt too. It seems to be popular to feel guilty among those who were brought up with the idea of "original sin".
Here's how I look at guilt in terms of religious belief. To get rid of guilt, forgive yourself. It will be easier to believe God can forgive you if you can forgive yourself. Because if you can do something, God should be able to do it too. Right? So if you can show yourself that you can forgive yourself, then it would be easy to believe that God could do the same. If not, if you can do something that God cannot do, then what does that say about God? Wouldn't it be weird if you could do something that God couldn't do?