Why is he making me feel guilty?

YOU SHOULD NOT FEEL GUILTY! It was his fault for not putting any time and effort into your relationship. Now that he has lost something good, he is starting to realize how much he wants it back. Here's my advice, let him feel the pain that he gave to you. You may feel guilty for watching him like this, but remember what he did to you? Tell him to back off! You need some space, some space to collect your thoughts.