You're forcing one style of meditation on yourself, even if you don't like it. Focusing on the breath is not the only way to meditate. There are so many people who have written off meditation as boring or not for them because they think that there is only one way to meditate. I believe that we may be drawn to different styles of meditation depending on what we are facing in our lives. This may just not be the season where focusing on the breath is of interest to you. That is okay. The main point of meditation is learn how to focus on something beyond the chatter of your mind. You get to decide what the object of that focus should be. Here are some of the many different types of meditation:
You aren't gradually building up your endurance. If the people who have given up on meditation were advised to start doing only 1 minute and work their way up as they see fit, those people would probably still be meditating today. It is unrealistic for a beginner to think that they can go from the habit of constantly thinking nearly every moment of their entire lives to quieting the mind for 20 minutes straight without any obstacles. That is enough to make anyone want to give up. Start small. Meditation should be enjoyable. If you are pushing yourself to meditate for so long that you are feeling relieved and ready to sprint off of the cushion or chair when it's over, you need to scale back on your time and only increase when you develop a genuine desire to challenge yourself and go longer.
You're doing too many things in your daily life that make you restless. The state of our minds during meditation is directly correlated with the things we do in daily life. Maybe it's not a coincidence that every time you sit to meditate your mind is restless. Ask yourself what did you do leading up to the meditation or what did you do the day before? Were you screaming at the TV because your favorite football team lost? Were you headbanging to heavy metal music? Did you consume too much information online? Things like this plant seeds of restlessness in our minds that usually carry over into our meditation and make the session an uphill battle. In order for our meditations to be easier, we must do things in our lives outside of the session that support quieting the mind. Consider trading in your hobbies that make you restless for something like yoga, reading or tai chi.