I have been training with the Southern California Aquatics (SCAQ) on and off for a number of years now and during this time, I have noticed clear differences in the level of performance that the other guys are achieving over me. There are 4 lanes of increasing ability and I have always been stuck in lane 4, the slowest.
I have always put this performance difference down to a combination of better technique and swimming fitness over me. The fitness is easy to address. I just need to attend training more often and for longer periods in the year. The technique question has always bothered me though. I can see them pulling out body lengths on me each stroke and I want to know what I am doing wrong.
As I try to keep up with them, the first thing I notice is that I cannot sustain my breathing. I am straight away out of my comfort zone and I know that I need to back off otherwise I will have no chance of completing the sets we are doing for the day.

I have always been holding my breathe during my stroke to some extent. In the worst case, I have been holding my breathe throughout the stroke and when I turn to breathe, I have been exhaling and inhaling in the short time my head is above the water. This results in short, shallow and rushed breathing, but even worse, as I have just found out, I have been letting Carbon Dioxide build up in my lungs during my stroke. This is apparently what gives you the feeling of needing to breathe more so than the lack of oxygen.

I tried this the last time I went swimming, before I started my warmup. I was surprised how much I had to exhale to achieve this. I was also surprised how easy it was to breathe back in afterwards. This is the bit I have always struggled with and it seemed to come very naturally after my initial practice.
As a side affect, I also found that not holding my breathe totally relaxes my body and helps me improve my swimming form dramatically. I think I have found the key to my technique issues so I now need to exploit this and hopefully solve the other issues with my stroke without being held back anymore. I am already swimming in lane 3 at SCAQ now and am keeping pace. I still have a long way to go, but this is a huge improvement for me.