Since I do not fast draw on a regular basis, I never gave the draw time any thought really. I have both leather and kydex holsters, and each is used for various duties as called for that day. I will say that over time, the leather holster will absorb sweat, moisture and subsequent dirt particles, and this will cause more wear to the gun then the Kydex. But since my range holsters are mostly Kydex, and this is where the gun is drawn from a holster the most, I cannot say which would be faster. My IWB holsters are Kydex hybrids, so that the gun can be drawn easily and returned to the holster easily. I will say that in the Fast Draw cowboy comps, they use holsters which are leather hybrids now that incorporate kydex in some of them.
I believe in using the same holster for carry or competition. I believe you want the same feel of the gun in the same spot all the time and the draw point to be in the same place all the time. When you practice do you practice for competition or every day carry. I know alot of gamers that practice for competition. Some of them fire thousands of rounds a practice session. But they carry day to day with a different style holster (usually leather). I have invited them come to the range with their carry holster and draw on a target and they cannot believe they are not as fast or accurate drawing with their carry holster. Im sure you have noticed some leather versus plastic are different in many ways as in height, weight, angle and amount of friction required to draw or holster. I feel it makes a difference, maybe Im wrong.