You are losing your angles too early in the downswing.
In the image below you will see the white line is where your shaft is. The yellow circle (this is where you want your hands) and line (shaft) is where you want it to be when you reach A6, Club shaft parallel to ground on the downswing.
You want to maintain the angles in your wrist longer. Try this drill.