There are a number of so-called VNA calibration kits sold on eBay. Some claim higher accuracy than those from Agilent! So many people think that these will do, or they can make their own easily from an open circuit and a short circuit. Unfortunately this is not so, as we will see.
As described in our document about what is the purpose of a VNA calibration kit, the combination of the VNA and connecting cables has a number of errors, even if the VNA itself was perfect. The VNA hardware, complete with all its errors, measures the device under test (DUT). The measured data will be wrong, but this does not matter. The reason is does not matter is that the VNA then mathematically corrects for those errors using a set of error-terms. If those error terms are wrong, so the results will be wrong, but if the error terms are correct, the VNA will read very accurately.
So it is important the error terms are as accurate as possible. The purpose of the calibration kit is to allow those error terms to be be computed accurately. The error terms are computed by measuring 3 standards, whose properties must to be known as accurately as possible. It is pointless trying to correct for errors by measuring standards if you don't know the properties of the standards!
All Kirkby Microwave coaxial vector network analyzer calibration kits come with a 50Ω load. The return loss of the 50 ohm load has been checked on a VNA using a calibration kit with both fixed and sliding loads, to give the greatest accuracy in measurements. (Waveguide loads are not 50 ohm, but have an impedance which is a function of frequency.)
For a simple analogy, consider the following example, where a single error term needs to be defined. It is just one scalar measurement - not multiple vector error terms of a VNA.
An engineer wants a meter to read battery voltage in the range 0-5 V. He finds a moving coil meter marked 0-50 (PSI)
The engineer looks on the back, sees the meter has only electrical connections, so an educated guess suggests that it is probably a moving coil ammeter, and could be made to read battery voltage if a resistor was put in series. The engineer passes the meter to a new technician, outlines how a resistor can be used, to see how well the technician does this.
The technician adds a variable resistor, and to calibrate the meter decides to use a 1.5 V Duracell battery he finds lying around in a cupboard.
The technician adjusts the variable resistor until the meter reads 1.5 V. The incompetent technician has no idea if the battery is new or used, so has no idea what the voltage is. So adjusting the meter to read 1.5 V may give an result drastically different from the actual voltage marked on the battery.
The uncertainty of measurements made with a VNA will be no better than the uncertainty in the standards used to calibrate the VNA.