@tech@zwo#47645 Guiding really needs some technique
You don't seem to understand.
It is not a case of knowing how to guide. The ZWO OAG is simply problematical when used with a larger sensor like the IMX174, or even an IMX290 sized sensor. The aperture is simply too small for the sensor, and that leads to aperture vignetting, and causes the shape of stars to be even more distorted than the usual optical aberrations at the edge of the OTA's image circle.
Even for smaller sensors, as Walter has shown, the aperture of the prism stem already needs to be enlarged to avoid guiding problems.
The mirror size alone (at 8mm x 8mm) will pass no more than 70% of the light to the IMX174 sensor, not to mention additional aperture vignetting. Aperture vignetting is a Fourier domain problem; a non symmetrical vignetting problem will cause irregular diffraction pattern and spikes, etc -- it is not simply the case of the reduction of the illumination of the image circle, but also additional distortion of the star shapes.
Check out this thread on what a simple bilateral aperture vignetting (symmetrical on only one axis) does. The kind of vignetting that the original poster saw is completely unsymmetrical and will cause much worse problems (as Walter mentions, effects such banana star shapes):
Notice that distorted stars will spread the star's flux over a large area. That alone will reduce the signal to noise ratio of a star, something that is crucial in finding centroids of stars for multi-star guiding, assuming that the centroid finding algorithm can even find a stable centroid of a star that is banana shaped and stretched over 20 pixels.
When a star is not radially symmetric, moving it by sub pixels will cause the true centroid to move irregularly when the guide scale is not over sampled (and if you over sample, you also loose signal to noise ratio -- so there is no win there).
@glancy3's latest seems to point in the direction of PHD2 (which is a reliable gold standard) of just such a problem. He obviously could not calibrate because PHD2 could not get a reliable fix on the centroid.
@glancy3: you might be able to reduce the problem by trying to get the image circle to at least fall into the center of your guide plate instead of that oval in the corner. It might involve some filing to manipulate that stem around. Also, try to insert the prism stem closer to the optical axis of your OTA; that will do two things, it gets you more illumination, and it reduces the effect of aperture vignetting. But there is a fine line there: if you stick the prism in too far, you will be creating a shadow in your main OTA (removable by flats, not of the time). Try to stick the prism in as far as possible, but no farther (some quote from the Da Vinci Code :-).
A flattener might help with the illumination, but will not help with the vignetting caused by the OAG's structure. Unfortunately, a reducer will usually make matters worse at the edges of the illumination (unless it is designed as a reducer/flattener).
The Celestron OAG that Walter mentioned has a 12mm x 12mm prism, and is large enough for telescopes with large f-numbers.
@tech@zwo: do the modification that Walter suggested (or 3D print a stem with an enlarged aperture), give it to your algorithms person to test, and you will find that guiding will suddenly improve by leaps and bounds, even with a small ASI290MM sensor.