A new preferred point geometric structure for statistical analysis, closely related to Amari's $\alpha$-geometries, is introduced. The added preferred point structure is seen to resolve the problem that divergence measures do not obey the intutively natural axioms for a distance function as commonly used in geometry. Using this tool, two key results of Amari which connect geodesics and divergence functions are developed. The embedding properties of the Kullback-Leibler divergence are considered and a strong curvature condition is produced under which it agrees with a statistically natural (squared) preferred point geodesic distance. When this condition fails the choice of divergence may be crucial. Further, Amari's Pythagorean result is shown to generalise in the preferred point context.