We develop a stochastic analysis for a Gaussian process $X$ with singular covariance by an intrinsic procedure focusing on several examples such as covariance measure structure processes, bifractional Brownian motion, processes with stationary increments. We introduce some new spaces associated with the self-reproducing kernel space and we define the Paley-Wiener integral of first and second order even when $X$ is only a square integrable process continuous in $L^2$. If $X$ has stationary increments, we provide necessary and sufficient conditions so that its paths belong to the self-reproducing kernel space. We develop Skorohod calculus and its relation with symmetric-Stratonovich type integrals and two types of Itô's formula. One of Skorohod type, which works under very general (even very singular) conditions for the covariance; the second one of symmetric-Stratonovich type, which works, when the covariance is at least as regular as the one of a fractional Brownian motion of Hurst index equal to $H = \frac{1}{4}$.