Ttdfjt

I don't understand how predictions can trace the actual data so closely (see the code below)? Does that make sense? The model is $Y_t = theta Y_t-1 + Z_t$ where $Z_t$ is random noise. Hence the random noise term should have prevented such a close tracking. But it does not. Why? Data is here.

from statsmodels.tsa.ar_model import AR
import matplotlib.pyplot as plt 
import pandas as pd
dat = pd.read_csv("C:\Users\boris\Documents\lake.txt", sep='t')
dat.drop(dat.index[:5], inplace = True)
x=dat.iloc[:,0]
x=pd.to_numeric(x)
model = AR(x)
model_fit = model.fit(maxlag=1, trend='nc')

predictions =model_fit.predict(start=1, end=len(x)-1)
plt.figure(num=None, figsize=(8, 5), dpi=70)
plt.plot(list(x), marker = '.', color = 'red')
plt.plot(list(predictions), marker = '.', color = 'blue')
plt.show()

Edit 1 and additional comment.

If I relabel the row indices:

x=dat.iloc[:,0]
x=pd.to_numeric(x)
ind=x.index.values
ind[:]=range(0,len(x))

and then run the prediction:

model = AR(x)
model_fit = model.fit(maxlag=1, trend='nc')

predictions =model_fit.predict(start=1,end=len(x)+30)

It returns predictions with row indices starting with 1 while the original x was starting with 0 index. However, the $predictions[1]$ is almost equal to $x[0]$. This leads to an unjustified shift when plotting both x and predictions. I tried putting $start=0$ but python complained bitterly. Is this some sort of a bug or it was supposed to be like this? If yes, why?

BTW, statsmodel is the package most commonly used for autoregression? Might there be better ones?

My original answer below was applicable to general (i.e non-stationary) time series. In the OP case, if the data can be modeled as $Y_t = theta Y_t-1 + Z_t$, then it should be stationary, and therefore the variance is constant, so my statement about subsequent values $Z_t+1$, $Z_t+s$,..getting larger is not correct. The residual standard deviation $sigma$ is a good estimate for all values of $Z$.

(To add to the confusion, the data in the plot looks almost, but not quite stationary - there seems to be a slight downward trend - I'm surprised statsmodels.tsa was able to fit an $AR(1)$ without throwing an error, or at least a warning)

Keep in mind that $Y_t = theta Y_t-1 + Z_t$ where $Z_t$ represents the "true" model underlying your component. Here by true, I mean the theoretical model that you have selected to represent your time series.

The estimated model, that is the one actually calculated by your software and plotted by the software, will be $hatY_t = theta Y_t-1$.

$Z_t$ is a stochastic process (i.e. random variable) and hence cannot be estimated by a deterministic calculation, which is what your point forecasting model is.

For a model $hatY_t$ fitted on the data $[Y_0,...Y_t-1]$, a good estimate $hatZ_t$ is the standard deviation $sigma$ of the residuals $hatY-Y$. But $sigma$ is a good estimate only for the first value $Z_t$.
Subsequent values $Z_t+1$, $Z_t+s$,..will become larger and larger. Intuitively, this corresponds to the idea that the farther out into the future your forecast, the more uncertain it will be.

Sometimes, depending on the forecasting method used, you can estimate the value of $Z_t$ analytically. So you have a formula $Z_t+k+1 = f(Z_t+k)$ which you apply iteratively to get the value $Z_t+h$ at your forecast horizon $h$.

Other times, you can't calculate $Z_t$ analytically, so you have to simulate it instead. For this you need to use something like an MCMC model, or sample path simulation (here is a good example on how to do sample path simulation, it is regarding neural networks, not AR processes, and the code is in R not Python, but it is well written enough that it is still relevant to your question).

When things are too good to be true , they often aren't true ! Here is the ar(1) model and the residual plot ( always a good idea ! ) and the Actual,Fit and Forecast graph where the 1 period out forecast is heavily based upon the prior value .

It appears that the 1 period out forecast is upwards whereas I get the following downwards mean-reverting forecast which is to be expected ( play on words here ! )

EDITED AFTER OP'S COMMENT.:

The 97 estimable equations lead to an intercept of 1.467 and regression coefficient of .836411 using the last previous value.

Having used data for the 98 periods . the approach is to predict the second from the first .. the third from the second all the way out to predict the 98th value from the 97th actual.

these 97 predictions are then the fitted values and in general the one-period out forecasts. . Now the optimal parameters that were found are the "best" possible combination of intercept and lag1 regression coefficient.

To illustrate how this works consider predicting the 99th value ( 1 period out)

To continue if we wish to predict the 100th vale we pretend that the first period out prediction (9.79768) is the actual value for period 99.

Thus we then plug in the value 9.79768 and obtain a prediction for period 100 ... getting 9.662

If this clears up your issues .. upvote my answer and accept it ..

if not call me on my landline (US) as I am the office as I cannot think of anything else to write about this.

You are fitting an AR(1) model to this data, so you are postulating:

$$y_t = phi y_t-1 + varepsilon_t$$

Your data looks close to non-stationary, which means that your parameter estimate for $phi$ is probably close to 1. You can check with print(model_fit.params).

For the sake of argument, let's suppose that the estimate is exactly 1 (of course it won't be exactly, but I bet it's pretty close). Then you would have:

$$y_t = y_t-1 + varepsilon_t$$

But the error term $varepsilon_t$ is assumed to be white noise, so your forecast of it will be zero. That means your forecast will be:

$$hat y_t approx y_t-1$$

As you noted, your original data starts at index 0, but your predictions start at index 1. That's not a mistake. The model is using the index-0 datapoint to forecast the index-1 datapoint, and because of the simplicity of the model and the parameter estimate close to 1, you will get a one-step-ahead forecast of $hat y_1 approx y_0$. In the same way, $hat y_2 approx y_1$, etc.