Talk:Akaike information criterion/Archive 2

Archive 1

Archive 2

"relative quality of statistical models"

@BetterMath: I'm not going to revert again because I'm tired of your one-sentence explanations for removing my contribution. I gave two reliable sources that state verbatim what I added to the article; here are three more sources that confirm that this definition applies to time series models as well (which are regression, too, anyways, but let's leave this discussion aside). So after I've done my due diligence, how about you do your part: (i) please show me a reliable source that contradicts either of the three five that I've found, and more importantly (ii) explain to me what "relative quality of statistical models" is supposed to mean: what "quality" are we talking about? Best looking? Fastest converging? --bender235 (talk) 14:07, 18 September 2019 (UTC)

The quality of a statistical model is explained in the second paragraph of the article: in particular, "the less information a model loses, the higher the quality of that model". The explanation is elaborated in the section "Definition".

I will look at the references you cite and get back to you on them.

BetterMath (talk) 18:03, 18 September 2019 (UTC)

This "information loss" definition is absurdly vague. What AIC weighs models by is predictive power, i.e. minimal forecasting (or out-of-sample) error. As described in all five sources I've named. --bender235 (talk) 20:19, 18 September 2019 (UTC)

A time series is different from a regression. With a regression, the log-likelihood function is additive: 𝓁(⟨ y₁, …, y_n⟩) = Σ_i 𝓁(y_i). Such additivity does not generally hold for time series, nor for most other non-regression models. Hence, citing sources that only apply to regressions is invalid.

The three more sources that you cite (Lütkepohl, Hyndman & Athanasopoulos, Chatfield) do not seem to mention the word “deviance”.

AIC is fundamentally founded on information theory. That is why it is called an information criterion. The Definition section makes the foundation clear, including citing the original paper of Akaike. The issue is discussed in much detail by Burnham & Anderson (2002)—a book about AIC that has over 47000 citations on Google Scholar.

Your claim about the Definition section being “absurdly vague” makes no sense. What parts of it are vague to you?

You obviously do not have familiarity with the relevant literature. Maybe you should spend some time reading it before doing more on this.

BetterMath (talk) 21:11, 18 September 2019 (UTC)

Thanks for the personal attack. I take it as a badge of honor, given that you have yet to cite any literature contradicting what the five sources I mentioned clearly state.

Also you seem to not understand what "regression" means. Log-likelihoods are additive when observations are independently distributed. This assumption is violated when serial correlation is present (as is the case in time series or panel data), but—putting the necessary tweaks to the loglikelihood function aside—we'd still call it a regression. However, that's not even the point, so let me move on.

AIC is a model selection criteria that "ranks" models according to their predictive (out-of-sample) accuracy. It gives an estimator of the same "measure" that comes from cross-validation, only without actually having to do the computationally burdensome CV. This fact was derived by Akaike using information theory, but that doesn't mean we have to stick to an ominously vague definition of AIC when several modern textbooks give us clear and easy-to-understand definition.

PS: Lütkepohl, Hyndman & Athanasopoulos, and Chatfield mention forecast errors or mean-square errors as criteria, which are obviously related to deviance (which you would know if you had read the article). Generally deviance is about how far off a model's prediction is from the real observation. RSS et al. are special cases of this idea. --bender235 (talk) 22:30, 18 September 2019 (UTC)

I believe that the article would benefit from a brief discussion of prediction errors. Such a discussion would need to be technically valid and properly sourced. Maybe put a draft version here on Talk and then get a consensus of editors to accept that.

BetterMath (talk) 03:27, 19 September 2019 (UTC)

Why would I waste my time on this? I've already named five reliable sources for the half-a-sentence correction I've proposed (compared to precisely zero from your side), and you just keep reverting my contributions anyways. --bender235 (talk) 21:01, 20 September 2019 (UTC)

@Bender235: I thought that I should come back to this. I’ve used statistical deviance very little in my own studies, and didn’t previously notice an error in your reference.

The reference McElreath (2015) does say that AIC is an “estimate of the average out-of-sample deviance”. McElreath, however, is wrong. Indeed, we could easily reduce the deviance to 0, by increasing the number of parameters—but that would lead to overfitting. This issue is discussed in the Definition section.

The error by McElreath points to a more general issue. In most science-related fields, it is reasonable to cite undergraduate-level references with the assumption that the reference is valid. In some fields, however, that assumption is not reasonable. Thermodynamics is one such field. Statistics is another. Many statistics references contain serious errors. So if the references are cited blindly, those errors will corrupt Wikipedia.

That is what almost happened here. You made an edit in good faith (I assume), and cited a reference that easily meets WP:RS. If your edit had not been undone, then the article would’ve contained an error. The way to avoid such errors is to ensure that the editor (in this case, you) has a deep comprehension of the topic and the reference.

Something similar happened before. You edited this article in January 2018, including a citation of Stone (1977). That edit too was based on your comprehension of the reference not being deep enough (as I explained on this Talk page at the time).

Just now, I was looking at the article Parametric model. There was a problem in the definition given there. I fixed the problem, and looked at the article history. The prior edit to the article was made by you. I checked and discovered that it was your edit that introduced the problem.

The above are three illustrations of a general issue. It is important to make textual edits to a statistical article only if the editor has a deep comprehension of the topic.

BetterMath (talk) 22:36, 5 October 2019 (UTC)

@BetterMath: The reference McElreath (2015) does say that AIC is an “estimate of the average out-of-sample deviance”. McElreath, however, is wrong. Indeed, we could easily reduce the deviance to 0, by increasing the number of parameters—but that would lead to overfitting.

You're wrong. Adding model parameters reduces in-sample deviance arbitrarily (and all the way to zero if the number of parameters is equal to the number of observation). However, McElreath talks about out-of-sample deviance. Which is correct. And adding parameters obviously does not decrease out-of-sample deviance (to the contrary). Decreasing out-of-sample deviance arbitrarily by adding parameters (or any other way) is not possible; if it were, neural networks could be fit to 100% accuracy, stock prices could be predicted perfectly, etc. (which, in case you didn't notice, hasn't happened yet). The rest of your post is not worth commenting on, because you seem not to understand the issue at hand if you confuse in- and out-of-sample statistics. Just for the record, though, let me tell you that neither McElreath nor Taddy are undergraduate textbooks. As one can read in the preface, McElreath's book is meant for first-year PhD students; Taddy's book is meant for (and used in) data science classes in MBA program at Chicago Booth. --bender235 (talk) 01:24, 6 October 2019 (UTC)

P.S.: I had all but forgotten my edit from January 2018. Indeed I added the important out-of-sample target of the AIC, which unfortunately you don't understand today more than you did two years ago. --bender235 (talk) 01:37, 6 October 2019 (UTC)

Yes, I had misunderstood on out-of-sample. There are still the prior problems that I raised though. First, mentioning the term deviance is poor practice, because it is not known to most readers. Using the term in the body of the article would be fine, but it should not be used in the opening sentence. The goal is to explain the topic in a way that readers will tend to find as easy to read as practicable. Second, the claim about out-of-sample deviance is not known to be true, in general. In particular, there are issues when the log-likelihood function is not additive. There is some relevant discussion in the cited paper of Ing & Wei.  BetterMath (talk) 20:17, 7 October 2019 (UTC)

AIC is a technical term that will inevitably require other technical terms to properly define it. That's how it works. I'm opposed to using a vague, non-nonsensical definition ("relative quality") instead of a proper one, because contrary to your beliefs the current definition poses more questions to the reader than it answers ("what is quality?"). And unlike for deviance, there is no article defining it (mostly because its a meaningless term to begin with).

Second, the out-of-sample deviance target for AIC is true. It only fails when the assumptions behind AIC aren't met, but then AIC is improperly applied to begin with. That paper you mentioned, Ing & Wei, does not mention log-likelihoods anywhere (let alone a supposed issue with non-additivity), so I'd like to see a proper citation for your claim. --bender235 (talk) 20:46, 7 October 2019 (UTC)

There are two main derivations of AIC. One requires strong assumptions: this is the original derivation by Akaike. The other is very general: this is based on the work of Takeuchi. (Some discussion of those is in the History section.)

Quality is precisely defined as K–L discrepancy from the true model, and is the basis for AIC, regardless of the derivation. That’s discussed in the Definition section. The first two paragraphs give an intuitive explanation of that, which most readers will (I think) comprehend.

Requiring most readers to click on a wikilink in order to comprehend the opening sentence is surely a poor practice.

Ing & Wei discuss some problems that arise when observations are not independent. Independence of observations is equivalent to additivity of the log-likelihood function. Additivity of the log-likelihood function is assumed by McElreath when defining deviance: see his definition of deviance (in section 6.2.4).

BetterMath (talk) 22:18, 8 October 2019 (UTC)

@BetterMath: I'm not sure that I understand what you mean by "independence of observations is equivalent to additivity of the log-likelihood function," because log-likelihood-based models are ubiquitous in time series analysis where, obviously, observations are not independent, and yet because of the likelihood underpinning, AIC is used frequently in model selection. Of the books I cited above most are specifically from time series econometrics. Lutkepohl, for instance, discusses AIC and its use for time series models used for forecasting (in which case AIC targets the forecasting error).

Are you claiming that all of these texts are wrong? That all of these authors, not to mention the ones relying on them, are using AIC wrong because they somehow forgot about the serial dependence of their time series observations? If so, that is a pretty bold statement that sure as hell needs a reliable source. So far you haven't presented any. I'm waiting. In particular, I'm looking for (i) a definition of "additivity" in this context, and (ii) why only independent observations lead to an "additive" log-likelihood.

Apart from all that, I find it absurd that you think "out-of-sample deviance" is too technical, but "quality defined as K–L discrepancy" is fine. Do you really believe that the modal reader has a more intuitive understanding of information theory and Kullback–Leibler divergence than he has of forecasting errors? --bender235 (talk) 22:19, 11 October 2019 (UTC)

Maybe this discussion might additionally consider WP:Lead? TheSeven (talk) 03:57, 12 October 2019 (UTC)

MOS:LEAD is always a concern for articles. But what in particular do you mean? --bender235 (talk) 17:40, 13 October 2019 (UTC)

McElreath defines deviance (in section 6.2.4) as −2 Σ_i log(q_i), and states “i indexes each observation (case), and each q_i is just the likelihood of case i”. The quoted statement does not make sense, because a likelihood is a function of the model parameter, not of an observation.

In prior sections (6.2.2–6.2.3), McElreath actually defines q_i as the probability of case i. Thus, in the quoted statement, McElreath seems to be using likelihood to mean probability (some people do confuse the two). My earlier writing copied his usage. Although his usage is wrong, I didn’t want to distract from the point that I was making by raising the issue.

The definition of independence is that Pr(A and B) = Pr(A) ⋅ Pr(B). Assuming the probabilities are non-zero, that equation is equivalent to log_Pr(A and B) = log_Pr(A) + log_Pr(B).

Deviance is usually defined as −2 log(Pr(x | θ)) with θ set to the maximum likelihood estimate. If the data, x, consists of independent observations, then we get the same formula as McElreath, but with q_i denoting the probability of observation i. The formula given by McElreath is wrong if the observations are not independent.

As to Lütkepohl, the part that you cite is based on FPE, which is due to Akaike. Akaike’s approach is considered by Ing & Wei, and shown to be in error.

The work of Ing & Wei seems to be seminal, and it has been discussed in many subsequent papers (see Google Scholar). I’ve looked at only a few of those papers. One example is doi:10.1017/S0266466609990107, by Ing et al.: the authors find the mean-squared prediction error for a certain class of time series. Another example is doi:10.1111/j.1467-842X.2007.00487.x. Both those examples show how difficult it is to understand predictions, when the observations are not independent.

A third example is doi:10.1016/j.jmva.2015.01.004. An extract from that is:

One of the primary obstacles to deriving the properties of out-of-sample forecasts is the potential dependence between the variable to be forecast and the sample used in model estimation. Often the dependence problem is circumvented altogether by using the so-called independent realization (IR) condition, whereby the forecast variable is assumed to be a statistically independent replicate of the process used for model estimation. Such an assumption is unnatural in many applications. For example, in autoregressive (AR) time series forecasting, future realizations of the time series are dependent on earlier data used to estimate the AR model.

A more appropriate condition in many applications is same-sample realization (SSR), whereby the variable to be forecast is generated by the same process as the data used to estimate the model. However, under sufficient restrictions on the dependence in the data, the IR assumption often delivers a sufficiently accurate approximation of SSR forecast loss. For a class of short memory processes, Ing and Wei show that the mean square forecast error (MSFE) of a least squares autoregression under IR is equivalent to the MSFE under SSR up to an o(kT⁻¹) approximation.

.........

Panel data models are increasingly being used in forecasting applications. The purpose of this paper is to investigate whether IR provides a similar shortcut for deriving the properties of panel data forecasts. Due to parameterized heterogeneity in the forecasting model (such as fixed effects), the relevant approximation of the LS MSFE under SSR is not equivalent to the MSFE under the IR assumption. This is because the LS transformation used to partial out these heterogeneous parameters generates dependence in the transformed time series comprising the panel. Because SSR is a more realistic condition in many empirical forecasting applications, the IR assumption should only be employed with caution when deriving the properties of panel data forecasting models.

That’s essentially the main point that I made in my remark of 17 January 2018. (And I had not seen the paper when I made the remark.)

As to your last paragraph, my comments referred to the opening sentence of the article. K–L discrepancy is not discussed in the opening sentence, nor should it be. The same is true for deviance.

Thanks to TheSeven for citing WP:Lead. I see it has a section about the First sentence. The section well supports my comments pertaining to the opening/first sentence.

BetterMath (talk) 16:18, 14 October 2019 (UTC)

I'm not quite sure why you are so obsessed with disproving McElreath, especially since you do not seem to understand the content of his book. The sentence you quote from page 182 is unquestionably correct, since his (contrived but nonetheless valid) example estimates one (binomial) probability parameter per event (see p. 178), in which case i indexes both the parameter vector and the observations. McElreath does not confuse likelihood and probability; his description of probability and likelihood (and AIC, for that matter) are perfectly accurate.

Second, and more important than this nitpicking of a valid source: you have yet to explain (i) why only independence leads to "additive likelihoods," and (ii) why that matters in the first place. Your toy example with two independent events makes no sense (if B is not independent of A, you have P(A and B) = P(A)P(B|A) and thus log[P(A)] + log[P(B|A)], which you could generalize ad infinitum, so where exactly is the problem?). Apart, everybody and their mother uses AIC to evaluate models of non-independent (time series) data. If you really think they are all wrong, I suggest you publish a peer-reviewed article on the issue. But to save you both effort and embarrassment, just have a look at Lutkepohl, who lists numerous examples of likelihood functions of time series models, for instance here the log-likelihood of a VAR process, which obvious is not constructed under the assumption of independence. Is Lutkepohl's equation (3.4.5) an "additive likelihood" according to whatever definition you apply or not?

Finally, we would very well be within in the guidelines of MOS:LEAD if we define and explain AIC as an estimate of out-of-sample (or forecasting) error. We gain an enormous amount of precision and intuition in exchange for just a minor increase in technicality. --bender235 (talk) 20:45, 14 October 2019 (UTC)

Your reply claims that AIC is an estimate of out-of-sample forecasting error. My prior comment explained why that claim is not known to be true when the observations are not independent. It also gave multiple references that discussed that issue. It even quoted from one of those references, including the phrase "out-of-sample forecasts". Etc. Your reply ignores most of my comment. BetterMath (talk) 20:27, 15 October 2019 (UTC)

I ignored most of your comment because you haven't yet addressed the very basic issues, like what the heck you mean by "additive likelihood," and how dependent random variables would somehow not generate such a likelihood. Now please define that term, or cite a reliable source that does.

You see, the problem it seems to me is that you don't understand even the literature that you are citing. Above you claim Ing & Wei (2005) to be a source for why AIC does not apply to time series models, when in fact the abstract of that very paper states: "We present the first theoretical verification that AIC and its variants are still asymptotically efficient for same-realization predictions," and they define "same-realization prediction" as "predicting the future of the observed time series." What else would prediction and forecasting refer to anyways? --bender235 (talk) 01:56, 16 October 2019 (UTC)

Your second paragraph asserts that I “claim Ing & Wei (2005) to be a source for why AIC does not apply to time series models”. Yet I’ve never made such a claim. (Moreover, in my work, I’ve many times used AIC when analyzing time series.)

As to your first paragraph, I suggest that we focus on one issue. The issue is this: most references that discuss the properties of predictions (from models selected via AIC) do not adequately address the difficulties that arise when the observations are not independent. My comments have given four reliable sources pertaining to this issue. Your comments haven’t adequately considered that.

BetterMath (talk) 17:41, 16 October 2019 (UTC)

There is a clear disconnect between what those sources you've listed are saying, and what you think they are saying. So let's go at it one-by-one: if observations are not independently distributed, how would a likelihood function look like in your opinion (in particular, how would it not be "additive," whatever you define this to be)? --bender235 (talk) 17:45, 16 October 2019 (UTC)

Your first sentence seems be based on ignoring the first paragraph of my last comment.

About additivity, McElreath defines deviance (in section 6.2.4) as −2 Σ_i log(q_i). The Σ denotes addition: I assume you know that. The usual definition of deviance is −2 log(Pr(x | θ)) with θ set to the maximum likelihood estimate. The usual definition is not the same as McElreath’s definition for all statistical models (i.e. in general), only for some statistical models.

If you want to go back earlier, start with my first comment, of 17 January 2018. You previously indicated that you didn’t understand that comment. Your reply of 16 October 2019, though, quoted from the paper of Ing & Wei. Since you have now looked at the paper, I suggest that you reread my first comment.

BetterMath (talk) 21:28, 17 October 2019 (UTC)

Ok, let me take you by the hand and derive it for you: in a Bernoulli distribution 0 ≤ q ≤ 1 is the parameter indicating the probability of success, i.e. the random variable x being, say, 1 rather than 0. In other words, $Pr(x=1|q)=q$. If you have multiple observations $i=1,2,\ldots ,n$ of said variable x then $Pr(x_{1}=1,x_{2}=1,\ldots ,x_{n}=1|q)=Pr(\mathbf {x} =1|q)=q^{n}$. The last step obviously resulted from multiplying the success probability n-times, and if you take log you'd get $\log \left[Pr(\mathbf {x} =1|q)\right]=n\log q$. Now assume that each observation comes from a Bernoulli distribution with its very own $q$, i.e. $Pr(x_{i}=1|q_{i})=q_{i}$. Then $Pr(\mathbf {x} =1|\theta )=q_{1}\cdot q_{2}\cdot \ldots \cdot q_{n}=\prod _{i=1}^{n}q_{i}$, where $\theta =\left[q_{1}\,q_{2}\,\ldots q_{n}\right]$ is the parameter vector. Then hopefully you'll agree that $\log \left[Pr(\mathbf {x} =1|\theta )\right]=\sum _{i=1}^{n}\log q_{i}$, where the Σ denotes addition. Do you see now (finally!) that $-2\sum _{i=1}^{n}\log q_{i}$ is in fact equal to $-2\log \left[Pr(\mathbf {x} =1|\theta )\right]$ if, as in McElreath's example, the relevant probability is Bernoulli? --bender235 (talk) 23:41, 17 October 2019 (UTC)

The derivation that your reply gives (to show equality) is correct and obvious. And, it is consistent with the second paragraph of my prior comment.

Your reply is not based on what my prior comment said. You have previously also given replies that were not based on what my comments said.

BetterMath (talk) 03:39, 19 October 2019 (UTC)

I am losing my mind here. First you claim McElreath used a "wrong definition," and after I demonstrate that he was correct, you call it "correct and obvious." Are you just trolling me? Seriously, I had enough. So far you twice claimed the sources I named contained errors, which in both cases resulted from basic misunderstanding of statistical concepts (like "in-sample" vs. "out-of-sample" statistics). At this point you have lost any credibility in terms of expertise on this subject. I had enough of your ownership behavior, and I'm tired of feeding the troll. I will restore my contribution to the lead, because it is factually correct, well sourced, and intuitively easy to understand. --bender235 (talk) 18:33, 19 October 2019 (UTC)

I do not know how to deal with someone who exhibits your level of reasoning capabilities. Your choice is to select a method of Dispute Resolution or have me report you. BetterMath (talk) 18:47, 19 October 2019 (UTC)

Filed. This needs a third opinion. --bender235 (talk) 20:02, 19 October 2019 (UTC)

External editor here I didn't read all arguments, but I think that the lede needs some rewriting (not necessarily limiting to what has been done/attempted). Per MOS:FIRST, the first sentence should be accessible to the nonspecialist reader (not necessarily lay man), and in plain english and standalone (most infos about the topic should be grasped by reading the lede). However, it should also be a concise summary of the most pertinent characteristics of this concept, so accuracy is important to take into account. My remarks: 1. the lede seems too broad IMO, as it mostly focuses on what its usage is (model selection), rather than what it actually is and how it works (for instance, I wouldn't be able to say what differentiates AIC from other model selection criteria), maybe some inspiration could be drawn from F1 score's lede? ; 2. any new info should be first implemented in a section, rather than in the lede, as the lede is simply a summary of the entry's content, so I feel this discussion should first be redirected into reworking the Definition section (and maybe others), before adding these infos in the lede (notably the "out-of-sample" info which is not at all present in the entry currently). That's all, I may try to read the whole arguments if I have enough time and maybe provide an additional comments --Signimu (talk) 18:47, 20 October 2019 (UTC)

I much appreciate this. About the first sentence, I am still thinking about a constructive way forward. About your point #2, I strongly agree. About your point #1....

Regarding what AIC actually is, the second paragraph discusses the concept of information and then says “AIC estimates the relative amount of information lost by a given model”; so that, I thought, is a reasonable explanation of what AIC is. (Perhaps it would be better as “AIC is an estimator of the relative amount of ...”?) Would you elaborate on your reasons for finding the second paragraph inadequate?

AIC is used for model selection, and that’s about it. Perhaps, though, the lead does focus on that too much, as your comment says. What would you think of deleting the third paragraph (or merging the third paragraph into the Definition section)?

BetterMath (talk) 16:58, 21 October 2019 (UTC)

@BetterMath: could you finally please give us a single concrete example of a log-likelihood that is not additive, or equivalently a likelihood that is not multiplicative, so as to rewrite the law of probability that states P(A ∩ B) = P(A)P(B|A). I'm tired of this evasion. --bender235 (talk) 18:54, 21 October 2019 (UTC)

About my 1st suggestion, by reading the entry, I understand AIC is a way select the model that minimizes the error (like any model selection criteria), but here using information error to be more precise. Is that all there is to it? It seems very broad to me, as there are lots and lots of ways to measure information. Is AIC tied to a specific information metric, or is covering all instances? Eg, if I used KLD, or MI, or NMI as the metric to measure the model's error, is this in all cases an AIC? I however agree that the Lede needs to be accessible, but it needs also to be accurate. Also it does not necessarily need to target lay people IMO, but the audience that is the most likely to end up on this page, so most likely people that are interested into statistics or machine learning. bender235 please take a moment to read WP:CALC and [1]. Using a mathematical demonstration can be useful in the talk page to select appropriate sources and information to include in the article, but a reliable source is still needed, and the source needs to describe your end point. If for example a source provides only the premisses and then the conclusion is one only you derive, this cannot be included in the article (but it can be used in the talk page to argue about which source to choose to include on the article). That said, I did not read everything on this talk page and I don't have the competence to check the mathematical demonstrations, but the requirement to provide a reliable source for any content on WP's mainspace is mandatory, just make sure to comply with that --Signimu (talk) 21:48, 21 October 2019 (UTC)

BetterMath, about your suggestion, yes I think the 2nd and 3rd paragraphs could be merged as they are a bit redundant (eg, no need to write that "AIC is founded on information theory", simply link "information" to "information theory" and that's it). Here's my suggestion (feel free to edit, particularly if I introduced an inaccuracy ):

When a statistical model is used to represent the process that generated the data, the representation will almost never be exact, as some information will be lost when using the model to represent the process. AIC estimates the relative amount of information lost by a given model: the less information a model loses, the higher the quality of that model, which allows AIC to deal with the trade-off between the model's goodness of fit and simplicity. In other words, AIC deals with both the risk of overfitting and the risk of underfitting.

That said, I think the fact that the lede has no references attached anywhere is a good indicator that it needs a rewriting. In a similar situation, I would first try to rework the article's body and solve any dispute in the most pertinent section (here I'd suggest in definition) and then reuse that in the lede. There is BTW nothing that prevents the Definition section to be entirely mathematical, in fact if the lede contains a "layman" or at least easy to access description of AIC, it is often advised to be also somewhere in the article's content, per MOS:LEADREL. Just my 2 cents, hope this can help --Signimu (talk) 22:04, 21 October 2019 (UTC)

@Signimu:, you should've indeed read the above discussion before suggesting WP:CALC. I presented two RS for the definition of AIC; two sources that BetterMath claims to be wrong, because they supposedly assume "additive log-likelihood," a concept BetterMath has yet to define (let alone name a source for). I've been asking him to define this concept for weeks now; at this point, I'd be satisfied with just a simple example of a "non-additive log-likelihood". Just so we finally understand where BetterMath's remaining misconceptions are (again, Signimu, if you read the above discussion, you'll see there are plenty). --bender235 (talk) 22:53, 21 October 2019 (UTC)

Yes, it was only a potentially useful reminder, but I did not mean that you or BetterMath broke it ;-) (and the 2nd link provides a very interesting discussion on when original research is permitted and even useful, and I think your discussion here falls in this category). I have read a bit more the rest of the discussion, that's very interesting, and I think I isolated the 2 main roots of this disagreement:

1. Accuracy vs accessibility of the lede: BetterMath argues that the lede should be simple enough to be accessible to any reader, whereas Bender235 argues that it should be accurate. MOS:FIRST reads: "The first sentence should tell the nonspecialist reader what, or who, the subject is. It should be in plain English" and "If its subject is definable, then the first sentence should give a concise definition: where possible, one that puts the article in context for the nonspecialist. Similarly, if the title is a specialised term, provide the context as early as possible". Emphasis is mine. As I read it, the lede should indeed be accessible, but not necessarily to the lay person, we can hence assume some reasonable degree of familiarity with the necessary foundational concepts, with no need to dwelve too much in off-topic details. For other concepts that may be necessary but there is doubt whether the reader would be familliar with, wikilinks can be used (eg, deviance). The lede should however provide the context, so a mention of AIC being a model selection criterion is necessary, as it is now seems good. In conclusion, I would suggest to drop this constraint, don't limit yourselves on writing for lay people, aim for an accurate description written in simple English for a reader that assumedly has some statistical litteracy.
2. On the definition of AIC in terms of forecasting error: there is a disagreement between both authors on whether the sources that define AIC as a forecasting error measure (to oversimplify) are correct or not. This discussion is interesting, edging on WP:OR but is permitted on talk pages per consensus[2] if it allows to select what sources are valid and hence usable for the encyclopedia. I don't have the competence to help on this, so I would advise 2 things, that were already advised: 1. first try to work and reach a consensus for other sections such as Definition, and later work on the lede depending on what result you get (and reuse the same refs), 2. contact the WikiProject Statistics for a 3rd opinion, the DRN already made a request[3], hopefully someone will show up... --Signimu (talk) 01:09, 22 October 2019 (UTC)

As a first concrete step, I would suggest that both of you lay down a simple list of a few reliable sources, along with quotes (not too long) from each of them where they support the point you are arguing, with succinct or no comment on your part. This would allow to give a clear starting/reset ground to discuss further, for you and for potential third parties. --Signimu (talk) 01:16, 22 October 2019 (UTC)

Ok I've read everything. It's less complicated than I thought. I would suggest you both lay down a clear list of the reliable sources that support your point, as a way to recap, and we'll see from there Thank you for your patience, we'll make this work! --Signimu (talk) 01:45, 22 October 2019 (UTC)

@Signimu: I agree with MOS:LEAD 100%. So, as somebody without a deep knowledge in statistics, you tell me if "prediction error" is a concept that is too technical for the lead. --bender235 (talk) 02:53, 22 October 2019 (UTC)

Bender235 Indeed no I don't think this requires deep knowledge in statistics, and I'm not a statistician, although I have some statistical literacy. But anyway there's a more foolproof way to reach a consensus on this point, by looking at other consensus: Hidden_Markov_model, Nearest-neighbor_chain_algorithm and Theil–Sen_estimator are all good articles in the Statistics WikiProjects, and they all contain a fairly accurate lede that is not accessible to the lay man, but are accessible to statistical literate individuals. This further confirms that this point is moot. What remains is what accurate definition/description to use, and for that the reliable sources prime --Signimu (talk) 15:30, 22 October 2019 (UTC)

Agreed. Here's what the sources I named write: "AIC provides a surprisingly simple estimate of the average out-of-sample deviance." (McElreath) and "The AIC is an estimate for OOS deviance." (Taddy). Deviance is a more general term than prediction error, but the idea is the same. --bender235 (talk) 15:36, 22 October 2019 (UTC)

Thank you for providing these sources, I quickly read the context of the quotes you provided, I think they would be pertinent additions to the definition and lede. And they answer my initial questions, of what is AIC and how it differs from other information criteria. Here are what caught my eyes: From McElreath:

AIC provides an approximation of predictive accuracy, as measured by out-of-sample deviance. All information criteria aim at the same target, but are derived under more and less general assumptions. AIC is just the oldest and most restrictive. AIC is an approximation that is reliable only when...

From Taddy:

The AIC is an estimate for OOS deviance. It is targeting the same statistic that you are estimating in a CV experiment: what your deviance would be on another independent sample of size n. You know that the IS deviance is too small—since the model is tuned to this data, the IS errors are an underestimate of the OOS errors. Some more deep theory7 shows that IS minus OOS deviance will be approximately equal to 2df, and this is the basis for Akaike’s AIC. [...] Basically, while AIC and AICc are trying to optimize prediction, the BIC is attempting to get at a “true” model. This leads the BIC to be more conservative, and in small to medium-sized samples it behaves much like the CV-1se rule. However, in large samples we find that it tends to underfit for prediction purposes—it chooses λ that are too big and models that are too simple.

In Taddy there is also a paragraph about the intuition behind the "corrected AIC" which I think is interesting and would be pertinent to add to extend the AICc section.

BetterMath, could you please provide reliable sources and concise quotes for your point too? Thank you in advance --Signimu (talk) 17:37, 22 October 2019 (UTC)

For a summary, kindly see "Summary of dispute by BetterMath" on the Dispute resolution noticeboard. The main issue is that predictions are difficult to understand when the observations are not independent.

McElreath assumes that the observations are statistically independent; so it is not relevant here. I have not looked at Taddy, but assume that it is the same.

My above comments noted four reliable sources: Ing & Wei (doi:10.1214/009053605000000525); doi:10.1017/S0266466609990107; doi:10.1111/j.1467-842X.2007.00487.x; doi:10.1016/j.jmva.2015.01.004. For the last source, a relevant extract is in my above comment of 14 October.

BetterMath (talk) 21:07, 22 October 2019 (UTC)

Thank you BetterMath I will try to summarize each source you provided, please correct me if I misrepresent them or the point/quote you would extract from them:

• [4]: studies AIC under same-realization predictions (in other words: original AIC assumed independent observations, here they assume dependent observations). Quote: "This study shows that AIC also yields a satisfactory same-realization prediction in finite samples. On the other hand, a limitation of AIC in same-realization settings is pointed out. It is interesting to note that this limitation of AIC does not exist for corresponding independent cases."
• [5] extends the previous, but from assumed stationarity to non-stationary same-realization predictions. Quote: "[...] which shows that the asymptotic efficiency (see (32) in Section 3 of the present paper) of AIC and a two-stage information criterion of Ing (2007) in various stationary time series models carries over to nonstationary cases."
• [6] proposes another information-based criterion, FIC. Quote: "We illustrated, by means of simulations, that the FIC selects models that give predictions with a comparable MSE to that of the AIC over the entire parameter space."
• [7] I could not find a direct link with AIC, maybe you can clarify BetterMath?

My comment: given the sources you cite BetterMath, I in fact see no conflict with what bender235 wrote? The sources seem to conclude that AIC and its interpretation as an estimate of predictive error on the test set can be extended to same-realization predictions and to non-stationary cases. I note however that the first source explicits a caveat:

"On the other hand, a limitation of AIC in same-realization settings is demonstrated. Empirical results, given in Table 2 in Section 4, reveal that it seems very difficult for AIC to possess strong asymptotic efficiency; [...] If the order of the true model is finite, then the BIC-like criterion, for example, BIC [24] and HQ [13], can choose the smallest true model with probability tending to 1, but AIC does not possess this optimal property (see [26]). Therefore, to achieve optimal same-realization predictions in situations where the underlying AR model has a possibly finite order, further investigation is still required."

This is indeed very interesting and I think pertinent to the entry, but as I understand it, this states that AIC lacks "strong asymptotic efficiency", but there is still some asymptotic efficiency, although less than BIC-like criteria. As the 2nd source writes:

" Under a less stringent assumption on k than that of Gerencser (1992), Ing and Wei (2003, 2005) obtained an asymptotic expression for the MSPE of the least squares predictor and showed that AIC and its variants are still asymptotically efficient for same-realization predictions."

It looks to me this goes along the way of AIC being defined in terms of prediction error, or more precisely out-of-sample deviance, or rather these sources do not contradict, but they provide caveats on when AIC may not work as efficiently as expected, which is for sure of interest. If I am misunderstanding your point BetterMath, please correct me, but if that is correct, I would suggest that in fact all of your sources, Bender235 and BetterMath, be used to expand the entry and then the lede I would suggest to start with McElreath and Taddy as the intro/textbook definition, and then the other sources can be used to inform about AIC extensions and caveats under dependency and non-stationarity What you do you guys think? --Signimu (talk) 23:24, 22 October 2019 (UTC)

@Signimu: did you notice how BetterMath evaded my question yet again? How often am I supposed to ask it? --bender235 (talk) 00:40, 23 October 2019 (UTC)

@BetterMath: could you finally please give us a single concrete example of a log-likelihood that is not additive, or equivalently a likelihood that is not multiplicative, so as to rewrite the law of probability that states P(A ∩ B) = P(A)P(B|A). I'm tired of this evasion. --bender235 (talk) 00:40, 23 October 2019 (UTC)

I understand your impatience Bender235, but please let's not side track, I promise I'll do everything I can to help solve this as quick as possible . This question might be of interest, but I think for the moment we may be able to solve this issue without, by relying on reliable sources, as WP:V requires And so far, what I see is that all sources provided are of interest and pertinent, and not contradictory but rather complementary I'd like to hear what BetterMath thinks of the descriptions I wrote from his/her sources and of my proposition of a plan to rewrite the article with all the sources provided by both of you --Signimu (talk) 00:58, 23 October 2019 (UTC)

@Signimu: The general situation seems to be as follows....  It used to be believed that AIC had a nice property: AIC selects the model that has the minimal expected error when making predictions. Then, in 2005, Ing & Wei showed that the analysis underlying that belief was invalid when the observations are not independent.

That leads to a Question: when the observations are not independent, what can be said about the expected error from a model selected via AIC?

Ing & Wei gave a partial answer to the Question. They introduced a technique that they called independent realization. They then applied the technique to one special case: a certain class of autoregressive models. For that special case, they showed that AIC essentially still had the above-noted nice property.

Ing & Wei did not, however, attempt to answer the Question for arbitrary models. About classes of models other than the special case, Ing & Wei said essentially nothing—except that prior analyses were invalid.

Many researchers have worked on extending the analysis of Ing & Wei, in attempts to answer the Question more generally. As an example, the second source that my comment cited extends the work of Ing & Wei to autoregressive models that are nonstationary (as your comment rightly summarized). As another example, the fourth source that my comment cited argues that the independent-realization technique is generally inappropriate for linear models of panel data.

For a list of the substantial literature that Ing & Wei has generated, see Google Scholar. As the list demonstrates, and the above examples illustrate, there has been a substantial amount of work, but only little progress.

To summarize, researchers are not even close to answering the Question for general models, despite substantial effort. The special case of autoregressive models has been analyzed, but there seems to have been nothing done for most other important classes of models (e.g. nonlinear, threshold, etc.). Thus, when the observations are not independent, prediction errors are generally poorly understood.

BetterMath (talk) 18:08, 23 October 2019 (UTC)

I'm gonna ask this one time and one time only: in situations where observations "are not independent of the previous data" (Ing & Wei), what does AIC try to measure/estimate, regardless of whether it does so correctly or wrongly? --bender235 (talk) 19:05, 23 October 2019 (UTC)

For a given candidate model, AIC provides a (relative) estimate of the K–L divergence from the true model. This is discussed in the Definition section.  BetterMath (talk) 20:02, 23 October 2019 (UTC)

And how in the world is that different from deviance? --bender235 (talk) 20:55, 23 October 2019 (UTC)

Today, after a few weeks of hiatus, Bender235 proceeded to update the lede, and BetterMath subsequently reverted. In the absence of BetterMath's replies, Bender235 was right in being BOLD. BetterMath, I appreciate your clarification, but I remain unconvinced that a model's limitation is a sufficient reason to not describe the model's definition or theoretical goal. It seems to me it would be like writing that Naive Bayes can't estimate the maximum likelihood because the model can't cope with dependent observations. In both cases, there are limiting assumptions, but that doesn't change the theoretical goal/measure target. IMO the most reasonable thing to do in such case would be to 1) describe the intended model's goal, per the sources provided by Bender235, 2) add a sentence about caveats, properly backed by sources. As the lede is right now, it's indifferentiable from the description of any other model selection criterion ("relative quality" is too vague). --Signimu (talk) 04:15, 9 November 2019 (UTC)

The theoretical goal, per se, is not to get the prediction with the least error. The theoretical goal is the goal given in the Definition section. BetterMath (talk) 07:57, 9 November 2019 (UTC)

BetterMath what is in the definition section does not preclude a more succinct description in the lede such as those provided in the sources of Bender235. You only restate here your own previous arguments, that's not going to convince anyone further (WP:BRDD)! For the moment, Bender235 provided direct quotes from reliable sources supporting the edit, where you only provide your own statement, as from what I could read from your sources, they do not contradict Bender235 sources' definition. In other words, the burden of proof is on you for the moment. Thus, I find it quite distateful for you to revert based only on your own conviction. You may be sure you are right, but Wikipedia is not interested in truth, but in verifiable content, so you need to provide a source with a direct quote for your statement if you would like to keep your position further. Another solution is to compromise, as I proposed several times above. Unfortunately, it seems from previous similar discussions you participated in that you never compromise nor consider the other editors' arguments[8][9][10][11], so unfortunately I am led to believe that it seems you misunderstand the purpose of WP:BRD, as it is not a way for you to get your way, but to reach a consensus, even if it means it's an imperfect one. TL;DR bottom line: please provide a direct quote from a source supporting your refusal of Bender235 addition to the definition, or if you can't please let the addition be done. --Signimu (talk) 13:22, 9 November 2019 (UTC)

An edit war is developing... BetterMath, please do not revert until you provide sources with direct quotes for your argument, and that we reach a consensus on this. Reverting a sourced content will yield no good... --Signimu (talk) 13:55, 9 November 2019 (UTC)

There should be no compromise on technical validity.

As for quotes, consider the quote in the comment by Bender235 at 01:56, 16 October, which is from the paper of Ing & Wei. That quote supports the position of Bender235. Then see the extract (i.e. quote) in my comment of 14 October. The extract is explicitly explains that quote from Bender235 is not for panel data.

BetterMath (talk) 13:57, 9 November 2019 (UTC)

BetterMath Thank you for taking the time to reply (even though you reverted again...). I think you are here talking about doi:10.1016/j.jmva.2015.01.004. First off, this paper does not mention Akaike information criterion, so applying it here is a bit of a WP:SYNTH. But I am of the opinion that logically it should applies and it is a pertinent addition (but don't be surprised if other editors disagree). Secondly, the source mentions specifically that it is studying out of sample forecasts. So if AIC is not an OOS forecast, how could this paper apply to it? So either Bender's source is correct, and thus your source is applicable (with a bit of a synthesis stretch), or Bender's source is wrong (AIC is not an OOS deviance estimator) and your source is not applicable either, so it looks paradoxical to use the source you propose to reject Bender's addition. If your argument is correct, surely there must be another source saying the same thing (and we should be wary when only a single source declares something), can you propose another source, if possible more directly supporting your claim BetterMath? --Signimu (talk) 15:36, 9 November 2019 (UTC)

In the end, I think all of this comes down to BetterMath's misunderstanding of the subject at hand. And I'm not talking about the "in-sample vs. out-of-sample" gaffe, but his failure to produce a definition of a "non-additive log-likelihood," which seemingly is the key to produce a counterexample to AIC not estimating out-of-sample error. A likelihood function is, in the end, just a joint probability (interpreted as a function of the parameters), i.e. $P\left(\bigcap _{i=1}^{n}x_{i}\right)=P(x_{1})P(x_{2})\dots P(x_{n})$ from the multiplicative law of probability (here an intro level text). Of course this product of densities turns into a sum once taking logarithm. So if that's the "additivity" BetterMath is referring to, it is always satisfied. Unfortunately I don't know if that's what BetterMath actually means, because so far (for the past two months!) he has failed to produce a definition. --bender235 (talk) 19:08, 9 November 2019 (UTC)

Thank you Bender235 for the explanation, but so far it seems going down the route of mathematical demonstrations is likely not going to help reach a consensus seeing how it failed in the past, so I would humbly suggest this discussion simply sticks to reliable sources per WP:V. If a content can be sourced, it should be added, if it is not, then it should not shape the article, whether it's true or not. --Signimu (talk) 19:18, 9 November 2019 (UTC)

Signimu, I definitely agree with you on the application of our WP:V principle. It's just that I do agree with BetterMath insofar as that textbooks sometimes are wrong (for example, here in constrained optimization). However, such a claim either needs a WP:RS or a valid mathematical argument. So far I have seen neither from BetterMath. --bender235 (talk) 22:31, 9 November 2019 (UTC)

AIC for nested models?

The article states that AIC is applicable for nested and non-nested models, with a reference to Anderson (2008). However, looking up the source, there's no explicit indication that the AIC should be used for nested models. Instead, the indicated reference just states that the AIC can be valuable for non-nested models. Are there other sources that might be more explicit? — Preceding unsigned comment added by Redsilk09 (talk • contribs) 10:11, 18 July 2012 (UTC)

I agree with the above comment. I've tried using the AIC for nested models as specified by the article, and the results were nonsensical. — Preceding unsigned comment added by 152.160.76.249 (talk) 20:14, 1 August 2012 (UTC)

I agree, and provide a counter example https://stats.stackexchange.com/q/369850/99274CarlWesolowski (talk) 06:30, 31 March 2020 (UTC)

Justification/derivation

is AIC *derived* from anything or is it just a hack? BIC is at least derivable from some postulate. WHy would you ever use AIC over BIC or, better, cross validation?

There is a link on the page ([12]) which shows a proof that AIC can be derived from the same postulate as BIC and vice versa. Cross validation is good but computationally expensive compared to A/BIC - a problem for large scale optimisations. The actual discussion over BIC/AIC as a weapon of choice seems to be long, immensely technical/theoretical and not a little boring 128.240.229.7 12:37, 28 February 2007 (UTC)

Does the definition of AIC make sense with respect to dimension? That is...why would the log of the likelihood function have the same dimension as the number of parameters, so that subtracting them would make sense? Cazort 20:00, 14 November 2007 (UTC)

AIC is not a "hack" but it is not a general method either. It relates to Shannon entropy, which is self-information, and as such it can only compare various uses of single data objects. Physically, entropy has units of energy divided by temperature. Temperature relates to the relative information content of two different data objects, which latter, relative information content, is the basis for comparison between data objects. The reference in this Wikipedia article to this effect is obtuse. That AIC is related to "information theory" is vague enough to qualify as what physicists call "hand-waving." Namely, when one runs out of logical explanation the "waving of hands in the air" takes over. The relationship is only to Shannon entropy, which in turn may have some historical relevance, but is only a small part of information theory. Thus, it is like saying AIC is related to "statistics." It is too vague a statement to be of any use, and it does not explain anything. CarlWesolowski (talk) 21:40, 4 October 2016 (UTC)

An external link http://www4.ncsu.edu/~shu3/Presentation/AIC.pdf mentioned in the current article gives a coherent explanation of the goal of the AIC and the "corrected" AIC. (See the section "Model Selection Criterion" on page 7) Using general concepts, it defines a quantity that we seek to maximize. The AIC and the "corrected" AIC formulae are presumably ways to compute something proportional to that quantity in special situations. Tashiro~enwiki (talk) 17:38, 9 December 2017 (UTC)

It took me years to understand what the text of this article is actually about, pity the poor reader. One major source of confusion is the omission of a crisp definition of what are the 'statistical models' that AIC applies to, to wit, it applies only to 1D random variates, which is a small subset of the uses of density models. For example, if we want to fit a concentration of drug in blood curve with a 'statistical model,' like a gamma distribution, GD(t|a,b), we would minimize the error of C(t) ≈ AUC GD(t|a,b), where C(t) are concentration 2D "blood samples" in time, and AUC, in this case, is area from time is {0,infinity} of concentration under the AUC GD(t|a,b) curve, as the AUC of GD(t|a,b) = 1. Note that AIC values can be obtained from concentration curves, but only indirectly. For example, maximum likelihood fitting can be performed, but one would apply maximum likelihood to a 1D list of residuals from {C(t),AUC GD(t|a,b)} and not to the 2D concentration data itself. One could call this process 'residual maximum likelihood.' Now the way this article presents AIC and 'statistical models' excludes discussion of all other density functions, which, like concentration density functions, are not probability density functions (PDF), and that is the majority of models out there. Using the definitions implied by, but not explained in this article, the gamma distribution usage above would not be called a statistical model, which is not a good idea; too confusing, especially given that its residuals can be 'statistical.' To be clear, a gamma distribution is a density function, and although density functions can be applied to random variates, in which case, and as far as I can tell only in that case, can we speak of density functions as being PDF. Models are never 'statistical' or 'not statistical' only data can be statistical in the sense of being random, and one should not consider a PDF, or a PMF for that matter, as being anything other than deterministic; they just are not random; they have no noise. Thus, I highly recommend not saying 'statistical models,' when you mean models applied to statistical data. That is a grammatical error called a misplaced modifier, for other examples see https://www.scribbr.com/language-rules/misplaced-modifier/. There is no such literal thing as a 'statistical model' as that says that the model is randomized. Statistical modelling is the modelling of statistics, it does not use 'statistical models,' but rather models of statistical processes. CarlWesolowski (talk) 06:38, 24 July 2022 (UTC)

Information criteria are hardly a statistical paradigm

There is a misleading statement in the section on "Foundations of Statistics," where one editor claims that the Akaike Information Criterion (AIC) can form a foundation of statistics that is distinct from both frequentism and Bayesianism. Upon reviewing the cited reference (Burnham & Anderson, 2002 [p. 99]), it appears that the statement is not adequately supported. The relevant statement in the reference mentions that information criteria can be computed and interpreted without subjective judgment or the use of significance levels or Bayesian priors, but it does not suggest that information criteria constitute a separate statistical paradigm.

Additionally, after examining the book "Philosophy of Statistics," it is evident that no claim is made regarding AIC or information criteria forming a distinct paradigm within statistics. In the chapter specifically discussing AIC, BIC, and model selection, the authors treat AIC as a model selection rule that aids in statistical inference.

Based on these observations, it seems that the editor who wrote that particular part of the section may have aggregated and overextended the statements in the references. I recommend that the section be removed entirely, but gathering input from other editors would be valuable before making a final decision. Jourdy345 (talk) 15:04, 10 May 2023 (UTC)