Logistic regression - Revision history

DeployBot: [deploy-bot] Deploy from CI (e406999)

2026-04-27T23:37:06Z

[deploy-bot] Deploy from CI (e406999)

DeployBot: Marked this version for translation

2026-04-27T23:32:43Z

Marked this version for translation

← Older revision		Revision as of 23:32, 27 April 2026
Line 14:		Line 14:
	'''Logistic regression''' is a foundational statistical model for binary classification that predicts the probability of a categorical outcome by passing a linear combination of features through the logistic (sigmoid) function. Despite its name, it is a classification method, not a regression method, and it remains one of the most widely used and interpretable models in statistics, epidemiology, and machine learning.		'''Logistic regression''' is a foundational statistical model for binary classification that predicts the probability of a categorical outcome by passing a linear combination of features through the logistic (sigmoid) function. Despite its name, it is a classification method, not a regression method, and it remains one of the most widely used and interpretable models in statistics, epidemiology, and machine learning.

	<!--T:2-->		== Overview == <!--T:2-->
	~~== Overview ==~~

	<!--T:3-->		<!--T:3-->
Line 23:		Line 22:
	The popularity of logistic regression stems from a rare combination of properties: it is a probabilistic classifier with a convex loss, parameters are easy to interpret as log-odds ratios, training scales to massive datasets via [[Stochastic Gradient Descent]], and it doubles as the final layer of most modern [[Neural Networks]] used for binary classification.		The popularity of logistic regression stems from a rare combination of properties: it is a probabilistic classifier with a convex loss, parameters are easy to interpret as log-odds ratios, training scales to massive datasets via [[Stochastic Gradient Descent]], and it doubles as the final layer of most modern [[Neural Networks]] used for binary classification.

	<!--T:5-->		== Key Concepts == <!--T:5-->
	~~== Key Concepts ==~~

	<!--T:6-->		<!--T:6-->
Line 34:		Line 32:
	* '''{{Term\|convex optimization\|Convex optimisation}}''': the loss is convex in the parameters, so any local minimum is a global one.		* '''{{Term\|convex optimization\|Convex optimisation}}''': the loss is convex in the parameters, so any local minimum is a global one.

	<!--T:7-->		== History == <!--T:7-->
	~~== History ==~~

	<!--T:8-->		<!--T:8-->
Line 46:		Line 43:
	In the 1970s and 1980s, logistic regression became the default model for case-control studies in epidemiology, partly because the odds ratio it produces is invariant to outcome-based sampling. With the rise of machine learning, the model found a second life as a baseline classifier and as the output layer of [[Neural Networks]]. Multinomial logistic regression, which generalises the model to more than two classes via the [[Softmax Function]], is the workhorse classifier underlying nearly all modern deep classification systems.		In the 1970s and 1980s, logistic regression became the default model for case-control studies in epidemiology, partly because the odds ratio it produces is invariant to outcome-based sampling. With the rise of machine learning, the model found a second life as a baseline classifier and as the output layer of [[Neural Networks]]. Multinomial logistic regression, which generalises the model to more than two classes via the [[Softmax Function]], is the workhorse classifier underlying nearly all modern deep classification systems.

	<!--T:11-->		== Key Approaches == <!--T:11-->
	~~== Key Approaches ==~~

	<!--T:12-->		=== Model Specification === <!--T:12-->
	~~=== Model Specification ===~~

	<!--T:13-->		<!--T:13-->
Line 67:		Line 62:
	A unit increase in <math>x_j</math> multiplies the odds of the positive class by <math>e^{w_j}</math>, holding other features fixed. This direct interpretation of coefficients as '''log-odds ratios''' is one of the model's defining strengths.		A unit increase in <math>x_j</math> multiplies the odds of the positive class by <math>e^{w_j}</math>, holding other features fixed. This direct interpretation of coefficients as '''log-odds ratios''' is one of the model's defining strengths.

	~~<!--T:18-->~~		=== Maximum Likelihood and Cross-Entropy === <!--T:18-->
	=== Maximum Likelihood and Cross-Entropy ===

	<!--T:19-->		<!--T:19-->
Line 91:		Line 85:
	i.e. the average feature vector weighted by the prediction error.		i.e. the average feature vector weighted by the prediction error.

	<!--T:26-->		=== Optimisation === <!--T:26-->
	~~=== Optimisation ===~~

	<!--T:27-->		<!--T:27-->
Line 102:		Line 95:
	* '''[[Stochastic Gradient Descent]]''': the default for large-scale and online settings, with the same gradient form as a single-layer neural network.		* '''[[Stochastic Gradient Descent]]''': the default for large-scale and online settings, with the same gradient form as a single-layer neural network.

	<!--T:29-->		=== Regularisation === <!--T:29-->
	~~=== Regularisation ===~~

	<!--T:30-->		<!--T:30-->
Line 114:		Line 106:
	L2 (ridge) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \tfrac{1}{2}\\|\mathbf{w}\\|_2^2</math>, shrinks weights toward zero and corresponds to a Gaussian prior. L1 (lasso) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \\|\mathbf{w}\\|_1</math>, promotes sparsity and acts as feature selection. Elastic Net combines both. See [[Overfitting and Regularization]] for the broader treatment.		L2 (ridge) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \tfrac{1}{2}\\|\mathbf{w}\\|_2^2</math>, shrinks weights toward zero and corresponds to a Gaussian prior. L1 (lasso) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \\|\mathbf{w}\\|_1</math>, promotes sparsity and acts as feature selection. Elastic Net combines both. See [[Overfitting and Regularization]] for the broader treatment.

	<!--T:33-->		=== Multinomial Extension === <!--T:33-->
	~~=== Multinomial Extension ===~~

	<!--T:34-->		<!--T:34-->
Line 126:		Line 117:
	This is exactly the [[Softmax Function]] applied to a linear score, and it forms the output layer of essentially every modern multi-class classifier built with [[Neural Networks]].		This is exactly the [[Softmax Function]] applied to a linear score, and it forms the output layer of essentially every modern multi-class classifier built with [[Neural Networks]].

	<!--T:37-->		== Connections == <!--T:37-->
	~~== Connections ==~~

	<!--T:38-->		<!--T:38-->
Line 135:		Line 125:
	Logistic regression is also a '''generalised linear model''' (GLM) with a Bernoulli response and the canonical {{Term\|logits\|logit}} link, placing it in the same family as Poisson regression and [[Linear Regression]] (with Gaussian response and identity link). It is closely related to '''linear discriminant analysis''' (LDA): both produce linear decision boundaries, but LDA models <math>P(\mathbf{x} \mid y)</math> while logistic regression models <math>P(y \mid \mathbf{x})</math> directly, making it a '''discriminative''' rather than '''generative''' classifier. The multinomial form connects directly to the [[Softmax Function]] and is the standard final layer for classifiers operating on [[Word Embeddings]] and the outputs of [[Attention Mechanisms]] alike.		Logistic regression is also a '''generalised linear model''' (GLM) with a Bernoulli response and the canonical {{Term\|logits\|logit}} link, placing it in the same family as Poisson regression and [[Linear Regression]] (with Gaussian response and identity link). It is closely related to '''linear discriminant analysis''' (LDA): both produce linear decision boundaries, but LDA models <math>P(\mathbf{x} \mid y)</math> while logistic regression models <math>P(y \mid \mathbf{x})</math> directly, making it a '''discriminative''' rather than '''generative''' classifier. The multinomial form connects directly to the [[Softmax Function]] and is the standard final layer for classifiers operating on [[Word Embeddings]] and the outputs of [[Attention Mechanisms]] alike.

	<!--T:40-->		== See also == <!--T:40-->
	~~== See also ==~~

	<!--T:41-->		<!--T:41-->
Line 148:		Line 137:
	* [[Loss Functions]]		* [[Loss Functions]]

	<!--T:42-->		== References == <!--T:42-->
	~~== References ==~~

	<!--T:43-->		<!--T:43-->

DeployBot: [deploy-bot] Claude-authored article: Logistic regression

2026-04-27T23:32:43Z

[deploy-bot] Claude-authored article: Logistic regression

← Older revision		Revision as of 23:32, 27 April 2026
Line 14:		Line 14:
	'''Logistic regression''' is a foundational statistical model for binary classification that predicts the probability of a categorical outcome by passing a linear combination of features through the logistic (sigmoid) function. Despite its name, it is a classification method, not a regression method, and it remains one of the most widely used and interpretable models in statistics, epidemiology, and machine learning.		'''Logistic regression''' is a foundational statistical model for binary classification that predicts the probability of a categorical outcome by passing a linear combination of features through the logistic (sigmoid) function. Despite its name, it is a classification method, not a regression method, and it remains one of the most widely used and interpretable models in statistics, epidemiology, and machine learning.

	~~== Overview ==~~ <!--T:2-->		<!--T:2-->
			== Overview ==

	<!--T:3-->		<!--T:3-->
Line 22:		Line 23:
	The popularity of logistic regression stems from a rare combination of properties: it is a probabilistic classifier with a convex loss, parameters are easy to interpret as log-odds ratios, training scales to massive datasets via [[Stochastic Gradient Descent]], and it doubles as the final layer of most modern [[Neural Networks]] used for binary classification.		The popularity of logistic regression stems from a rare combination of properties: it is a probabilistic classifier with a convex loss, parameters are easy to interpret as log-odds ratios, training scales to massive datasets via [[Stochastic Gradient Descent]], and it doubles as the final layer of most modern [[Neural Networks]] used for binary classification.

	~~== Key Concepts ==~~ <!--T:5-->		<!--T:5-->
			== Key Concepts ==

	<!--T:6-->		<!--T:6-->
Line 32:		Line 34:
	* '''{{Term\|convex optimization\|Convex optimisation}}''': the loss is convex in the parameters, so any local minimum is a global one.		* '''{{Term\|convex optimization\|Convex optimisation}}''': the loss is convex in the parameters, so any local minimum is a global one.

	~~== History ==~~ <!--T:7-->		<!--T:7-->
			== History ==

	<!--T:8-->		<!--T:8-->
Line 43:		Line 46:
	In the 1970s and 1980s, logistic regression became the default model for case-control studies in epidemiology, partly because the odds ratio it produces is invariant to outcome-based sampling. With the rise of machine learning, the model found a second life as a baseline classifier and as the output layer of [[Neural Networks]]. Multinomial logistic regression, which generalises the model to more than two classes via the [[Softmax Function]], is the workhorse classifier underlying nearly all modern deep classification systems.		In the 1970s and 1980s, logistic regression became the default model for case-control studies in epidemiology, partly because the odds ratio it produces is invariant to outcome-based sampling. With the rise of machine learning, the model found a second life as a baseline classifier and as the output layer of [[Neural Networks]]. Multinomial logistic regression, which generalises the model to more than two classes via the [[Softmax Function]], is the workhorse classifier underlying nearly all modern deep classification systems.

	~~== Key Approaches ==~~ <!--T:11-->		<!--T:11-->
			== Key Approaches ==

	~~=== Model Specification ===~~ <!--T:12-->		<!--T:12-->
			=== Model Specification ===

	<!--T:13-->		<!--T:13-->
Line 62:		Line 67:
	A unit increase in <math>x_j</math> multiplies the odds of the positive class by <math>e^{w_j}</math>, holding other features fixed. This direct interpretation of coefficients as '''log-odds ratios''' is one of the model's defining strengths.		A unit increase in <math>x_j</math> multiplies the odds of the positive class by <math>e^{w_j}</math>, holding other features fixed. This direct interpretation of coefficients as '''log-odds ratios''' is one of the model's defining strengths.

	=== Maximum Likelihood and Cross-Entropy === ~~<!--T:18-->~~		<!--T:18-->
			=== Maximum Likelihood and Cross-Entropy ===

	<!--T:19-->		<!--T:19-->
Line 85:		Line 91:
	i.e. the average feature vector weighted by the prediction error.		i.e. the average feature vector weighted by the prediction error.

	~~=== Optimisation ===~~ <!--T:26-->		<!--T:26-->
			=== Optimisation ===

	<!--T:27-->		<!--T:27-->
Line 95:		Line 102:
	* '''[[Stochastic Gradient Descent]]''': the default for large-scale and online settings, with the same gradient form as a single-layer neural network.		* '''[[Stochastic Gradient Descent]]''': the default for large-scale and online settings, with the same gradient form as a single-layer neural network.

	~~=== Regularisation ===~~ <!--T:29-->		<!--T:29-->
			=== Regularisation ===

	<!--T:30-->		<!--T:30-->
Line 106:		Line 114:
	L2 (ridge) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \tfrac{1}{2}\\|\mathbf{w}\\|_2^2</math>, shrinks weights toward zero and corresponds to a Gaussian prior. L1 (lasso) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \\|\mathbf{w}\\|_1</math>, promotes sparsity and acts as feature selection. Elastic Net combines both. See [[Overfitting and Regularization]] for the broader treatment.		L2 (ridge) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \tfrac{1}{2}\\|\mathbf{w}\\|_2^2</math>, shrinks weights toward zero and corresponds to a Gaussian prior. L1 (lasso) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \\|\mathbf{w}\\|_1</math>, promotes sparsity and acts as feature selection. Elastic Net combines both. See [[Overfitting and Regularization]] for the broader treatment.

	~~=== Multinomial Extension ===~~ <!--T:33-->		<!--T:33-->
			=== Multinomial Extension ===

	<!--T:34-->		<!--T:34-->
Line 117:		Line 126:
	This is exactly the [[Softmax Function]] applied to a linear score, and it forms the output layer of essentially every modern multi-class classifier built with [[Neural Networks]].		This is exactly the [[Softmax Function]] applied to a linear score, and it forms the output layer of essentially every modern multi-class classifier built with [[Neural Networks]].

	~~== Connections ==~~ <!--T:37-->		<!--T:37-->
			== Connections ==

	<!--T:38-->		<!--T:38-->
Line 125:		Line 135:
	Logistic regression is also a '''generalised linear model''' (GLM) with a Bernoulli response and the canonical {{Term\|logits\|logit}} link, placing it in the same family as Poisson regression and [[Linear Regression]] (with Gaussian response and identity link). It is closely related to '''linear discriminant analysis''' (LDA): both produce linear decision boundaries, but LDA models <math>P(\mathbf{x} \mid y)</math> while logistic regression models <math>P(y \mid \mathbf{x})</math> directly, making it a '''discriminative''' rather than '''generative''' classifier. The multinomial form connects directly to the [[Softmax Function]] and is the standard final layer for classifiers operating on [[Word Embeddings]] and the outputs of [[Attention Mechanisms]] alike.		Logistic regression is also a '''generalised linear model''' (GLM) with a Bernoulli response and the canonical {{Term\|logits\|logit}} link, placing it in the same family as Poisson regression and [[Linear Regression]] (with Gaussian response and identity link). It is closely related to '''linear discriminant analysis''' (LDA): both produce linear decision boundaries, but LDA models <math>P(\mathbf{x} \mid y)</math> while logistic regression models <math>P(y \mid \mathbf{x})</math> directly, making it a '''discriminative''' rather than '''generative''' classifier. The multinomial form connects directly to the [[Softmax Function]] and is the standard final layer for classifiers operating on [[Word Embeddings]] and the outputs of [[Attention Mechanisms]] alike.

	~~== See also ==~~ <!--T:40-->		<!--T:40-->
			== See also ==

	<!--T:41-->		<!--T:41-->
Line 137:		Line 148:
	* [[Loss Functions]]		* [[Loss Functions]]

	~~== References ==~~ <!--T:42-->		<!--T:42-->
			== References ==

	<!--T:43-->		<!--T:43-->

DeployBot: Marked this version for translation

2026-04-27T21:57:43Z

Marked this version for translation

← Older revision		Revision as of 21:57, 27 April 2026
Line 14:		Line 14:
	'''Logistic regression''' is a foundational statistical model for binary classification that predicts the probability of a categorical outcome by passing a linear combination of features through the logistic (sigmoid) function. Despite its name, it is a classification method, not a regression method, and it remains one of the most widely used and interpretable models in statistics, epidemiology, and machine learning.		'''Logistic regression''' is a foundational statistical model for binary classification that predicts the probability of a categorical outcome by passing a linear combination of features through the logistic (sigmoid) function. Despite its name, it is a classification method, not a regression method, and it remains one of the most widely used and interpretable models in statistics, epidemiology, and machine learning.

	<!--T:2-->		== Overview == <!--T:2-->
	~~== Overview ==~~

	<!--T:3-->		<!--T:3-->
Line 23:		Line 22:
	The popularity of logistic regression stems from a rare combination of properties: it is a probabilistic classifier with a convex loss, parameters are easy to interpret as log-odds ratios, training scales to massive datasets via [[Stochastic Gradient Descent]], and it doubles as the final layer of most modern [[Neural Networks]] used for binary classification.		The popularity of logistic regression stems from a rare combination of properties: it is a probabilistic classifier with a convex loss, parameters are easy to interpret as log-odds ratios, training scales to massive datasets via [[Stochastic Gradient Descent]], and it doubles as the final layer of most modern [[Neural Networks]] used for binary classification.

	<!--T:5-->		== Key Concepts == <!--T:5-->
	~~== Key Concepts ==~~

	<!--T:6-->		<!--T:6-->
Line 34:		Line 32:
	* '''{{Term\|convex optimization\|Convex optimisation}}''': the loss is convex in the parameters, so any local minimum is a global one.		* '''{{Term\|convex optimization\|Convex optimisation}}''': the loss is convex in the parameters, so any local minimum is a global one.

	<!--T:7-->		== History == <!--T:7-->
	~~== History ==~~

	<!--T:8-->		<!--T:8-->
Line 46:		Line 43:
	In the 1970s and 1980s, logistic regression became the default model for case-control studies in epidemiology, partly because the odds ratio it produces is invariant to outcome-based sampling. With the rise of machine learning, the model found a second life as a baseline classifier and as the output layer of [[Neural Networks]]. Multinomial logistic regression, which generalises the model to more than two classes via the [[Softmax Function]], is the workhorse classifier underlying nearly all modern deep classification systems.		In the 1970s and 1980s, logistic regression became the default model for case-control studies in epidemiology, partly because the odds ratio it produces is invariant to outcome-based sampling. With the rise of machine learning, the model found a second life as a baseline classifier and as the output layer of [[Neural Networks]]. Multinomial logistic regression, which generalises the model to more than two classes via the [[Softmax Function]], is the workhorse classifier underlying nearly all modern deep classification systems.

	<!--T:11-->		== Key Approaches == <!--T:11-->
	~~== Key Approaches ==~~

	<!--T:12-->		=== Model Specification === <!--T:12-->
	~~=== Model Specification ===~~

	<!--T:13-->		<!--T:13-->
Line 67:		Line 62:
	A unit increase in <math>x_j</math> multiplies the odds of the positive class by <math>e^{w_j}</math>, holding other features fixed. This direct interpretation of coefficients as '''log-odds ratios''' is one of the model's defining strengths.		A unit increase in <math>x_j</math> multiplies the odds of the positive class by <math>e^{w_j}</math>, holding other features fixed. This direct interpretation of coefficients as '''log-odds ratios''' is one of the model's defining strengths.

	~~<!--T:18-->~~		=== Maximum Likelihood and Cross-Entropy === <!--T:18-->
	=== Maximum Likelihood and Cross-Entropy ===

	<!--T:19-->		<!--T:19-->
Line 91:		Line 85:
	i.e. the average feature vector weighted by the prediction error.		i.e. the average feature vector weighted by the prediction error.

	<!--T:26-->		=== Optimisation === <!--T:26-->
	~~=== Optimisation ===~~

	<!--T:27-->		<!--T:27-->
Line 102:		Line 95:
	* '''[[Stochastic Gradient Descent]]''': the default for large-scale and online settings, with the same gradient form as a single-layer neural network.		* '''[[Stochastic Gradient Descent]]''': the default for large-scale and online settings, with the same gradient form as a single-layer neural network.

	<!--T:29-->		=== Regularisation === <!--T:29-->
	~~=== Regularisation ===~~

	<!--T:30-->		<!--T:30-->
Line 114:		Line 106:
	L2 (ridge) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \tfrac{1}{2}\\|\mathbf{w}\\|_2^2</math>, shrinks weights toward zero and corresponds to a Gaussian prior. L1 (lasso) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \\|\mathbf{w}\\|_1</math>, promotes sparsity and acts as feature selection. Elastic Net combines both. See [[Overfitting and Regularization]] for the broader treatment.		L2 (ridge) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \tfrac{1}{2}\\|\mathbf{w}\\|_2^2</math>, shrinks weights toward zero and corresponds to a Gaussian prior. L1 (lasso) {{Term\|regularization\|regularisation}}, <math>R(\mathbf{w}) = \\|\mathbf{w}\\|_1</math>, promotes sparsity and acts as feature selection. Elastic Net combines both. See [[Overfitting and Regularization]] for the broader treatment.

	<!--T:33-->		=== Multinomial Extension === <!--T:33-->
	~~=== Multinomial Extension ===~~

	<!--T:34-->		<!--T:34-->
Line 126:		Line 117:
	This is exactly the [[Softmax Function]] applied to a linear score, and it forms the output layer of essentially every modern multi-class classifier built with [[Neural Networks]].		This is exactly the [[Softmax Function]] applied to a linear score, and it forms the output layer of essentially every modern multi-class classifier built with [[Neural Networks]].

	<!--T:37-->		== Connections == <!--T:37-->
	~~== Connections ==~~

	<!--T:38-->		<!--T:38-->
Line 135:		Line 125:
	Logistic regression is also a '''generalised linear model''' (GLM) with a Bernoulli response and the canonical {{Term\|logits\|logit}} link, placing it in the same family as Poisson regression and [[Linear Regression]] (with Gaussian response and identity link). It is closely related to '''linear discriminant analysis''' (LDA): both produce linear decision boundaries, but LDA models <math>P(\mathbf{x} \mid y)</math> while logistic regression models <math>P(y \mid \mathbf{x})</math> directly, making it a '''discriminative''' rather than '''generative''' classifier. The multinomial form connects directly to the [[Softmax Function]] and is the standard final layer for classifiers operating on [[Word Embeddings]] and the outputs of [[Attention Mechanisms]] alike.		Logistic regression is also a '''generalised linear model''' (GLM) with a Bernoulli response and the canonical {{Term\|logits\|logit}} link, placing it in the same family as Poisson regression and [[Linear Regression]] (with Gaussian response and identity link). It is closely related to '''linear discriminant analysis''' (LDA): both produce linear decision boundaries, but LDA models <math>P(\mathbf{x} \mid y)</math> while logistic regression models <math>P(y \mid \mathbf{x})</math> directly, making it a '''discriminative''' rather than '''generative''' classifier. The multinomial form connects directly to the [[Softmax Function]] and is the standard final layer for classifiers operating on [[Word Embeddings]] and the outputs of [[Attention Mechanisms]] alike.

	<!--T:40-->		== See also == <!--T:40-->
	~~== See also ==~~

	<!--T:41-->		<!--T:41-->
Line 148:		Line 137:
	* [[Loss Functions]]		* [[Loss Functions]]

	<!--T:42-->		== References == <!--T:42-->
	~~== References ==~~

	<!--T:43-->		<!--T:43-->

DeployBot: [deploy-bot] Claude-authored article: Logistic regression

2026-04-27T21:57:42Z

[deploy-bot] Claude-authored article: Logistic regression

New page

<languages />
{{ArticleInfobox
| topic_area = Machine Learning
| difficulty = Introductory
}}
{{ContentMeta
| generated_by = claude-code-direct
| model_used = claude-opus-4-7
| generated_date = 2026-04-27
}}

<translate>

'''Logistic regression''' is a foundational statistical model for binary classification that predicts the probability of a categorical outcome by passing a linear combination of features through the logistic (sigmoid) function. Despite its name, it is a classification method, not a regression method, and it remains one of the most widely used and interpretable models in statistics, epidemiology, and machine learning.


== Overview ==


Logistic regression models the probability that an observation belongs to a positive class as a function of input features. Given a feature vector <math>\mathbf{x} \in \mathbb{R}^d</math> and a binary label <math>y \in \{0, 1\}</math>, the model assumes that the log-odds of the positive class are a linear function of <math>\mathbf{x}</math>. The output is constrained to the unit interval, making it directly interpretable as a probability and well-suited for downstream decisions, calibration, and risk scoring.


The popularity of logistic regression stems from a rare combination of properties: it is a probabilistic classifier with a convex loss, parameters are easy to interpret as log-odds ratios, training scales to massive datasets via [[Stochastic Gradient Descent]], and it doubles as the final layer of most modern [[Neural Networks]] used for binary classification.


== Key Concepts ==


* '''Sigmoid (logistic) function''': the squashing {{Term|activation function|nonlinearity}} <math>\sigma(z) = 1/(1 + e^{-z})</math> that maps any real number to <math>(0, 1)</math>.
* '''Linear decision boundary''': in {{Term|latent space|feature space}}, the set of points where <math>\mathbf{w}^{\!\top}\mathbf{x} + b = 0</math> separates the two classes; logistic regression is therefore a '''linear classifier'''.
* '''Log-odds ({{Term|logits|logit}})''': the inverse of the sigmoid, <math>\mathrm{logit}(p) = \log\frac{p}{1-p}</math>; logistic regression assumes the {{Term|logits|logit}} is linear in the features.
* '''Maximum likelihood estimation''': parameters are fit by maximising the probability of the observed labels under the model.
* '''{{Term|categorical cross-entropy|Cross-entropy loss}}''': the negative log-likelihood of the Bernoulli model, equivalent to the [[Cross-Entropy Loss]] used in {{Term|deep learning}}.
* '''{{Term|convex optimization|Convex optimisation}}''': the loss is convex in the parameters, so any local minimum is a global one.


== History ==


The logistic function was introduced by Belgian mathematician Pierre François Verhulst in 1838 to model population growth under resource constraints. Its use as a statistical tool grew through the early twentieth century in chemistry and biology, where it described autocatalytic reactions and dose-response curves.


The modern statistical formulation took shape in the mid-twentieth century. Joseph Berkson popularised the term '''{{Term|logits|logit}}''' in 1944 as an alternative to the probit model favoured by Chester Bliss and R. A. Fisher. David Cox's 1958 paper "The Regression Analysis of Binary Sequences" established logistic regression as the standard tool for binary outcomes in statistics, and Walker and Duncan (1967) extended it to multiple covariates.


In the 1970s and 1980s, logistic regression became the default model for case-control studies in epidemiology, partly because the odds ratio it produces is invariant to outcome-based sampling. With the rise of machine learning, the model found a second life as a baseline classifier and as the output layer of [[Neural Networks]]. Multinomial logistic regression, which generalises the model to more than two classes via the [[Softmax Function]], is the workhorse classifier underlying nearly all modern deep classification systems.


== Key Approaches ==


=== Model Specification ===


For a binary label <math>y \in \{0, 1\}</math>, logistic regression models


:<math>P(y = 1 \mid \mathbf{x}) = \sigma(\mathbf{w}^{\!\top}\mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^{\!\top}\mathbf{x} + b)}}</math>


Equivalently, the log-odds are linear:


:<math>\log \frac{P(y=1 \mid \mathbf{x})}{P(y=0 \mid \mathbf{x})} = \mathbf{w}^{\!\top}\mathbf{x} + b</math>


A unit increase in <math>x_j</math> multiplies the odds of the positive class by <math>e^{w_j}</math>, holding other features fixed. This direct interpretation of coefficients as '''log-odds ratios''' is one of the model's defining strengths.


=== Maximum Likelihood and Cross-Entropy ===


Given a dataset <math>\{(\mathbf{x}_i, y_i)\}_{i=1}^{N}</math>, the likelihood under the Bernoulli model is


:<math>\mathcal{L}(\mathbf{w}, b) = \prod_{i=1}^{N} p_i^{y_i}(1 - p_i)^{1 - y_i}, \quad p_i = \sigma(\mathbf{w}^{\!\top}\mathbf{x}_i + b)</math>


Taking the negative log gives the '''binary {{Term|categorical cross-entropy|cross-entropy}}''' loss:


:<math>\mathcal{J}(\mathbf{w}, b) = -\frac{1}{N}\sum_{i=1}^{N} \big[y_i \log p_i + (1 - y_i)\log(1 - p_i)\big]</math>


This loss is convex, and its gradient has the elegant form


:<math>\nabla_{\mathbf{w}} \mathcal{J} = \frac{1}{N} \sum_{i=1}^{N} (p_i - y_i)\,\mathbf{x}_i</math>


i.e. the average feature vector weighted by the prediction error.


=== Optimisation ===


Unlike linear regression, logistic regression has no closed-form solution. Standard optimisation choices include:


* '''Iteratively reweighted least squares (IRLS)''': the classical statistical algorithm, equivalent to Newton's method on the log-likelihood; converges in few iterations on small problems.
* '''[[Gradient Descent]] and L-BFGS''': practical for medium-scale problems where IRLS is too memory-hungry.
* '''[[Stochastic Gradient Descent]]''': the default for large-scale and online settings, with the same gradient form as a single-layer neural network.


=== Regularisation ===


To prevent {{Term|overfitting}} and stabilise estimates when features are correlated or numerous, the loss is augmented with a penalty:


:<math>\mathcal{J}_{\mathrm{reg}}(\mathbf{w}) = \mathcal{J}(\mathbf{w}) + \lambda\, R(\mathbf{w})</math>


L2 (ridge) {{Term|regularization|regularisation}}, <math>R(\mathbf{w}) = \tfrac{1}{2}\|\mathbf{w}\|_2^2</math>, shrinks weights toward zero and corresponds to a Gaussian prior. L1 (lasso) {{Term|regularization|regularisation}}, <math>R(\mathbf{w}) = \|\mathbf{w}\|_1</math>, promotes sparsity and acts as feature selection. Elastic Net combines both. See [[Overfitting and Regularization]] for the broader treatment.


=== Multinomial Extension ===


For <math>K > 2</math> classes, logistic regression generalises to '''multinomial logistic regression''' (also called {{Term|softmax}} regression):


:<math>P(y = k \mid \mathbf{x}) = \frac{\exp(\mathbf{w}_k^{\!\top}\mathbf{x} + b_k)}{\sum_{j=1}^{K} \exp(\mathbf{w}_j^{\!\top}\mathbf{x} + b_j)}</math>


This is exactly the [[Softmax Function]] applied to a linear score, and it forms the output layer of essentially every modern multi-class classifier built with [[Neural Networks]].


== Connections ==


Logistic regression sits at a crossroads of several major themes in statistics and machine learning. Structurally, it is the simplest case of a [[Neural Networks|neural network]]: a single neuron with a sigmoid {{Term|activation function|activation}}. Its {{Term|loss function}} is precisely the [[Cross-Entropy Loss]] used to train deep classifiers, and the gradient computation is a one-step instance of [[Backpropagation]]. The optimiser of choice in modern practice — [[Stochastic Gradient Descent]] — was originally developed and analysed in the context of generalised linear models such as this one.


Logistic regression is also a '''generalised linear model''' (GLM) with a Bernoulli response and the canonical {{Term|logits|logit}} link, placing it in the same family as Poisson regression and [[Linear Regression]] (with Gaussian response and identity link). It is closely related to '''linear discriminant analysis''' (LDA): both produce linear decision boundaries, but LDA models <math>P(\mathbf{x} \mid y)</math> while logistic regression models <math>P(y \mid \mathbf{x})</math> directly, making it a '''discriminative''' rather than '''generative''' classifier. The multinomial form connects directly to the [[Softmax Function]] and is the standard final layer for classifiers operating on [[Word Embeddings]] and the outputs of [[Attention Mechanisms]] alike.


== See also ==


* [[Linear Regression]]
* [[Cross-Entropy Loss]]
* [[Softmax Function]]
* [[Gradient Descent]]
* [[Stochastic Gradient Descent]]
* [[Neural Networks]]
* [[Overfitting and Regularization]]
* [[Loss Functions]]


== References ==


* Cox, D. R. (1958). "The Regression Analysis of Binary Sequences". ''Journal of the Royal Statistical Society, Series B'', 20(2), 215–242.
* Berkson, J. (1944). "Application of the Logistic Function to Bio-Assay". ''Journal of the American Statistical Association'', 39(227), 357–365.
* Hosmer, D. W., Lemeshow, S. and Sturdivant, R. X. (2013). ''Applied Logistic Regression'' (3rd ed.). Wiley.
* McCullagh, P. and Nelder, J. A. (1989). ''Generalized Linear Models'' (2nd ed.). Chapman and Hall.
* Bishop, C. M. (2006). ''Pattern Recognition and Machine Learning''. Springer, Chapter 4.
* Hastie, T., Tibshirani, R. and Friedman, J. (2009). ''The Elements of Statistical Learning'' (2nd ed.). Springer, Chapter 4.
</translate>

[[Category:Machine Learning]]
[[Category:Introductory]]