Logistic Models and the Margins Command
Introduction
When looking at the results of a logistic model, there are several different measures of the relationship between the predictors and the probability of a positive outcome that can be used to interpret the model:
- The odds ratios
- The log odds
- The odds
- Marginal probablities/percentages
If you are unclear which you are looking at, confusion can abound. This is doubly-confounded in Stata (in my opinion) where certain margins
commands will produce a different measure than perhaps expected.
Intercept only model
Let’s start simple and consider a model with only an intercept. We’ll use the “auto” data set, and fit a model predicting the probability of a car being foreign made.
sysuse auto
. data)
(1978 automobile
logit foreign, nolog
.
of obs = 74
Logistic regression Number chi2(0) = 0.00
LR chi2 = .
Prob >
Log likelihood = -45.03321 Pseudo R2 = 0.0000
------------------------------------------------------------------------------
foreign | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------_cons | -.8602013 .2543331 -3.38 0.001 -1.358685 -.3617176
------------------------------------------------------------------------------
logit, or
.
of obs = 74
Logistic regression Number chi2(0) = 0.00
LR chi2 = .
Prob >
Log likelihood = -45.03321 Pseudo R2 = 0.0000
------------------------------------------------------------------------------
foreign | Odds Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------_cons | .4230769 .1076025 -3.38 0.001 .2569985 .696479
------------------------------------------------------------------------------
The default output from the logit
command are the log odds, -.86 and passing the or
option gives the odds ratio of .423. The conversion between these values is straightforward:
\[ \textrm{log odds} = log(\textrm{odds}) \] \[ \textrm{odds} = \exp^{\textrm{log odds}} \]
Let’s backtrack to see how we arrive at those values.
First, let’s look at the breakdown of foreign and domestic cars.
tab foreign
.
Car origin | Freq. Percent Cum.
------------+-----------------------------------
Domestic | 52 70.27 70.27
Foreign | 22 29.73 100.00
------------+----------------------------------- Total | 74 100.00
We see that \(P(\textrm{foreign})\) = .2973 and \(P(\textrm{domestic})\) = .7027. We can convert these probabilities into odds, using the formula
\[ \textrm{Odds}(\textrm{foreign}) = \frac{P(\textrm{foreign})}{1 - P(\textrm{foreign})}. \]
Therefore we can easily see that
\[ \textrm{Odds}(\textrm{foreign}) = \frac{ .2973}{1 - .2973} = .4231 \]
and
\[ \textrm{Odds}(\textrm{domestic}) = \frac{ .7027}{1 - .7027} = 2.364. \]
For completeness, to convert from odds to probability you can use
\[ P(\textrm{foreign}) = \frac{\textrm{Odds}(\textrm{foreign})}{1 + \textrm{Odds}(\textrm{foreign})}. \]
Note that the odds of a car being foreign is exactly the result we saw above from logit, or
. So in a logistic model with only an intercept, the coefficient on the intercept is the odds of a positive outcome.
Rather than calculate these manually, Stata can produce these automatically.
\(P(\textrm{foreign})\):
. marginsover observations.
warning: prediction constant
of obs = 74
Predictive margins Number VCE: OIM
Model
predict()
Expression: Pr(foreign),
------------------------------------------------------------------------------
| Delta-methodstd. err. z P>|z| [95% conf. interval]
| Margin
-------------+----------------------------------------------------------------_cons | .2972973 .0531331 5.60 0.000 .1931583 .4014363
------------------------------------------------------------------------------
\(\textrm{Odds}(\textrm{foreign})\):
exp(xb()))
. margins, expression(over observations.
warning: prediction constant
of obs = 74
Predictive margins Number VCE: OIM
Model
exp(xb())
Expression:
------------------------------------------------------------------------------
| Delta-methodstd. err. z P>|z| [95% conf. interval]
| Margin
-------------+----------------------------------------------------------------_cons | .4230769 .1076025 3.93 0.000 .21218 .6339739
------------------------------------------------------------------------------
The expression(exp(xb()))
is a bit odd, but the easiest way to obtain what we need. Think of it as just saying “give me the odds”.
A single binary predictor
In the intercept only example, we had no concept of an odds ratio. Let’s add a fixed effect, in this case a binary predictor, which will require interpreting an odds ratio.
gen highmileage = mpg > 25
.
label define highmileage 0 "Low Mileage" 1 "High Mileage"
.
label value highmileage highmileage
.
tab foreign highmileage
.
| highmileage
Car origin | Low Milea High Mile | Total
-----------+----------------------+----------
Domestic | 45 7 | 52
Foreign | 15 7 | 22
-----------+----------------------+----------
Total | 60 14 | 74
logit foreign i.highmileage, nolog
.
of obs = 74
Logistic regression Number chi2(1) = 3.18
LR chi2 = 0.0746
Prob >
Log likelihood = -43.444169 Pseudo R2 = 0.0353
------------------------------------------------------------------------------
foreign | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
highmileage |
High Mile.. | 1.098612 .6120483 1.79 0.073 -.1009804 2.298205_cons | -1.098612 .2981424 -3.68 0.000 -1.682961 -.5142639
------------------------------------------------------------------------------
logit, or
.
of obs = 74
Logistic regression Number chi2(1) = 3.18
LR chi2 = 0.0746
Prob >
Log likelihood = -43.444169 Pseudo R2 = 0.0353
------------------------------------------------------------------------------ratio Std. err. z P>|z| [95% conf. interval]
foreign | Odds
-------------+----------------------------------------------------------------
highmileage |
High Mile.. | 3 1.836145 1.79 0.073 .9039507 9.956295_cons | .3333333 .0993808 -3.68 0.000 .185823 .5979406
------------------------------------------------------------------------------_cons estimates baseline odds. Note:
Probabilities
Let’s look at the probabilities. Here we have conditional probabilities since we have a predictor. So we are interested in \(P(\textrm{foreign} | \textrm{high mileage})\) and \(P(\textrm{foreign} | \textrm{low mileage})\).
From the table above, we can easily compute this:
tab foreign highmileage, col
.
+-------------------+
| Key |
|-------------------|frequency |
|
| column percentage |
+-------------------+
| highmileage
Car origin | Low Milea High Mile | Total
-----------+----------------------+----------
Domestic | 45 7 | 52
| 75.00 50.00 | 70.27
-----------+----------------------+----------
Foreign | 15 7 | 22
| 25.00 50.00 | 29.73
-----------+----------------------+----------
Total | 60 14 | 74 | 100.00 100.00 | 100.00
We see \(P(\textrm{foreign} | \textrm{high mileage}) = .25\) and \(P(\textrm{foreign} | \textrm{low mileage}) = .5\).
We can also obtain these via margins
1:
. margins highmileage
of obs = 74
Adjusted predictions Number VCE: OIM
Model
predict()
Expression: Pr(foreign),
------------------------------------------------------------------------------
| Delta-methodstd. err. z P>|z| [95% conf. interval]
| Margin
-------------+----------------------------------------------------------------
highmileage |
Low Mileage | .25 .0559017 4.47 0.000 .1404347 .3595653
High Mile.. | .5 .1336306 3.74 0.000 .2380888 .7619112 ------------------------------------------------------------------------------
We can also test for equality between these percentages:
. margins highmileage, pwcompare(pv)
of adjusted predictions Number of obs = 74
Pairwise comparisons VCE: OIM
Model
predict()
Expression: Pr(foreign),
---------------------------------------------------------------------
| Delta-method Unadjustedstd. err. z P>|z|
| Contrast
-----------------------------+---------------------------------------
highmileage |
High Mileage vs Low Mileage | .25 .1448521 1.73 0.084 ---------------------------------------------------------------------
Odds
We can compute the odds using the formulas above, giving us
\[ \textrm{Odds}(\textrm{foreign} | \textrm{high mileage}) = 1 \]
and
\[ \textrm{Odds}(\textrm{foreign} | \textrm{low mileage}) = = .3333. \]
To obtain with margins
, we again pass the expression
option:
exp(xb()))
. margins highmileage, expression(
of obs = 74
Adjusted predictions Number VCE: OIM
Model
exp(xb())
Expression:
------------------------------------------------------------------------------
| Delta-methodstd. err. z P>|z| [95% conf. interval]
| Margin
-------------+----------------------------------------------------------------
highmileage |
Low Mileage | .3333333 .0993808 3.35 0.001 .1385505 .5281161
High Mile.. | 1 .5345225 1.87 0.061 -.0476448 2.047645 ------------------------------------------------------------------------------
Note that we do not want to test if the odds are different using pwcompare
as that’s what the odds ratio is for!
Odds ratio
The odds ratio is often very confusing to interpret, but is straightforward: An odds ratio predicts the number of positive outcomes we expect to see for every negative outcome. So an odds ratio of 2 would mean for every domestic car, we’d expect to see 2 foreign cars. An odds ratio of .25 would mean for every domestic car, we’d expect .25 foreign cars - or, for every 4 domestic cars, we’d expect 1 foreign car (since .25 = 1/4).
The odds ratio is literally the ratio of the odds.
\[ \textrm{OR}(\textrm{foreign} | \textrm{high mileage}) = \frac{\textrm{odds}(\textrm{foreign} | \textrm{high mileage})}{\textrm{odds}(\textrm{foreign} | \textrm{low mileage})} = 1/.333 = 3 \]
Looking at the regression results again:
logit, or
.
of obs = 74
Logistic regression Number chi2(1) = 3.18
LR chi2 = 0.0746
Prob >
Log likelihood = -43.444169 Pseudo R2 = 0.0353
------------------------------------------------------------------------------ratio Std. err. z P>|z| [95% conf. interval]
foreign | Odds
-------------+----------------------------------------------------------------
highmileage |
High Mile.. | 3 1.836145 1.79 0.073 .9039507 9.956295_cons | .3333333 .0993808 -3.68 0.000 .185823 .5979406
------------------------------------------------------------------------------_cons estimates baseline odds. Note:
The intercept is \(\textrm{odds}(\textrm{foreign} | \textrm{low mileage})\), the odds of a positive outcome in the baseline group, and the coefficient on highmileage
is the odds ratio!
Note that we cannot use the margins
command to obtain the odds ratio2. Instead, we use lincom
:
lincom _b[1.highmileage], or
.
( 1) [foreign]1.highmileage = 0
------------------------------------------------------------------------------ratio Std. err. z P>|z| [95% conf. interval]
foreign | Odds
-------------+----------------------------------------------------------------
(1) | 3 1.836145 1.79 0.073 .9039507 9.956295 ------------------------------------------------------------------------------
(I obtained the _b[1.highmileage]
name by running logit, coeflegend
to obtain the legend.) Note the or
option, without it we obtain the log odds.
A categorical predictor
Moving from a binary predictor to a categorical predictor is fairly straightforward; instead of a single odds ratio, we have two.
gen pricecat = price < 7500
.
replace pricecat = 2 if price >= 7500 & price < 10000
. real changes made)
(5
replace pricecat = 3 if price >= 10000 & price < .
. real changes made)
(10
logit foreign i.pricecat, or nolog
.
of obs = 74
Logistic regression Number chi2(2) = 2.47
LR chi2 = 0.2905
Prob >
Log likelihood = -43.797041 Pseudo R2 = 0.0275
------------------------------------------------------------------------------ratio Std. err. z P>|z| [95% conf. interval]
foreign | Odds
-------------+----------------------------------------------------------------
pricecat |
2 | 3.705882 3.546757 1.37 0.171 .5678577 24.18487
3 | .6176471 .5195704 -0.57 0.567 .1187686 3.212026
|_cons | .4047619 .1163527 -3.15 0.002 .2304165 .7110264
------------------------------------------------------------------------------_cons estimates baseline odds. Note:
Probabilities
tab foreign pricecat, col
.
+-------------------+
| Key |
|-------------------|frequency |
|
| column percentage |
+-------------------+
| pricecat
Car origin | 1 2 3 | Total
-----------+---------------------------------+----------
Domestic | 42 2 8 | 52
| 71.19 40.00 80.00 | 70.27
-----------+---------------------------------+----------
Foreign | 17 3 2 | 22
| 28.81 60.00 20.00 | 29.73
-----------+---------------------------------+----------
Total | 59 5 10 | 74 | 100.00 100.00 100.00 | 100.00
. margins pricecat
of obs = 74
Adjusted predictions Number VCE: OIM
Model
predict()
Expression: Pr(foreign),
------------------------------------------------------------------------------
| Delta-methodstd. err. z P>|z| [95% conf. interval]
| Margin
-------------+----------------------------------------------------------------
pricecat |
1 | .2881356 .0589618 4.89 0.000 .1725725 .4036987
2 | .6 .219089 2.74 0.006 .1705934 1.029407
3 | .2 .1264911 1.58 0.114 -.047918 .447918 ------------------------------------------------------------------------------
Odds
The intercept is the odds of a foreign car in pricecat
1, or .2881/.7119 = .4048. We can obtain the odds of each pricecat
in the typical way.
exp(xb()))
. margins pricecat, expression(
of obs = 74
Adjusted predictions Number VCE: OIM
Model
exp(xb())
Expression:
------------------------------------------------------------------------------
| Delta-methodstd. err. z P>|z| [95% conf. interval]
| Margin
-------------+----------------------------------------------------------------
pricecat |
1 | .4047619 .1163527 3.48 0.001 .1767148 .632809
2 | 1.5 1.369306 1.10 0.273 -1.183791 4.183791
3 | .25 .1976424 1.26 0.206 -.1373719 .6373719 ------------------------------------------------------------------------------
Odds ratios
Finally, the two coefficients in the model are the odds ratios of being in pricecat
2 or 3 versus 1. Again we can use lincom
to obtain these.
lincom _b[2.pricecat], or
.
( 1) [foreign]2.pricecat = 0
------------------------------------------------------------------------------ratio Std. err. z P>|z| [95% conf. interval]
foreign | Odds
-------------+----------------------------------------------------------------
(1) | 3.705882 3.546757 1.37 0.171 .5678577 24.18487
------------------------------------------------------------------------------
lincom _b[3.pricecat], or
.
( 1) [foreign]3.pricecat = 0
------------------------------------------------------------------------------ratio Std. err. z P>|z| [95% conf. interval]
foreign | Odds
-------------+----------------------------------------------------------------
(1) | .6176471 .5195704 -0.57 0.567 .1187686 3.212026 ------------------------------------------------------------------------------
Note here that multiplying the odds ratios by the odds in pricecat
1 (the intercept) gives the odds in the other group. E.g. 3.705*.4047 = 1.5.
A continuous predictor
Now let’s replace the categorical predictor with a continuous one. Again, most interpretations stay the same.
logit foreign headroom, or nolog
.
of obs = 74
Logistic regression Number chi2(1) = 6.72
LR chi2 = 0.0095
Prob >
Log likelihood = -41.671379 Pseudo R2 = 0.0747
------------------------------------------------------------------------------ratio Std. err. z P>|z| [95% conf. interval]
foreign | Odds
-------------+----------------------------------------------------------------
headroom | .4304066 .1490474 -2.43 0.015 .2183295 .8484873_cons | 4.780366 4.755595 1.57 0.116 .680253 33.59324
------------------------------------------------------------------------------_cons estimates baseline odds. Note:
Now instead of talking about probability or odds in a level of a categorical predictor, it is instead at a specific level of headroom. The intercept is the odds of having a positive outcome when the headroom is identically 0, which in this case, as is often the case, is not interesting.
Probabilities
We cannot look at crosstabs as we did before the compute probabilities, but the margins command still works.
at(headroom = (2.5 5))
. margins,
of obs = 74
Adjusted predictions Number VCE: OIM
Model
predict()
Expression: Pr(foreign), _at: headroom = 2.5
1._at: headroom = 5
2.
------------------------------------------------------------------------------
| Delta-methodstd. err. z P>|z| [95% conf. interval]
| Margin
-------------+----------------------------------------------------------------_at |
1 | .3674795 .0655664 5.60 0.000 .2389717 .4959873
2 | .0659516 .050366 1.31 0.190 -.0327639 .164667 ------------------------------------------------------------------------------
These are the predicted probabilties of a positive outcome at the referenced levels of headroom
, i.e. \(P(\textrm{foreign} | \textrm{headroom} = 2.5)\) and \(P(\textrm{foreign} | \textrm{headroom} = 5)\).
Odds
We can directly compute the odds given the probabilities above, but it’s easier to continue using margins
.
at(headroom = (2.5 5)) expression(exp(xb()))
. margins,
of obs = 74
Adjusted predictions Number VCE: OIM
Model
exp(xb())
Expression: _at: headroom = 2.5
1._at: headroom = 5
2.
------------------------------------------------------------------------------
| Delta-methodstd. err. z P>|z| [95% conf. interval]
| Margin
-------------+----------------------------------------------------------------_at |
1 | .5809764 .1638824 3.55 0.000 .2597729 .9021799
2 | .0706083 .0577296 1.22 0.221 -.0425396 .1837562 ------------------------------------------------------------------------------
Odds ratio
The coefficient in the logistic regression is interpreted as the odds ratio when increasing headroom by 1. In other words, if we had a collection of cars with headroom of \(x\) and magically change their headroom to \(x + 1\), we would expect for every one additional domestic car, we’d see <<dd_display: %9.4f exp(_b[headroom])>> additional foreign cars.
We can obtain this odds ratio by again using lincom
.
lincom _b[headroom], or
.
( 1) [foreign]headroom = 0
------------------------------------------------------------------------------ratio Std. err. z P>|z| [95% conf. interval]
foreign | Odds
-------------+----------------------------------------------------------------
(1) | .4304066 .1490474 -2.43 0.015 .2183295 .8484873 ------------------------------------------------------------------------------
Interactions
Let’s consider interactions now. We’ll interact two binary variables for each.
gen highprice = price > 5000
.
label define highprice 0 "Low Price" 1 "High Price"
.
label value highprice highprice
.
logit foreign i.highprice##i.highmileage, or nolog
.
of obs = 74
Logistic regression Number chi2(3) = 10.54
LR chi2 = 0.0145
Prob >
Log likelihood = -39.763201 Pseudo R2 = 0.1170
------------------------------------------------------------------------------ratio Std. err. z P>|z| [95% conf. interval]
foreign | Odds
-------------+----------------------------------------------------------------
highprice |
High Price | 6.795455 5.54509 2.35 0.019 1.372897 33.63558
|
highmileage |
High Mile.. | 11.5 10.7684 2.61 0.009 1.83505 72.06888
|
highprice#|
highmileage |
High Price #|
High Mile.. | .1471572 .2548493 -1.11 0.269 .0049392 4.384364
|_cons | .0869565 .0641052 -3.31 0.001 .0205016 .3688215
------------------------------------------------------------------------------_cons estimates baseline odds. Note:
Probabilities
Because we have two categorical predictors, we can return to looking at crosstabs as a way of obtaining probabilities. The margins
call will also return them.
table foreign highmileage highprice
.
----------------------------------------------------
| highprice
| Low Price High Price Total
-------------------+--------------------------------
Car origin |
Domestic |
highmileage |
Low Mileage | 23 22 45
High Mileage | 6 1 7
Total | 29 23 52
Foreign |
highmileage |
Low Mileage | 2 13 15
High Mileage | 6 1 7
Total | 8 14 22
Total |
highmileage |
Low Mileage | 25 35 60
High Mileage | 12 2 14
Total | 37 37 74
----------------------------------------------------
. margins highprice#highmileage
of obs = 74
Adjusted predictions Number VCE: OIM
Model
predict()
Expression: Pr(foreign),
------------------------------------------------------------------------------
| Delta-methodstd. err. z P>|z| [95% conf. interval]
| Margin
-------------+----------------------------------------------------------------
highprice#|
highmileage |
Low Price #|
Low Mileage | .08 .0542586 1.47 0.140 -.026345 .186345
Low Price #|
High Mile.. | .5 .1443376 3.46 0.001 .2171036 .7828964
High Price #|
Low Mileage | .3714286 .0816735 4.55 0.000 .2113515 .5315056
High Price #|
High Mile.. | .5 .3535534 1.41 0.157 -.1929519 1.192952 ------------------------------------------------------------------------------
This is similar to the categorical predictor, where there are four groups. For example, low price and low mileage, 2 out of 25 cars are foreign, so the probability is 2/25 = .08.
Odds
exp(xb()))
. margins highprice#highmileage, expression(
of obs = 74
Adjusted predictions Number VCE: OIM
Model
exp(xb())
Expression:
------------------------------------------------------------------------------
| Delta-methodstd. err. z P>|z| [95% conf. interval]
| Margin
-------------+----------------------------------------------------------------
highprice#|
highmileage |
Low Price #|
Low Mileage | .0869565 .0641052 1.36 0.175 -.0386874 .2126004
Low Price #|
High Mile.. | 1 .5773503 1.73 0.083 -.1315857 2.131586
High Price #|
Low Mileage | .5909091 .2067148 2.86 0.004 .1857554 .9960628
High Price #|
High Mile.. | 1 1.414214 0.71 0.480 -1.771808 3.771808 ------------------------------------------------------------------------------
If you look at the logistic results above, the baseline categories are low mileage and low price. So, as before, the intercept is the odds of a foreign car in that subcategory, which we see here.
We do not obtain the odds for the other categories in the regression output.
Odds ratios
The odds ratios reported in the regression output only present part of the story. Let’s take a look at them again.
logit, or
.
of obs = 74
Logistic regression Number chi2(3) = 10.54
LR chi2 = 0.0145
Prob >
Log likelihood = -39.763201 Pseudo R2 = 0.1170
------------------------------------------------------------------------------ratio Std. err. z P>|z| [95% conf. interval]
foreign | Odds
-------------+----------------------------------------------------------------
highprice |
High Price | 6.795455 5.54509 2.35 0.019 1.372897 33.63558
|
highmileage |
High Mile.. | 11.5 10.7684 2.61 0.009 1.83505 72.06888
|
highprice#|
highmileage |
High Price #|
High Mile.. | .1471572 .2548493 -1.11 0.269 .0049392 4.384364
|_cons | .0869565 .0641052 -3.31 0.001 .0205016 .3688215
------------------------------------------------------------------------------_cons estimates baseline odds. Note:
The coefficient on highprice
is the odds ratio of being foreign between high price and low price cars, in the low mileage category.
\[ \frac{\textrm{OR}(\textrm{foreign}|\textrm{high price, low mileage})}{\textrm{OR}(\textrm{foreign}|\textrm{low price, low mileage})} \]
The coefficient on highmileage
is the odds ratio of being foreign between high mileage and low mileage cars, in the low price category.
\[ \frac{\textrm{OR}(\textrm{foreign}|\textrm{low price, high mileage})}{\textrm{OR}(\textrm{foreign}|\textrm{low price, low mileage})} \]
The interaction can be interpreted in one of two ways.