Logistic Models and the Margins Command
Introduction
When looking at the results of a logistic model, there are several different measures of the relationship between the predictors and the probability of a positive outcome that can be used to interpret the model:
- The odds ratios
- The log odds
- The odds
- Marginal probablities/percentages
If you are unclear which you are looking at, confusion can abound. This is doubly-confounded in Stata (in my opinion) where certain margins commands will produce a different measure than perhaps expected.
Intercept only model
Let’s start simple and consider a model with only an intercept. We’ll use the “auto” data set, and fit a model predicting the probability of a car being foreign made.
. sysuse auto
(1978 automobile data)
. logit foreign, nolog
Logistic regression Number of obs = 74
LR chi2(0) = 0.00
Prob > chi2 = .
Log likelihood = -45.03321 Pseudo R2 = 0.0000
------------------------------------------------------------------------------
foreign | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
_cons | -.8602013 .2543331 -3.38 0.001 -1.358685 -.3617176
------------------------------------------------------------------------------
. logit, or
Logistic regression Number of obs = 74
LR chi2(0) = 0.00
Prob > chi2 = .
Log likelihood = -45.03321 Pseudo R2 = 0.0000
------------------------------------------------------------------------------
foreign | Odds Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
_cons | .4230769 .1076025 -3.38 0.001 .2569985 .696479
------------------------------------------------------------------------------The default output from the logit command are the log odds, -.86 and passing the or option gives the odds ratio of .423. The conversion between these values is straightforward:
\[ \textrm{log odds} = log(\textrm{odds}) \] \[ \textrm{odds} = \exp^{\textrm{log odds}} \]
Let’s backtrack to see how we arrive at those values.
First, let’s look at the breakdown of foreign and domestic cars.
. tab foreign
Car origin | Freq. Percent Cum.
------------+-----------------------------------
Domestic | 52 70.27 70.27
Foreign | 22 29.73 100.00
------------+-----------------------------------
Total | 74 100.00We see that \(P(\textrm{foreign})\) = .2973 and \(P(\textrm{domestic})\) = .7027. We can convert these probabilities into odds, using the formula
\[ \textrm{Odds}(\textrm{foreign}) = \frac{P(\textrm{foreign})}{1 - P(\textrm{foreign})}. \]
Therefore we can easily see that
\[ \textrm{Odds}(\textrm{foreign}) = \frac{ .2973}{1 - .2973} = .4231 \]
and
\[ \textrm{Odds}(\textrm{domestic}) = \frac{ .7027}{1 - .7027} = 2.364. \]
For completeness, to convert from odds to probability you can use
\[ P(\textrm{foreign}) = \frac{\textrm{Odds}(\textrm{foreign})}{1 + \textrm{Odds}(\textrm{foreign})}. \]
Note that the odds of a car being foreign is exactly the result we saw above from logit, or. So in a logistic model with only an intercept, the coefficient on the intercept is the odds of a positive outcome.
Rather than calculate these manually, Stata can produce these automatically.
\(P(\textrm{foreign})\):
. margins
warning: prediction constant over observations.
Predictive margins Number of obs = 74
Model VCE: OIM
Expression: Pr(foreign), predict()
------------------------------------------------------------------------------
| Delta-method
| Margin std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
_cons | .2972973 .0531331 5.60 0.000 .1931583 .4014363
------------------------------------------------------------------------------\(\textrm{Odds}(\textrm{foreign})\):
. margins, expression(exp(xb()))
warning: prediction constant over observations.
Predictive margins Number of obs = 74
Model VCE: OIM
Expression: exp(xb())
------------------------------------------------------------------------------
| Delta-method
| Margin std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
_cons | .4230769 .1076025 3.93 0.000 .21218 .6339739
------------------------------------------------------------------------------The expression(exp(xb())) is a bit odd, but the easiest way to obtain what we need. Think of it as just saying “give me the odds”.
A single binary predictor
In the intercept only example, we had no concept of an odds ratio. Let’s add a fixed effect, in this case a binary predictor, which will require interpreting an odds ratio.
. gen highmileage = mpg > 25
. label define highmileage 0 "Low Mileage" 1 "High Mileage"
. label value highmileage highmileage
. tab foreign highmileage
| highmileage
Car origin | Low Milea High Mile | Total
-----------+----------------------+----------
Domestic | 45 7 | 52
Foreign | 15 7 | 22
-----------+----------------------+----------
Total | 60 14 | 74
. logit foreign i.highmileage, nolog
Logistic regression Number of obs = 74
LR chi2(1) = 3.18
Prob > chi2 = 0.0746
Log likelihood = -43.444169 Pseudo R2 = 0.0353
------------------------------------------------------------------------------
foreign | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
highmileage |
High Mile.. | 1.098612 .6120483 1.79 0.073 -.1009804 2.298205
_cons | -1.098612 .2981424 -3.68 0.000 -1.682961 -.5142639
------------------------------------------------------------------------------
. logit, or
Logistic regression Number of obs = 74
LR chi2(1) = 3.18
Prob > chi2 = 0.0746
Log likelihood = -43.444169 Pseudo R2 = 0.0353
------------------------------------------------------------------------------
foreign | Odds ratio Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
highmileage |
High Mile.. | 3 1.836145 1.79 0.073 .9039507 9.956295
_cons | .3333333 .0993808 -3.68 0.000 .185823 .5979406
------------------------------------------------------------------------------
Note: _cons estimates baseline odds.Probabilities
Let’s look at the probabilities. Here we have conditional probabilities since we have a predictor. So we are interested in \(P(\textrm{foreign} | \textrm{high mileage})\) and \(P(\textrm{foreign} | \textrm{low mileage})\).
From the table above, we can easily compute this:
. tab foreign highmileage, col
+-------------------+
| Key |
|-------------------|
| frequency |
| column percentage |
+-------------------+
| highmileage
Car origin | Low Milea High Mile | Total
-----------+----------------------+----------
Domestic | 45 7 | 52
| 75.00 50.00 | 70.27
-----------+----------------------+----------
Foreign | 15 7 | 22
| 25.00 50.00 | 29.73
-----------+----------------------+----------
Total | 60 14 | 74
| 100.00 100.00 | 100.00 We see \(P(\textrm{foreign} | \textrm{high mileage}) = .25\) and \(P(\textrm{foreign} | \textrm{low mileage}) = .5\).
We can also obtain these via margins1:
. margins highmileage
Adjusted predictions Number of obs = 74
Model VCE: OIM
Expression: Pr(foreign), predict()
------------------------------------------------------------------------------
| Delta-method
| Margin std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
highmileage |
Low Mileage | .25 .0559017 4.47 0.000 .1404347 .3595653
High Mile.. | .5 .1336306 3.74 0.000 .2380888 .7619112
------------------------------------------------------------------------------We can also test for equality between these percentages:
. margins highmileage, pwcompare(pv)
Pairwise comparisons of adjusted predictions Number of obs = 74
Model VCE: OIM
Expression: Pr(foreign), predict()
---------------------------------------------------------------------
| Delta-method Unadjusted
| Contrast std. err. z P>|z|
-----------------------------+---------------------------------------
highmileage |
High Mileage vs Low Mileage | .25 .1448521 1.73 0.084
---------------------------------------------------------------------Odds
We can compute the odds using the formulas above, giving us
\[ \textrm{Odds}(\textrm{foreign} | \textrm{high mileage}) = 1 \]
and
\[ \textrm{Odds}(\textrm{foreign} | \textrm{low mileage}) = = .3333. \]
To obtain with margins, we again pass the expression option:
. margins highmileage, expression(exp(xb()))
Adjusted predictions Number of obs = 74
Model VCE: OIM
Expression: exp(xb())
------------------------------------------------------------------------------
| Delta-method
| Margin std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
highmileage |
Low Mileage | .3333333 .0993808 3.35 0.001 .1385505 .5281161
High Mile.. | 1 .5345225 1.87 0.061 -.0476448 2.047645
------------------------------------------------------------------------------Note that we do not want to test if the odds are different using pwcompare as that’s what the odds ratio is for!
Odds ratio
The odds ratio is often very confusing to interpret, but is straightforward: An odds ratio predicts the number of positive outcomes we expect to see for every negative outcome. So an odds ratio of 2 would mean for every domestic car, we’d expect to see 2 foreign cars. An odds ratio of .25 would mean for every domestic car, we’d expect .25 foreign cars - or, for every 4 domestic cars, we’d expect 1 foreign car (since .25 = 1/4).
The odds ratio is literally the ratio of the odds.
\[ \textrm{OR}(\textrm{foreign} | \textrm{high mileage}) = \frac{\textrm{odds}(\textrm{foreign} | \textrm{high mileage})}{\textrm{odds}(\textrm{foreign} | \textrm{low mileage})} = 1/.333 = 3 \]
Looking at the regression results again:
. logit, or
Logistic regression Number of obs = 74
LR chi2(1) = 3.18
Prob > chi2 = 0.0746
Log likelihood = -43.444169 Pseudo R2 = 0.0353
------------------------------------------------------------------------------
foreign | Odds ratio Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
highmileage |
High Mile.. | 3 1.836145 1.79 0.073 .9039507 9.956295
_cons | .3333333 .0993808 -3.68 0.000 .185823 .5979406
------------------------------------------------------------------------------
Note: _cons estimates baseline odds.The intercept is \(\textrm{odds}(\textrm{foreign} | \textrm{low mileage})\), the odds of a positive outcome in the baseline group, and the coefficient on highmileage is the odds ratio!
Note that we cannot use the margins command to obtain the odds ratio2. Instead, we use lincom:
. lincom _b[1.highmileage], or
( 1) [foreign]1.highmileage = 0
------------------------------------------------------------------------------
foreign | Odds ratio Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
(1) | 3 1.836145 1.79 0.073 .9039507 9.956295
------------------------------------------------------------------------------(I obtained the _b[1.highmileage] name by running logit, coeflegend to obtain the legend.) Note the or option, without it we obtain the log odds.
A categorical predictor
Moving from a binary predictor to a categorical predictor is fairly straightforward; instead of a single odds ratio, we have two.
. gen pricecat = price < 7500
. replace pricecat = 2 if price >= 7500 & price < 10000
(5 real changes made)
. replace pricecat = 3 if price >= 10000 & price < .
(10 real changes made)
. logit foreign i.pricecat, or nolog
Logistic regression Number of obs = 74
LR chi2(2) = 2.47
Prob > chi2 = 0.2905
Log likelihood = -43.797041 Pseudo R2 = 0.0275
------------------------------------------------------------------------------
foreign | Odds ratio Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
pricecat |
2 | 3.705882 3.546757 1.37 0.171 .5678577 24.18487
3 | .6176471 .5195704 -0.57 0.567 .1187686 3.212026
|
_cons | .4047619 .1163527 -3.15 0.002 .2304165 .7110264
------------------------------------------------------------------------------
Note: _cons estimates baseline odds.Probabilities
. tab foreign pricecat, col
+-------------------+
| Key |
|-------------------|
| frequency |
| column percentage |
+-------------------+
| pricecat
Car origin | 1 2 3 | Total
-----------+---------------------------------+----------
Domestic | 42 2 8 | 52
| 71.19 40.00 80.00 | 70.27
-----------+---------------------------------+----------
Foreign | 17 3 2 | 22
| 28.81 60.00 20.00 | 29.73
-----------+---------------------------------+----------
Total | 59 5 10 | 74
| 100.00 100.00 100.00 | 100.00 . margins pricecat
Adjusted predictions Number of obs = 74
Model VCE: OIM
Expression: Pr(foreign), predict()
------------------------------------------------------------------------------
| Delta-method
| Margin std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
pricecat |
1 | .2881356 .0589618 4.89 0.000 .1725725 .4036987
2 | .6 .219089 2.74 0.006 .1705934 1.029407
3 | .2 .1264911 1.58 0.114 -.047918 .447918
------------------------------------------------------------------------------Odds
The intercept is the odds of a foreign car in pricecat 1, or .2881/.7119 = .4048. We can obtain the odds of each pricecat in the typical way.
. margins pricecat, expression(exp(xb()))
Adjusted predictions Number of obs = 74
Model VCE: OIM
Expression: exp(xb())
------------------------------------------------------------------------------
| Delta-method
| Margin std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
pricecat |
1 | .4047619 .1163527 3.48 0.001 .1767148 .632809
2 | 1.5 1.369306 1.10 0.273 -1.183791 4.183791
3 | .25 .1976424 1.26 0.206 -.1373719 .6373719
------------------------------------------------------------------------------Odds ratios
Finally, the two coefficients in the model are the odds ratios of being in pricecat 2 or 3 versus 1. Again we can use lincom to obtain these.
. lincom _b[2.pricecat], or
( 1) [foreign]2.pricecat = 0
------------------------------------------------------------------------------
foreign | Odds ratio Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
(1) | 3.705882 3.546757 1.37 0.171 .5678577 24.18487
------------------------------------------------------------------------------
. lincom _b[3.pricecat], or
( 1) [foreign]3.pricecat = 0
------------------------------------------------------------------------------
foreign | Odds ratio Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
(1) | .6176471 .5195704 -0.57 0.567 .1187686 3.212026
------------------------------------------------------------------------------Note here that multiplying the odds ratios by the odds in pricecat 1 (the intercept) gives the odds in the other group. E.g. 3.705*.4047 = 1.5.
A continuous predictor
Now let’s replace the categorical predictor with a continuous one. Again, most interpretations stay the same.
. logit foreign headroom, or nolog
Logistic regression Number of obs = 74
LR chi2(1) = 6.72
Prob > chi2 = 0.0095
Log likelihood = -41.671379 Pseudo R2 = 0.0747
------------------------------------------------------------------------------
foreign | Odds ratio Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
headroom | .4304066 .1490474 -2.43 0.015 .2183295 .8484873
_cons | 4.780366 4.755595 1.57 0.116 .680253 33.59324
------------------------------------------------------------------------------
Note: _cons estimates baseline odds.Now instead of talking about probability or odds in a level of a categorical predictor, it is instead at a specific level of headroom. The intercept is the odds of having a positive outcome when the headroom is identically 0, which in this case, as is often the case, is not interesting.
Probabilities
We cannot look at crosstabs as we did before the compute probabilities, but the margins command still works.
. margins, at(headroom = (2.5 5))
Adjusted predictions Number of obs = 74
Model VCE: OIM
Expression: Pr(foreign), predict()
1._at: headroom = 2.5
2._at: headroom = 5
------------------------------------------------------------------------------
| Delta-method
| Margin std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
_at |
1 | .3674795 .0655664 5.60 0.000 .2389717 .4959873
2 | .0659516 .050366 1.31 0.190 -.0327639 .164667
------------------------------------------------------------------------------These are the predicted probabilties of a positive outcome at the referenced levels of headroom, i.e. \(P(\textrm{foreign} | \textrm{headroom} = 2.5)\) and \(P(\textrm{foreign} | \textrm{headroom} = 5)\).
Odds
We can directly compute the odds given the probabilities above, but it’s easier to continue using margins.
. margins, at(headroom = (2.5 5)) expression(exp(xb()))
Adjusted predictions Number of obs = 74
Model VCE: OIM
Expression: exp(xb())
1._at: headroom = 2.5
2._at: headroom = 5
------------------------------------------------------------------------------
| Delta-method
| Margin std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
_at |
1 | .5809764 .1638824 3.55 0.000 .2597729 .9021799
2 | .0706083 .0577296 1.22 0.221 -.0425396 .1837562
------------------------------------------------------------------------------Odds ratio
The coefficient in the logistic regression is interpreted as the odds ratio when increasing headroom by 1. In other words, if we had a collection of cars with headroom of \(x\) and magically change their headroom to \(x + 1\), we would expect for every one additional domestic car, we’d see <<dd_display: %9.4f exp(_b[headroom])>> additional foreign cars.
We can obtain this odds ratio by again using lincom.
. lincom _b[headroom], or
( 1) [foreign]headroom = 0
------------------------------------------------------------------------------
foreign | Odds ratio Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
(1) | .4304066 .1490474 -2.43 0.015 .2183295 .8484873
------------------------------------------------------------------------------Interactions
Let’s consider interactions now. We’ll interact two binary variables for each.
. gen highprice = price > 5000
. label define highprice 0 "Low Price" 1 "High Price"
. label value highprice highprice
. logit foreign i.highprice##i.highmileage, or nolog
Logistic regression Number of obs = 74
LR chi2(3) = 10.54
Prob > chi2 = 0.0145
Log likelihood = -39.763201 Pseudo R2 = 0.1170
------------------------------------------------------------------------------
foreign | Odds ratio Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
highprice |
High Price | 6.795455 5.54509 2.35 0.019 1.372897 33.63558
|
highmileage |
High Mile.. | 11.5 10.7684 2.61 0.009 1.83505 72.06888
|
highprice#|
highmileage |
High Price #|
High Mile.. | .1471572 .2548493 -1.11 0.269 .0049392 4.384364
|
_cons | .0869565 .0641052 -3.31 0.001 .0205016 .3688215
------------------------------------------------------------------------------
Note: _cons estimates baseline odds.Probabilities
Because we have two categorical predictors, we can return to looking at crosstabs as a way of obtaining probabilities. The margins call will also return them.
. table foreign highmileage highprice
----------------------------------------------------
| highprice
| Low Price High Price Total
-------------------+--------------------------------
Car origin |
Domestic |
highmileage |
Low Mileage | 23 22 45
High Mileage | 6 1 7
Total | 29 23 52
Foreign |
highmileage |
Low Mileage | 2 13 15
High Mileage | 6 1 7
Total | 8 14 22
Total |
highmileage |
Low Mileage | 25 35 60
High Mileage | 12 2 14
Total | 37 37 74
----------------------------------------------------
. margins highprice#highmileage
Adjusted predictions Number of obs = 74
Model VCE: OIM
Expression: Pr(foreign), predict()
------------------------------------------------------------------------------
| Delta-method
| Margin std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
highprice#|
highmileage |
Low Price #|
Low Mileage | .08 .0542586 1.47 0.140 -.026345 .186345
Low Price #|
High Mile.. | .5 .1443376 3.46 0.001 .2171036 .7828964
High Price #|
Low Mileage | .3714286 .0816735 4.55 0.000 .2113515 .5315056
High Price #|
High Mile.. | .5 .3535534 1.41 0.157 -.1929519 1.192952
------------------------------------------------------------------------------This is similar to the categorical predictor, where there are four groups. For example, low price and low mileage, 2 out of 25 cars are foreign, so the probability is 2/25 = .08.
Odds
. margins highprice#highmileage, expression(exp(xb()))
Adjusted predictions Number of obs = 74
Model VCE: OIM
Expression: exp(xb())
------------------------------------------------------------------------------
| Delta-method
| Margin std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
highprice#|
highmileage |
Low Price #|
Low Mileage | .0869565 .0641052 1.36 0.175 -.0386874 .2126004
Low Price #|
High Mile.. | 1 .5773503 1.73 0.083 -.1315857 2.131586
High Price #|
Low Mileage | .5909091 .2067148 2.86 0.004 .1857554 .9960628
High Price #|
High Mile.. | 1 1.414214 0.71 0.480 -1.771808 3.771808
------------------------------------------------------------------------------If you look at the logistic results above, the baseline categories are low mileage and low price. So, as before, the intercept is the odds of a foreign car in that subcategory, which we see here.
We do not obtain the odds for the other categories in the regression output.
Odds ratios
The odds ratios reported in the regression output only present part of the story. Let’s take a look at them again.
. logit, or
Logistic regression Number of obs = 74
LR chi2(3) = 10.54
Prob > chi2 = 0.0145
Log likelihood = -39.763201 Pseudo R2 = 0.1170
------------------------------------------------------------------------------
foreign | Odds ratio Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
highprice |
High Price | 6.795455 5.54509 2.35 0.019 1.372897 33.63558
|
highmileage |
High Mile.. | 11.5 10.7684 2.61 0.009 1.83505 72.06888
|
highprice#|
highmileage |
High Price #|
High Mile.. | .1471572 .2548493 -1.11 0.269 .0049392 4.384364
|
_cons | .0869565 .0641052 -3.31 0.001 .0205016 .3688215
------------------------------------------------------------------------------
Note: _cons estimates baseline odds.The coefficient on highprice is the odds ratio of being foreign between high price and low price cars, in the low mileage category.
\[ \frac{\textrm{OR}(\textrm{foreign}|\textrm{high price, low mileage})}{\textrm{OR}(\textrm{foreign}|\textrm{low price, low mileage})} \]
The coefficient on highmileage is the odds ratio of being foreign between high mileage and low mileage cars, in the low price category.
\[ \frac{\textrm{OR}(\textrm{foreign}|\textrm{low price, high mileage})}{\textrm{OR}(\textrm{foreign}|\textrm{low price, low mileage})} \]
The interaction can be interpreted in one of two ways.