|Title||Why am I getting the error message “confidence intervals not symmetric” after I run meta set?|
|Author||Dan (Mia) Lv, StataCorp|
meta set has two main syntax specifications. One is for specifying generic effect sizes with their standard errors, and the other one is for specifying generic effect sizes with their confidence intervals. When the second syntax is used, you may encounter the error of “confidence intervals not symmetric”.
Consider the following fictional meta-analysis dataset, where es is the effect size and cil and ciu are the confidence interval lower point and upper point, respectively.
clear input es cil ciu 1.48 -.35 3.31 1 -.93 2.93 1.27 .43 2.12 1 .75 1.25 1.18 -.53 2.88 1.94 .43 3.45 2.38 1.01 3.75 .69 -.57 1.96 1.1 -.15 2.34 1.81 -.15 3.76 end
Running meta set gives you the following error:
. meta set es cil ciu confidence intervals not symmetric CIs defined by variables cil and ciu must be symmetric and based on a normal distribution. If you are working with effect sizes in the original metric, such as odds ratios or hazard ratios, with meta set, you should specify the effect sizes and CIs in a normalizing metric, such as the log metric. The default tolerance to determine the CI asymmetry is 1e-6. Effect sizes and their CIs are often reported with limited precision that, after the normalizing transformation, may lead to asymmetric CIs. In that case, the default of 1e-6 may be too stringent. You may loosen the tolerance by specifying option civartolerance(). r(459);
This error indicates that the confidence interval we specified is not symmetric around the effect size for at least one study in the dataset. As explained in the error message, meta set assumes that confidence intervals for effect sizes are based on the normal distribution, and therefore they should be symmetric. Mathematically, the effect sizes and symmetric confidence intervals should satisfy
CI upper limit – effect size = effect size – CI lower limit
There are two situations that may produce asymmetric CIs, leading meta set to issue an error.
In the first situation, confidence intervals for the effect sizes are indeed based on the normal distribution, but they are recorded with less precision than meta set expects. In this situation, confidence intervals will look roughly symmetric around the effect sizes. For example, you might have collected data from a publication that only gives you two decimal digits, and therefore they might not pass the command's symmetry check. By default, the symmetry check is performed with a tolerance level of 1e-6. That means the largest relative difference between CI upper limit – effect size and effect size – CI lower limit cannot be greater than 1e-6. (The relative difference between \(x\) and \(y\) is defined as |\(x\)–\(y\)|/(|\(y\)|+1).)
Let’s examine our data for this purpose.
. generate double relative_diff = reldif(es-cil, ciu-es) . summarize relative_diff
|Variable||Obs Mean Std. dev. Min Max|
|relative_d~f||10 .0021369 .0023118 0 .0054054|
We can see the largest relative difference is about .0054, which exceeds 1e-6; that is why the symmetry check failed. Because the confidence intervals in this data are roughly symmetric around effect sizes, we know this is the first situation. In this case, you could consider using the option civartolerance() with meta set to relax the symmetry check for confidence intervals. We already mentioned that the default setting for this check is civartolerance(1e-6). We could change that number to be slightly bigger than the largest relative difference we obtain above. For example, we can use
. meta set es cil ciu,civartolerance(0.0055)
Here we specify 0.0055 in civartolerance(), and this time the symmetry check is passed.
When you specify civartolerance() with meta set, Stata will obtain a symmetric confidence interval based on the old confidence interval width for each study in your original data. The new confidence intervals are saved in the variables _meta_cil and _meta_ciu, and they will be used throughout the meta-analysis (which explains why you may get a forest plot with confidence intervals that slightly differ from the original ones in your dataset).
The new confidence intervals satisfy
_meta_ciu – _meta_cil = ciu – cil (The CI width remains the same)
_meta_ciu – es = es – meta_cil (symmetric CI)
In the second situation, the effect sizes are in a metric for which normality is not expected, and the confidence intervals are not close to symmetric. You may find the dataset requires specifying a relatively big number in civartolerance() in order to run meta set without error. This is usually indicative of potential problems beyond the number of significant digits reported. Perhaps typos were introduced when collecting the data, or perhaps the data are not reported in a normalizing metric and you may need to apply a normalizing transformation to obtain symmetric intervals. For example, if your effect sizes are risk ratios (RRs), odds ratios (ORs), incidence-rate ratios (IRRs), or hazard ratios (HRs), you should apply a logarithmic transformation to normalize your data.
For more details on this process, please see How do I meta set my data when the effect sizes are risk ratios (RRs), odds ratios (ORs), incidence-rate ratios (IRRs), or hazard ratios (HRs) with their confidence intervals?.