NEWS
phyr 1.3.0
ranef() now returns the estimates of each level of each random term (e.g. one value per species for (1 | sp)), as lme4::ranef() does, for models fitted by maximum likelihood (conditional modes, optionally with their conditional standard deviations via condVar = TRUE) and by INLA (posterior summaries).
- Breaking change: the table of variances and standard deviations of random terms that
ranef() used to return is now returned by the new VarCorr() method. Replace ranef(fit) with VarCorr(fit) in existing code that uses this table.
predict() now supports newdata, with or without random effects (re.form), for models with bayes = FALSE or TRUE. Levels of grouping variables that were not in the data used to fit the model need allow.new.levels = TRUE, and have their random effects set to zero.
pglmm() with bayes = FALSE is much faster for large data sets, with the same results: quantities that do not depend on the parameters are now calculated once instead of at every evaluation of the likelihood (models without nested terms; the observation-level term 1|obs of poisson and binomial models is no longer treated as a dense nested term), the block structure of nested terms is also used when there are non-nested terms, and the design matrix of random terms is built as a sparse matrix.
- On a Mac where R uses the reference BLAS library,
pglmm() now shows a message (once per session, for models with at least 1000 observations) with the command to switch R to Apple's Accelerate, which can make large models several times faster. Nothing is changed on the computer; turn the message off with options(phyr.blas.hint = FALSE).
- Breaking change: the fitted model no longer includes
iV (the n by n inverse of the covariance matrix), which needed a lot of memory and time for large data sets; no function of the package needs it anymore. Use the new function pglmm_iV() to calculate it from a fitted model.
- Fixed bugs that gave wrong results silently:
- a covariance matrix (not a phylogeny) in
cov_ranef that had the same names as the levels of the data, but in a different order, was not reordered; the model was then fitted with the wrong covariance matrix.
- with
cpp = TRUE, models with nested terms that are not block-diagonal (e.g. (1 | sp__@site) + (1 | sp@site__), or nested terms only with unbalanced data sorted by site) could be fitted as if they were, with results that depended on the order of the terms in the formula or of the rows of the data.
offset() in the formula was ignored when bayes = FALSE; it now gives an error.
cor_phylo(): measurement errors were assigned to the wrong species when the rows of the data were not in the same order as the tips of the phylogeny.
psv() and psr() with cpp = FALSE gave wrong results for abundances that are not larger than one (e.g. relative abundances).
pcd() and match_comm_tree() now work with a community matrix (not only a data frame).
- Fixed predicted values of gaussian models (
bayes = FALSE), which also changes fitted(), residuals(), predict() and simulate() of these models:
gaussian.pred = "nearest_node" (default) now gives the fitted values conditional on the estimated random effects (Xb + Zu), the same as lme4. It used an approximation that was not correct for models with crossed or phylogenetic random terms.
gaussian.pred = "tip_rm" now includes the fixed effects (Xb); it used to return only the leave-one-out prediction of the residuals.
pglmm_predicted_values() and predict() of models with bayes = TRUE now follow type (the default is the scale of the linear predictor, as documented and as for models with bayes = FALSE; they used to be always on the scale of the response) and re.form.
residuals(): scaled = TRUE now works for gaussian models; binomial models with more than one trial now use the proportion of successes.
simulate() with re.form = NA or ~0 now works for models with bayes = FALSE (it failed before); models with bayes = TRUE and binomial data with more than one trial are now simulated with the number of trials (it used one trial).
- Nested random terms without a covariance matrix, e.g.
(1 | sp@site), no longer fail when cov_ranef is not specified.
pglmm_compare(bayes = TRUE) now returns the INLA model (inla.model).
- New argument
sparse of pglmm() (default FALSE). With sparse = TRUE, models with bayes = FALSE are fitted with the sparse precision matrix of the phylogeny (including its internal nodes) and a sparse Cholesky decomposition, instead of the dense covariance matrix. This works for gaussian, binomial and poisson models; binomial and poisson models use the same penalized quasi-likelihood (PQL) iterations as sparse = FALSE, with the working weights changing only the values of the sparse matrix and the observation-level term 1|obs folded into the weights. The likelihood is the same, but time and memory grow about linearly with the number of species, so models with tens of thousands of species can be fitted. It works with nested terms (e.g. (1 | sp__@site)) and repulsion = TRUE, and needs phylogenies (class phylo) in cov_ranef; transformed trees (e.g. OU with a fixed alpha, Pagel's lambda) can be used. The default optimizer is then "subplex". pglmm_iV() (not needed by any function), predictions with tip_rm, and the plotting functions that need the covariance matrices are not available yet for these models. Large models with phylogenies fitted with sparse = FALSE show a one-time message suggesting sparse = TRUE (turn it off with options(phyr.sparse.hint = FALSE)).
- New argument
method of pglmm() and pglmm_compare() for binomial and poisson models with bayes = FALSE: "PQL" (default, penalized quasi-likelihood, as before) or "Laplace", the Laplace approximation of the marginal likelihood, as in lme4::glmer() (nAGQ = 1) and glmmTMB. PQL underestimates the variances of random terms (and shrinks the fixed effects) for binary data and small counts; with method = "Laplace", the estimates and the log-likelihood are the same as those of glmmTMB() with a phylogenetic covariance matrix (propto()) and of glmer() without phylogeny, so that logLik(), AIC() and pglmm_profile_LRT() use a proper approximation of the marginal likelihood. It uses the sparse precision matrices of sparse = TRUE (whatever the value of sparse; covariance matrices in cov_ranef are used through their inverse), so it is fast for large phylogenies (about 3 s for 5000 species and 25000 observations), and supports nested terms, repulsion and the observation-level term 1|obs. With REML = TRUE (the default), the fixed effects are integrated out as in glmmTMB(REML = TRUE); use REML = FALSE for maximum likelihood estimates. summary() shows which approximation was used.
simulate(re.form = NA) now works for models fitted with sparse = TRUE (except with repulsion).
repulsion = TRUE (a single value) now inverts only the phylogenies of species in nested terms (e.g. sp in (1 | sp__@site__)), not those of sites, as in Ives and Helmus (2011, Appendix A), where repulsion comes from competition among related species. Before, it also inverted the phylogeny of sites in (1 | sp@site__) and (1 | sp__@site__); to get that, give one value for each phylogeny, e.g. repulsion = c(TRUE, TRUE) for (1 | sp__@site__).
- With
bayes = TRUE and bayes_nested_matrix_as_list = TRUE, repulsion = TRUE was ignored (the covariance matrix was not inverted); fixed.
- New argument
standardize_cov of pglmm() and prep_dat_pglmm() (default TRUE, the previous behaviour). With standardize_cov = FALSE, the covariance matrices of cov_ranef are not standardized to have a determinant of one but used as given (a phylogeny gives ape::vcv(tree)), so the variances of random terms are on the scale of these matrices. It also works with sparse = TRUE.
pglmm() returns scale_factors, the mean diagonal of each standardized covariance matrix. The documentation of pglmm(), VarCorr() and summary() now explains how to extract the variances of random terms: use s2r, s2n and s2resid (not ss^2, which for gaussian models are standard deviations relative to the residual standard deviation), and multiply the variances of terms with a covariance matrix by scale_factors before comparing them with other terms or with other software, because those matrices are standardized to have a determinant of one. The example of VarCorr() that calculated the proportion of phylogenetic variance without this rescaling is corrected.
pglmm_compare() is now built on the formula interface of pglmm(): it fits (1 | sp__) with a column of species made from the row names of the data, but without the non-phylogenetic term (1 | sp) that pglmm() adds (confounded with the residual variance when there is one row per species). The estimates are the same as before. It gains the features of pglmm():
- new arguments
sparse (default FALSE; with TRUE, the sparse precision matrix of the phylogeny is used, for large phylogenies), standardize_cov (default TRUE, the previous standardization), calc.WAIC, bayes_options, posterior.sample and return.fitted.marginals, passed to pglmm().
- the fitted model now also has class
communityPGLMM (c("pglmm_compare", "communityPGLMM", "pglmm")) and all the elements returned by pglmm(), so fixef(), ranef() (one estimate per species), VarCorr(), fitted(), residuals(), predict() (with newdata, species given by the row names), simulate(), logLik(), AIC(), pglmm_profile_LRT() and pglmm_iV() work. The elements of earlier versions (phy, vcv.phy, s2n, ...) are kept; vcv.phy is NULL with sparse = TRUE.
- the conditional likelihood ratio test of phylogenetic signal mentioned in the documentation is now returned (
LRT, from pglmm_profile_LRT(fit, re.number = 1)) and printed by summary().
- the default of
calc.DIC is now TRUE, as in pglmm() (only for bayes = TRUE; the estimates do not change). add.obs.re = TRUE (the default, as documented) now adds the observation-level term 1|obs for poisson data and binomial data with more than one trial; before, the argument was ignored and the term was never added, so the phylogenetic variance also absorbed any overdispersion. This changes the estimates for these models; use add.obs.re = FALSE to get the previous ones. Gaussian and binary (0/1) models are unchanged. s2.init keeps its default of 1 (pglmm() uses 0.25 for binomial and poisson models, which gives slightly different estimates within the tolerance of the optimizer).
- rows with a missing response are dropped (with their tips) instead of failing.
pglmm() shows a message (once per session) suggesting pglmm_compare() when a phylogenetic term such as (1 | sp__) has one row per species; turn it off with options(phyr.compare.hint = FALSE).
- New
logLik() method for models fitted by pglmm() with bayes = FALSE, so that AIC() works.
pglmm_profile_LRT() failed for models with nested terms only (e.g. gaussian models); fixed.
phyr 1.2.0
- Improved speed of
pglmm().
phyr 1.1.2
- pglmm_profile_LRT() now works with guassian models.
phyr 1.1.1
- Added parallel computing support for
cor_phylo() when bootstrapping is needed.
phyr 1.1.0 (2020-12-18)
- Added a
NEWS.md file to track changes to the package.
- Change function names to follow style of tidyverse (thus a minor version bump)
- change
pglmm.compare to pglmm_compare
- change multiple functions of pglmm to follow the
_ style