Detrending & standardization¶

Removing the biological growth trend to produce ring-width index (RWI) series. See the Detrending & standardization guide for when to use each method.

detrend¶

Detrends a given series or dataframe

Extended Summary

Detrends a given series or dataframe, first by fitting data to a growth curve (a smoothing spline by default), and then by forming the ring-width index as a ratio ('ratio', i.e. division -- the default) or a difference ('difference', i.e. subtraction) of the data to the fitted curve.

IMPORTANT -- fit chooses the CURVE, method chooses the ARITHMETIC. This differs from dplR, where method selects the curve and difference selects the arithmetic. In dplPy the curve type is fit= and the ratio/difference choice is method=; passing a curve name to method= raises a helpful error.

Parameters:

Name	Type	Description	Default
`data`	`DataFrame \| Series`	a data frame loaded using dpl.readers(), or a series extracted from such a datafame.	required
`fit`		the growth curve to fit. Accepted values (case-insensitive; dplR's names are used as the canonical spelling): 'Spline' (smoothing spline), 'AgeDepSpline' (age-dependent spline), 'ModNegExp' (modified negative exponential fit by nonlinear least squares, dplR-style, with a linear->mean fallback), 'NegExp' (the SAME neg-exp curve fit by Ed Cook's ARSTAN deterministic method -- a 1-D search for the decay rate with the amplitude/asymptote in closed form. It minimises the same data-space error as 'ModNegExp', so the two coincide when the nls converges; its advantage is robustness -- being deterministic it never fails to converge, so it still fits series where 'ModNegExp' would fall back to a line/mean. Same linear->mean fallback for the rejected cases), 'ModHugershoff' (Hugershoff growth curve fit by nonlinear least squares, dplR-style, with a linear->mean fallback), 'Hugershoff' (the SAME curve fit by ARSTAN's log-linearised closed form -- here the two genuinely differ, as the closed form optimises log-space rather than data-space error), 'GeneralExp' (the general exponential atexp(b*t) -- Hugershoff with the power fixed at 1, ARSTAN's menu option 8, deterministic), 'Mean' (horizontal line at the series mean), 'LinearAny' (best-fit straight line, any slope; ARSTAN option 4), and 'LinearNegative' (best-fit line constrained to a non-positive slope, falling back to the mean if the slope is positive; ARSTAN option 5). NegExp/ModNegExp and Hugershoff/ModHugershoff each pair ARSTAN's deterministic fit with dplR's nls fit of the same curve. Names are case-insensitive; legacy spellings ('ModNegEx', 'horizontal', and 'Linear' as an alias of 'LinearAny') are also accepted. `fit` may also be a LIST of curve types (e.g. ['Spline', 'ModNegExp']) to detrend by each and compare them (mirrors dplR's method vector). In that case a Series returns a DataFrame with one column per method, and a DataFrame returns a dict mapping each series name to such a DataFrame. (A list of methods cannot be combined with return_info=True.)	`'Spline'`
`method`	`str`	how the ring-width index is formed from the data and the fitted curve. 'ratio' (equivalently 'division') divides the data by the curve; 'difference' subtracts the curve from the data. This mirrors dplR's detrend(), where division is the default and 'difference=TRUE' subtracts. The former spelling 'residual' for the division result is deprecated -- in dplPy 'residual' now refers only to the residual (AR-prewhitened) chronology -- and is accepted for now as an alias of 'ratio' with a warning.	`'ratio'`
`plot`	`boolean`	flag indicating whether or not to plot the results.	`True`
`nyrs0`	`int`	initial spline stiffness for fit='AgeDepSpline' (see dpl.ads()).	`50`
`pos_slope`	`boolean`	for fit='ModNegExp' and fit='AgeDepSpline', whether to allow a positive slope (in the ModNegExp linear fallback, and at the end of the age- dependent spline). dplR's detrend uses pos.slope=FALSE.	`False`
`f`	`float`	frequency-response amplitude of the smoothing spline at the `period` wavelength (dplR's `f`): the fraction of amplitude retained there. Only used for fit='Spline'.	`0.5`
`verbose`	`boolean`	print, per series, the curve fitted and the detrending arithmetic used. (Fallbacks in fit='ModNegExp' are also reported via warnings.)	`False`
`return_info`	`boolean`	if True, return a dict (mirroring dplR's detrend return.info) instead of just the detrended data:: `{"rwi": detrended series (same shape as the input), "curves": the fitted detrending curves, "model_info": per-series fit description -- the method actually used (e.g. 'NegativeExponential', 'Line', 'Mean', 'Spline') and its parameters/coefficients, "data_info": per-series {n_zeros, zero_years} of the input, "dirty_dog": True if any series' curve went non-positive or fell back to the series mean (dplR's "dirty dog" flag)}`	`False`

Returns:

Name	Type	Description
`data`	`pandas dataframe or series of detrended data.`

Examples:

>>> import dplpy as dpl
>>> data = dpl.readers("../tests/data/csv/file.csv")
>>> dpl.detrend(data) # Detrends all series in a dataframe
>>> dpl.detrend(data["SeriesA"]) # Detrends only SeriesA
>>> dpl.detrend(data["SeriesA"], fit="ModNegExp", method="difference", plot=True)

References

.. [1] https:/opendendro.org/dplpy-man/#detrend

rcs¶

Regional Curve Standardisation of a set of ring-width series.

Extended Summary

Detrends by a single "regional curve": every series is placed on a common cambial-age axis according to its pith offset, the average ring width at each cambial age is taken (the regional curve), that curve is smoothed, and each ring is divided by (ratios) or has subtracted (difference) the curve value at its cambial age. This preserves low-frequency growth signal that per-series detrending would remove. A port of dplR's rcs().

Parameters:

Name	Type	Description	Default
`rwl`	`DataFrame`	ring-width series, years as the index and series as columns.	required
`po`	`DataFrame or None`	pith offsets, with columns `series` and `pith_offset` (the cambial age of each series' first measured ring). None assumes a pith offset of 1 for every series.	`None`
`nyrs`	`int or None`	smoothing stiffness for the regional curve. None uses floor(0.1 * curve length) for method='caps' or 50 for method='ads'.	`None`
`f`	`float`	frequency-response amplitude for method='caps'.	`0.5`
`biweight`	`bool`	build the regional curve with Tukey's biweight robust mean (C=9) rather than the arithmetic mean.	`True`
`ratios`	`bool`	detrend by division (ratios); if False, by subtraction (difference).	`True`
`rc_out`	`bool`	if True, return a dict with both the detrended `rwi` and the regional curve `rc`; otherwise return just `rwi`.	`False`
`make_plot`	`bool`	draw the regional-curve figure (series and curve vs cambial age).	`True`
`method`	`(caps, ads)`	smooth the regional curve with a fixed-stiffness spline (caps) or an age-dependent spline (ads).	`"caps"`
`min_n`	`int or None`	truncate the regional-curve tail where the by-age sample depth falls below this.	`None`
`pos_slope`	`bool`	passed to ads when method='ads'.	`True`
`preset`	`(None, crust)`	None follows dplR's rcs() (the parameters above apply). "crust" builds the regional curve the CRUST way (Melvin & Briffa 2014): an age-dependent spline smoothed only where sample depth >= 4 with the "no rise in the final third" tail rule, gaps in the curve infilled, a 0.02 mm minimum-value floor, and flat extension of the tail. Robust for the heterogeneous / relict data RCS is typically applied to. When set, `nyrs` (if given) overrides the age-spline offset (default 10) and `method`/`f`/`pos_slope` are ignored.	`None`

Returns:

Type	Description
`DataFrame or dict`	the detrended `rwi` (calendar-year indexed), or `{"rwi", "rc"}` when `rc_out` is True.

References

.. [1] https://rdrr.io/cran/dplR/man/rcs.html .. [2] Melvin, T. M. & Briffa, K. R. (2014) CRUST: Software for the implementation of Regional Chronology Standardisation: Part 1, Signal-Free RCS. Dendrochronologia, 32, 7-20. .. [3] Melvin, T. M. & Briffa, K. R. (2014) CRUST: Software for the implementation of Regional Chronology Standardisation: Part 2, Further RCS options and recommendations. Dendrochronologia, 32, 343-356.

sfrcs¶

Signal-Free Regional Curve Standardisation of a set of ring-width series.

Extended Summary

Ordinary RCS builds one "regional curve" (mean growth by cambial age) and detrends every series by it. When the sample's age structure varies over calendar time -- young trees clustered in one period, old trees in another -- that age curve absorbs some of the common climate signal, biasing the low frequencies ("trend distortion"). Signal-Free RCS breaks the feedback: it iterates, and on each pass rebuilds the regional curve from measurements that have had the current chronology (the common signal) divided out, while still forming the tree indices from the original measurements. A port of CRUST's single-curve signal-free RCS (Melvin & Briffa 2014).

Each iteration (CRUST rcs_detrend/det_sf_rcs): 1. signal-free measurements fx = tx / crn -- but only where the calendar year holds more than one tree and crn >= 0.01; elsewhere the raw tx is kept (the near-zero/low-depth division guard); 2. regional curve = mean ring width by cambial age of fx, smoothed with the CRUST age-dependent spline (spline3, 11-year minimum stiffness, fitted only where by-age depth >= 4, 0.02 mm floor, flat-extended tail); 3. tree indices = original tx divided by that curve; 4. rescale so the chronology mean is 1; 5. recompute the chronology (robust biweight mean by year). Iteration stops when max|crn_k - crn_(k-1)| < tol.

Parameters:

Name	Type	Description	Default
`rwl`	`DataFrame`	ring-width series, years as the index and series as columns.	required
`po`	`DataFrame or None`	pith offsets, columns `series` and `pith_offset` (cambial age of each series' first measured ring). None assumes a pith offset of 1 for all.	`None`
`ratios`	`bool`	form indices by division (ratios); if False, by subtraction (difference).	`True`
`biweight_curve`	`bool`	build the regional curve with Tukey's biweight robust mean by cambial age. Default False matches CRUST, which uses the arithmetic mean by age.	`False`
`biweight_crn`	`bool`	how the chronology mean by year is formed. False (default) uses the arithmetic mean, matching CRUST's `RCSdefault.fil` (KRB=1) -- this reproduces a headless build of CRUST to ~1e-3 on CRUST's own sample data. True uses Tukey's biweight robust mean (a robust alternative that departs from CRUST's default).	`False`
`ss`	`int`	age-dependent spline stiffness offset for the regional curve; per-point stiffness is `ss + cambial age` (ss=10 -> 11-year minimum), as in CRUST.	`10`
`rise`	`bool or None`	allow the smoothed regional curve to rise in its final third. None selects CRUST's rule automatically (allowed only for curves longer than 1500 years); otherwise the tail is flattened from its minimum to prevent an artefactual old-age upturn.	`None`
`max_iterations`	`int`	maximum signal-free iterations (CRUST's budget).	`40`
`tol`	`float`	convergence threshold on the maximum absolute year-to-year change in the chronology between successive iterations (CRUST uses 0.001).	`1e-3`
`make_plot`	`bool`	draw the regional-curve figure (series and final curve vs cambial age).	`True`
`return_info`	`bool`	if True, return a dict of diagnostics (see Returns) instead of just `rwi`.	`False`
`verbose`	`bool`	print a convergence line.	`True`

Returns:

Type Description

DataFrame or dict

By default the detrended rwi (calendar-year indexed, one column per series). With return_info=True, a dict::

{"rwi":        detrended series (calendar x series),
 "sfc":        the signal-free chronology (Series + samp.depth),
 "samp_depth": number of trees per calendar year,
 "conv":       max absolute chronology change at each iteration,
 "n_iter":     iterations run,
 "converged":  whether tol was reached within max_iterations,
 "rc":         the final regional curve (by cambial age)}

Notes

This is the single-curve case (CRUST trc=1, src=1). Multi-curve RCS (several regional curves with tree-to-curve allocation) is not yet implemented. Unlike dplPy's rcs and ssf, there is no dplR reference for this method: the port is faithful to CRUST's stand.f90 and its numerical core (the spline3 smoother) is validated against CRUST's compiled kernel, but the full iteration is not checked against a gold standard. Treat results as CRUST-faithful rather than machine-verified.

References

.. [1] Melvin, T. M. & Briffa, K. R. (2014) CRUST: Software for the implementation of Regional Chronology Standardisation: Part 1, Signal-Free RCS. Dendrochronologia, 32, 7-20. .. [2] Melvin, T. M. & Briffa, K. R. (2014) CRUST: Software for the implementation of Regional Chronology Standardisation: Part 2, Further RCS options and recommendations. Dendrochronologia, 32, 343-356.

ssf¶

Simple signal-free chronology (dplR's ssf()).

Extended Summary

Builds a signal-free chronology (Melvin & Briffa 2008): an initial chronology is formed, every series is divided by it to strip the common signal, the resulting "signal-free" measurements are rescaled and re-detrended, and a new chronology is built. This repeats until the high-frequency component of the chronology stops changing between iterations (the sample-depth-weighted median absolute difference falls below mad_threshold) or max_iterations is reached. Reproduces dplR's ssf().

Parameters:

Name	Type	Description	Default
`rwl`	`DataFrame`	ring-width series (raw), years as the index and series as columns.	required
`method`	`(Spline, AgeDepSpline)`	detrending curve used at each iteration: a cubic smoothing spline (⅔-length stiffness) or an age-dependent spline (dpl.ads()).	`"Spline"`
`nyrs`	`int or None`	spline stiffness. None uses ⅔ of each series' length for "Spline" or 50 for "AgeDepSpline"; a value in (0, 1) is a fraction of series length.	`None`
`difference`	`bool`	detrend by subtraction (series - curve) rather than division.	`False`
`max_iterations`	`int`	maximum signal-free iterations before giving up.	`25`
`mad_threshold`	`float`	convergence threshold on the median absolute difference of the high-frequency chronology between successive iterations.	`5e-4`
`recode_zeros`	`bool`	recode zero ring-widths to 0.001 before processing (avoids div-by-zero).	`False`
`return_info`	`bool`	if True, return a dict of the full iteration history (chronologies, signal-free measurements and curves, MAD vector, ...) instead of just the final chronology.	`False`
`verbose`	`bool`	print progress and the per-iteration convergence diagnostics.	`True`
`preset`	`(None, crust)`	None reproduces the basic signal-free method (dplR's ssf; Melvin & Briffa 2008). "crust" turns on the guards/refinements from CRUST (Melvin & Briffa 2014): the signal-free measurements are rescaled multiplicatively (the model-consistent "mean offset") rather than additively; the raw measurement is kept in any year where the chronology is near zero (< 0.01); and the fitted detrending curves are floored at 0.02 mm. These make ssf robust to sensitive sites with many absent rings (e.g. co021), which the basic method cannot standardise.	`None`

Returns:

Type	Description
`DataFrame`	the signal-free chronology with columns `sfc` and `samp_depth`, indexed by year. If `return_info` is True, a dict of intermediates is returned instead (see above).

References

.. [1] https://rdrr.io/cran/dplR/man/ssf.html .. [2] Melvin, T. M. & Briffa, K. R. (2008) A "signal-free" approach to dendroclimatic standardisation. Dendrochronologia, 26(2), 71-86. .. [3] Melvin, T. M. & Briffa, K. R. (2014) CRUST: Software for the implementation of Regional Chronology Standardisation: Part 1, Signal-Free RCS. Dendrochronologia, 32, 7-20. .. [4] Melvin, T. M. & Briffa, K. R. (2014) CRUST: Software for the implementation of Regional Chronology Standardisation: Part 2, Further RCS options and recommendations. Dendrochronologia, 32, 343-356.

ads¶

Age-dependent smoothing spline of a single series (dplR's ads()).

Extended Summary

Fits a smoothing spline whose stiffness increases with cambial age: the i-th ring is fit with an (nyrs0 + i - 1)-year spline, so young rings are smoothed least and the curve grows stiffer with age (Melvin 2004; Melvin et al. 2007). The underlying cubic smoothing spline follows Cook & Peters (1981) with a 50% frequency cutoff. Ports dplR's ads() (the same banded system, solved with SciPy).

Parameters:

Name	Type	Description	Default
`y`	`array - like`	a single ring-width series (no missing values), oldest ring first.	required
`nyrs0`	`int`	initial spline stiffness (wavelength, in years) at the first ring.	`50`
`pos_slope`	`bool`	if False, once the spline stops decreasing its tail is held flat and the spline is refit -- preventing an artefactual upturn at the end of the series (dplR uses pos.slope=FALSE inside detrend()).	`True`

Returns:

Type	Description
`ndarray`	the fitted spline curve, same length as `y`.

Examples:

>>> import dplpy as dpl
>>> rwl = dpl.readers("../tests/data/csv/ca533.csv")
>>> curve = dpl.ads(rwl["CAM011"].dropna().to_numpy(), nyrs0=50)

References

.. [1] https://rdrr.io/cran/dplR/man/ads.html .. [2] Melvin, T. M. (2004) Historical Growth Rates and Changing Climatic Sensitivity of Boreal Conifers. PhD Thesis, Climatic Research Unit, University of East Anglia. .. [3] Melvin, T. M., Briffa, K. R., Nicolussi, K. & Grabner, M. (2007) Time-varying-response smoothing. Dendrochronologia, 25(2), 65-69. .. [4] Cook, E. R. & Peters, K. (1981) The Smoothing Spline: A New Approach to Standardizing Forest Interior Tree-Ring Width Series for Dendroclimatic Studies. Tree-Ring Bulletin, 41, 45-53.

powt¶

Adaptive power transformation of ring-width series (variance stabilising).

Extended Summary

Stabilises the variance of ring-width series by raising each value to a data driven power p. The power comes from the empirical relationship between local level and local spread: for adjacent rings, level M = (x[t] + x[t-1]) / 2 and spread S = |x[t] - x[t-1]|; regressing log(S) on log(M) gives a slope b, and p = |1 - b| removes the level-spread dependence. This is ARSTAN's fourth data transformation option (Cook & Peters 1997) and a port of dplR's powt().

Two methods are offered:

"cook" (default) estimates p independently for each series.
"universal" estimates a single p for the whole data set with a linear mixed-effects model (log(S) ~ log(M) with a random intercept per year), matching dplR's default. Requires statsmodels; if the estimate is p <= 0 a log transform is used instead. Not part of ARSTAN.

Parameters:

Name	Type	Description	Default
`rwl`	`DataFrame or Series`	ring-width series (raw, non-negative), years as the index. A Series is allowed only with method='cook'.	required
`method`	`(cook, universal)`	per-series ('cook') or single mixed-model power ('universal').	`"cook"`
`rescale`	`bool`	if True, rescale each transformed series back to its original mean and standard deviation.	`False`
`return_power`	`bool`	if True, return a (transformed, power) tuple instead of just the transformed data; `power` is a Series over series for 'cook' or a single float for 'universal'.	`False`
`plot`	`bool`	if True, draw the "data transform statistics" diagnostic -- a 2x2 figure of per-series skew and spread-vs-level correlation, before (top row) and after (bottom row) the transform, with median and quartile reference lines. A successful transform drives the bottom-row bars toward zero (series made symmetric and variance decoupled from level). This emulates ARSTAN's power-transform diagnostic (dplR's powt has no such plot).	`False`
`return_stats`	`bool`	if True, also return a per-series diagnostics DataFrame with columns `skew_before`, `skew_after`, `r_before`, `r_after` (the spread-vs-level correlations) and `power`. The spread-vs-level correlation uses dplPy's mean-based level/spread (the same used to fit the power), which differs slightly from ARSTAN's trailing-ring level.	`False`

Returns:

Type	Description
transformed data of the same type as ``rwl``. With ``return_power`` and/or
``return_stats`` the return becomes a tuple: ``(data, power)``,
``(data, stats)``, or ``(data, power, stats)`` (power first, stats last).

Notes

Difference from ARSTAN's power transform. The power comes from regressing log(spread) on log(level) across adjacent rings. dplPy (following dplR) defines the level as the pair mean (x[t] + x[t-1]) / 2; ARSTAN's power path (trnfrm/mndf with iopt=2) instead uses the trailing ring x[t]. ARSTAN also clamps the fitted power to [0, 1] (with p = 0 becoming a log transform), whereas dplPy applies p = |1 - b| directly (log only when p <= 0). So dplPy reproduces dplR's power exactly, but a given series' power can differ slightly from ARSTAN's. The plot/return_stats diagnostic uses the same mean-based level as the fit, so it stays consistent with the power dplPy actually applies.

Examples:

>>> import dplpy as dpl
>>> rwl = dpl.readers("../tests/data/csv/ca533.csv")
>>> pt = dpl.powt(rwl)                          # per-series (cook)
>>> pt, p = dpl.powt(rwl, return_power=True)
>>> rwi = dpl.detrend(pt, plot=False)           # transform, then detrend

References

.. [1] https://rdrr.io/cran/dplR/man/powt.html .. [2] Cook & Peters (1997), The Holocene, 7, 361-370.