Crossdating tree-ring data with dplPy¶

Crossdating is the step that makes a tree-ring collection usable: it confirms that every ring is assigned to the exact calendar year in which it grew. A single missing or false ring throws off the dating of every older ring by a year, so before any climate or ecological analysis the collection has to be checked ring by ring.

Behind every tool here is one idea, the leave-one-out master. To test one series, dplPy averages all the other series into a master chronology and correlates the series against it. A correctly dated series tracks the master closely. A misdated segment drifts out of step, and the pattern of that drift shows where and how the dating went wrong.

This notebook walks through the full set of crossdating tools in dplPy:

• reading and detrending a collection
• the mean interseries correlation
• testing each series with xdate
• the overview plot
• locating a simulated dating error
• dating a floating (undated) series and rejoining it
• emulating the classic COFECHA program
• batch QA reports
• and dpl.check, which screens a collection for data problems that are not dating errors

dplR users will recognize this workflow from corr.rwl.seg, corr.series.seg, and ccf.series.rwl. Every example uses the sample files in ../tests/data/rwl/, so the notebook runs as-is from the notebooks/ folder of the repository.

In [1]:

import dplpy as dpl
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np

%matplotlib inline
dpl.__version__

import dplpy as dpl import matplotlib.pyplot as plt import pandas as pd import numpy as np %matplotlib inline dpl.__version__

Out[1]:

'0.8.0'

In [2]:

RWL = "../tests/data/rwl/"   # sample files, relative to the notebooks/ folder

RWL = "../tests/data/rwl/" # sample files, relative to the notebooks/ folder

1. Reading and detrending a collection¶

Crossdating compares the shared year-to-year signal between trees, so the first step is to remove each series' own biological growth trend. A young tree grows wide rings that narrow with age, and that slow trend is unique to each tree. Left in, it would swamp the common climate signal that crossdating relies on.

Detrending divides each measured series by a smooth curve fitted to it, leaving a dimensionless ring-width index (RWI) that hovers around 1.0 and carries the high-frequency signal. This is the same detrending used for standardization (see the detrending guide); here we use a spline, in the spirit of COFECHA's 32-year default.

We start with ca533, a bristlecone pine collection from Campito Mountain that ships with both dplR and dplPy.

In [3]:

rwl = dpl.readers(RWL + "ca533.rwl")

rwl = dpl.readers(RWL + "ca533.rwl")

ca533.rwl successfully extracted as rwl file with 34 series covering the period from 626 to 1983

In [4]:

dpl.report(rwl)                        # quick overview of the collection

dpl.report(rwl) # quick overview of the collection

Number of dated series: 34
Number of measurements: 23276
Avg series length: 684.5882
Range: 1358
Span: 626 - 1983
Mean (Std dev) series intercorrelation: 0.629 (0.086)
Mean (Std dev) AR1: 0.7093 (0.0981)
-------------
Years with locally absent rings (zero values) listed by series

      CAM011 (2): 1753 1782
      CAM031 (14): 1497 1500 1523 1533 1540 1542 1545 1578 1579 1580 1655 1668 1670 1681
      CAM032 (7): 1497 1523 1579 1654 1670 1681 1782
      CAM051 (1): 1475
      CAM061 (14): 1497 1523 1542 1545 1547 1579 1654 1655 1668 1670 1672 1782 1858 1960
      CAM062 (14): 1542 1545 1547 1548 1579 1654 1655 1670 1672 1782 1836 1857 1858 1929
      CAM071 (29): 1269 1497 1498 1523 1542 1547 1578 1579 1612 1655 1656 1668 1670 1672 1674 1690 1707 1708 1756 1782 1795 1820 1836 1845 1857 1858 1924 1948 1960
      CAM072 (21): 1218 1497 1498 1523 1533 1538 1542 1545 1546 1547 1571 1579 1580 1590 1654 1655 1668 1670 1672 1675 1690
      CAM081 (2): 1218 1336
      CAM082 (3): 1362 1858 1865
      CAM091 (5): 1655 1669 1670 1782 1858
      CAM092 (36): 1624 1654 1655 1670 1672 1675 1677 1690 1703 1705 1707 1708 1710 1733 1753 1756 1757 1774 1777 1781 1782 1783 1784 1795 1807 1824 1829 1836 1845 1857 1858 1899 1904 1929 1936 1961
      CAM101 (4): 1782 1783 1899 1929
      CAM102 (6): 1669 1690 1782 1858 1899 1929
      CAM111 (1): 1542
      CAM112 (1): 1542
      CAM121 (19): 1093 1218 1254 1361 1365 1460 1462 1468 1473 1475 1492 1497 1542 1544 1545 1547 1600 1899 1960
      CAM122 (10): 1117 1133 1147 1177 1218 1254 1361 1475 1497 1670
      CAM131 (1): 1361
      CAM151 (2): 1670 1703
      CAM161 (1): 1523
      CAM162 (3): 1618 1624 1641
      CAM181 (2): 1450 1523
      CAM191 (18): 1475 1497 1523 1533 1542 1558 1571 1578 1618 1655 1668 1670 1675 1677 1690 1705 1777 1929
      CAM201 (1): 1523
      CAM211 (17): 645 762 809 847 924 957 1014 1118 1123 1133 1147 1189 1350 1384 1468 1571 1641
      Total locally absent rings: 234
-------------
Years with internal NA (missing measurements) listed by series

      (none)

In [5]:

rwi = dpl.detrend(rwl, fit="Spline", plot=False)
rwi.head()                             # detrended indices, one column per series

rwi = dpl.detrend(rwl, fit="Spline", plot=False) rwi.head() # detrended indices, one column per series

Out[5]:

	CAM011	CAM021	CAM031	CAM032	CAM041	CAM042	CAM051	CAM061	CAM062	CAM071	...	CAM151	CAM152	CAM161	CAM162	CAM171	CAM172	CAM181	CAM191	CAM201	CAM211
Year
626	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	0.371562
627	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	0.284366
628	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	0.306488
629	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	0.416286
630	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	0.482406

5 rows × 34 columns

2. Mean interseries correlation¶

The single most useful number for a collection is its mean interseries correlation: the average, across all series, of how well each one correlates with the master built from the others. COFECHA prints this as the overall interseries correlation, and a value around 0.5 or higher is typical of a well-dated, climate-sensitive site.

interseries_corr takes the raw ring widths, not the detrended indices. It prewhitens and normalizes internally, the way COFECHA does.

In [6]:

mean_corr, ic = dpl.interseries_corr(rwl)
mean_corr                              # collection-wide mean interseries correlation

mean_corr, ic = dpl.interseries_corr(rwl) mean_corr # collection-wide mean interseries correlation

Out[6]:

0.629

In [7]:

ic.head()                              # per-series correlation and p-value

ic.head() # per-series correlation and p-value

Out[7]:

	interseries_corr	p_val
series
CAM011	0.526	2.552616e-32
CAM021	0.658	6.382256e-69
CAM031	0.528	2.265238e-45
CAM032	0.627	5.164840e-61
CAM041	0.489	1.187802e-19

3. Testing each series with xdate¶

xdate is the workhorse. It slides a window along each series, correlates each window against the leave-one-out master, and flags the windows that look wrong:

• flag A means the window does not correlate significantly with the master, so it is weak or possibly misdated.
• flag B means the window correlates better at a non-zero lag than at lag 0, the classic signature of a dating shift.

The flag report prints as it runs. The full results come back in a dictionary, including a per-series correlation table and a count of possible problems.

In [8]:

res = dpl.xdate(rwi)                   # prints the flag report as it runs

res = dpl.xdate(rwi) # prints the flag report as it runs

      Number of dated series      : 34
      Master series               : 626-1983, 1358 yrs
      Total rings in all series   : 23276
      Total dated rings checked   : 22225
      Series intercorrelation     : 0.639
      Avg mean sensitivity        : 0.322
      Segments, possible problems : 18
      Mean length of series       : 684.6

Flags for CAM011
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1900-1949     +6   -0.05 -0.31  0.18 -0.17  0.04 -0.20 -0.16  0.09 -0.17  0.21  0.14 -0.08 -0.03  0.08  0.12 -0.06  0.31  0.20 -0.17  0.09 -0.05
      1925-1974     -8   -0.07 -0.28  0.29 -0.19  0.18 -0.20 -0.04 -0.00  0.04  0.08  0.27 -0.17 -0.10 -0.22  0.08  0.17  0.28 -0.09 -0.34  0.07      

Flags for CAM031
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1775-1824     +8   -0.11  0.05 -0.10 -0.09 -0.11  0.09  0.15 -0.06 -0.07  0.07  0.29 -0.11  0.02 -0.12  0.02  0.01 -0.29 -0.10  0.36  0.05 -0.04

Flags for CAM051
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1375-1424     +9   -0.01 -0.20  0.28  0.11 -0.09  0.06  0.30  0.10 -0.01 -0.02  0.19 -0.03 -0.16  0.24 -0.04 -0.06 -0.04  0.03 -0.11  0.38 -0.11

Flags for CAM122
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1900-1949     +6    0.00 -0.17  0.24  0.07  0.00  0.06 -0.28  0.07 -0.08 -0.02  0.39  0.02  0.00 -0.29 -0.09 -0.18  0.42  0.01 -0.15 -0.08  0.03

Flags for CAM131
  [A] not significant as dated ('best' = lag of highest correlation):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1800-1849     +0   -0.13 -0.12 -0.05  0.04  0.09 -0.04 -0.13 -0.15 -0.01 -0.25  0.14 -0.10  0.11 -0.16  0.02 -0.34  0.09 -0.00  0.10 -0.09  0.05

Flags for CAM161
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1475-1524     +1   -0.11  0.10  0.11 -0.02 -0.11 -0.03  0.22 -0.16 -0.21 -0.05  0.29  0.31 -0.25  0.04 -0.14  0.20 -0.09 -0.09  0.01  0.17  0.01

Flags for CAM171
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1250-1299     -4   -0.06 -0.10  0.10 -0.03 -0.18 -0.06  0.36  0.14 -0.23 -0.06  0.34  0.04 -0.15  0.08  0.07  0.16 -0.10 -0.05 -0.20  0.06  0.18
      1275-1324     -4   -0.07  0.00 -0.09  0.01 -0.04 -0.04  0.47  0.26 -0.12  0.03  0.26  0.21  0.01  0.20  0.11 -0.04  0.05 -0.14 -0.01  0.01 -0.13

Flags for CAM181
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1775-1824     +8   -0.14  0.04  0.08 -0.05 -0.12  0.19  0.15 -0.36 -0.29  0.06  0.20 -0.02 -0.14  0.16  0.13 -0.05 -0.01  0.00  0.31  0.05 -0.14
      1800-1849     +9   -0.32  0.08  0.01 -0.01 -0.09  0.23  0.02 -0.20  0.02 -0.03  0.25  0.03 -0.10 -0.02  0.01 -0.11 -0.01  0.04  0.02  0.26 -0.11

Flags for CAM201
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1025-1074    -10    0.35  0.21 -0.13 -0.04 -0.23  0.05 -0.11 -0.10 -0.30 -0.05  0.30  0.26  0.06  0.04 -0.12 -0.03 -0.07  0.11 -0.32 -0.08 -0.02
      1125-1174     +1   -0.02 -0.02 -0.11 -0.18 -0.08  0.09  0.17  0.01  0.19 -0.26  0.29  0.39  0.11 -0.25  0.01 -0.06 -0.14 -0.00 -0.10 -0.14 -0.16
      1350-1399     -7   -0.03  0.03 -0.03  0.24 -0.08 -0.10 -0.13  0.01 -0.08  0.23  0.20  0.18 -0.13  0.14  0.08 -0.14 -0.25  0.03 -0.14 -0.13  0.13

Flags for CAM211
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      825-874       -1    0.04  0.12  0.09 -0.14  0.13 -0.02  0.00  0.15  0.19  0.36  0.31  0.18  0.24 -0.04 -0.06 -0.22 -0.04 -0.18 -0.07 -0.21  0.26
      1000-1049     -1    0.02  0.06 -0.14 -0.07  0.10 -0.06 -0.23 -0.10 -0.03  0.42  0.29 -0.32 -0.13  0.05  0.16 -0.06 -0.27 -0.25  0.08  0.12  0.15
      1025-1074     -1    0.01 -0.20 -0.07 -0.08 -0.20 -0.09 -0.18 -0.18  0.18  0.70  0.36 -0.14 -0.02  0.09 -0.12 -0.34 -0.26 -0.15 -0.04  0.10  0.14
      1350-1399     -7    0.12  0.04  0.33  0.37 -0.12 -0.03 -0.08 -0.04  0.04  0.23  0.36  0.25 -0.11  0.23  0.01 -0.11  0.05  0.03 -0.05  0.06  0.06

In [9]:

res["n_problems"]                      # number of flagged segments

res["n_problems"] # number of flagged segments

Out[9]:

18

In [10]:

res["overall"].head()                  # each series' correlation with the master

res["overall"].head() # each series' correlation with the master

Out[10]:

	rho	p_val
CAM011	0.522	9.135324e-33
CAM021	0.678	5.689132e-75
CAM031	0.552	6.968125e-51
CAM032	0.625	1.107723e-60
CAM041	0.484	3.122742e-19

4. The crossdating overview plot¶

xdate_plot draws the whole collection at a glance. Each series is a horizontal bar across its calendar extent:

• green is the series' full span.
• blue is a segment that dates well.
• red is a flagged segment to inspect.

A healthy collection is almost all blue. The red segments are where to look next.

In [11]:

dpl.xdate_plot(rwi, p_val=0.05)
plt.show()

dpl.xdate_plot(rwi, p_val=0.05) plt.show()

5. Locating a dating error¶

The most common real error is a missing ring. In a dry year a tree may grow a ring so thin it was never measured. Every ring older than the gap then gets labeled one year too young, and the series slips out of step with the master over its older half.

Following Andy Bunn's dplR crossdating workshop, we simulate exactly that on the co021 collection. We deliberately corrupt series 644222: remove its ring for the year 1600, as if it had been missed while measuring, and prepend an NaN so the series keeps its length while every ring older than 1600 shifts a year too young.

In [12]:

co = dpl.readers(RWL + "co021.rwl")

# corrupt one series: drop its 1600 ring and prepend NaN, as if it had been missed
series = "644222"
year = 1600
i = co.index.get_loc(year)             # row position of the ring to drop

miss = co.copy()
col = miss[series].to_numpy()
miss[series] = np.concatenate(([np.nan], np.delete(col, i)))
series, year                           # the corrupted series and the ring's true year

co = dpl.readers(RWL + "co021.rwl") # corrupt one series: drop its 1600 ring and prepend NaN, as if it had been missed series = "644222" year = 1600 i = co.index.get_loc(year) # row position of the ring to drop miss = co.copy() col = miss[series].to_numpy() miss[series] = np.concatenate(([np.nan], np.delete(col, i))) series, year # the corrupted series and the ring's true year

co021.rwl successfully extracted as rwl file with 35 series covering the period from 1176 to 1963

Out[12]:

('644222', 1600)

In [13]:

miss_rwi = dpl.detrend(miss, fit="Spline", plot=False)
res_miss = dpl.xdate(miss_rwi)         # the corrupted series now draws flags

miss_rwi = dpl.detrend(miss, fit="Spline", plot=False) res_miss = dpl.xdate(miss_rwi) # the corrupted series now draws flags

      Number of dated series      : 35
      Master series               : 1176-1963, 788 yrs
      Total rings in all series   : 19771
      Total dated rings checked   : 18650
      Series intercorrelation     : 0.872
      Avg mean sensitivity        : 0.604
      Segments, possible problems : 9
      Mean length of series       : 564.9

Flags for 644222
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1350-1399     -1    0.04  0.05 -0.16 -0.06 -0.19  0.02 -0.33 -0.12  0.07  0.69 -0.02  0.08 -0.15  0.16  0.03  0.12 -0.17 -0.03 -0.25 -0.04 -0.05
      1375-1424     -1   -0.28  0.10 -0.38 -0.10 -0.17 -0.03  0.01  0.26  0.05  0.84 -0.08  0.35 -0.06 -0.07 -0.01 -0.16 -0.28 -0.03 -0.28 -0.08 -0.11
      1400-1449     -1   -0.42 -0.02 -0.33 -0.16 -0.13  0.08  0.10  0.17  0.03  0.77  0.02  0.31  0.11  0.19  0.19 -0.19  0.06 -0.03 -0.18 -0.06 -0.06
      1425-1474     -1   -0.12  0.06  0.09 -0.14 -0.08  0.12  0.07 -0.04 -0.00  0.83 -0.10 -0.03 -0.11  0.21 -0.13 -0.19  0.22  0.13  0.02 -0.11  0.07
      1450-1499     -1   -0.05  0.30  0.32 -0.05 -0.05  0.06 -0.14 -0.01 -0.06  0.89 -0.07 -0.05 -0.21  0.10 -0.14 -0.04  0.12  0.24 -0.08  0.05 -0.02
      1475-1524     -1   -0.12 -0.01  0.15  0.07  0.09 -0.15 -0.32 -0.12  0.05  0.80  0.16 -0.05 -0.10 -0.13 -0.05  0.19 -0.10 -0.14 -0.29  0.09  0.01
      1500-1549     -1   -0.11 -0.28  0.09  0.02 -0.07 -0.10 -0.07 -0.26  0.05  0.81  0.13 -0.12 -0.02 -0.10 -0.14  0.06 -0.04 -0.32 -0.21  0.22 -0.05
      1525-1574     -1   -0.22 -0.24  0.27 -0.04 -0.28 -0.12  0.15 -0.16 -0.18  0.90  0.05 -0.25  0.02  0.08 -0.08 -0.21  0.26 -0.12 -0.18  0.14 -0.20
      1550-1599     -1   -0.28 -0.00  0.19  0.07  0.04 -0.32  0.11 -0.02  0.00  0.81  0.04 -0.14  0.13 -0.05  0.17 -0.04  0.10  0.19 -0.28  0.04  0.07

In [14]:

dpl.xdate_plot(miss_rwi)               # the corrupted series reads red over its older half
plt.show()

dpl.xdate_plot(miss_rwi) # the corrupted series reads red over its older half plt.show()

6. Drilling into the flagged series¶

series_corr zooms in on one series. The lag panel is the key part: for each window it shows the lag at which the window best matches the master.

For a missing ring, every window older than the gap correlates best at lag +1, while younger windows sit at lag 0. The year where the lag jumps from 0 to +1 is the year the ring was missed. The error is located, not just detected.

In [15]:

sc_miss = dpl.series_corr(miss_rwi, series, plot=True)
plt.show()

sc_miss = dpl.series_corr(miss_rwi, series, plot=True) plt.show()

NB: with series_x=True, positive lags indicate missing rings in the series

7. Dating a floating series¶

Sometimes a series carries no calendar dates at all, such as a beam from an old building or a subfossil log. Given a dated reference collection, xdate_floater finds where the floating series belongs in time.

Here we take one series from ca533, throw away its dates, and recover them.

In [16]:

floating = rwl["CAM021"].dropna().values     # just the ring values, undated
reference = rwl.drop(columns=["CAM021"])      # the rest of the collection as reference

fl = dpl.xdate_floater(reference, floating, series_name="CAM021", plot=True, return_rwl=True)
plt.show()

floating = rwl["CAM021"].dropna().values # just the ring values, undated reference = rwl.drop(columns=["CAM021"]) # the rest of the collection as reference fl = dpl.xdate_floater(reference, floating, series_name="CAM021", plot=True, return_rwl=True) plt.show()

Reference master years: 631 to 1983 (1353)
Floating series length: 551 (transform 'pw' -> 545)
Minimum overlap for search: 50

Best match for 'CAM021': 1433 to 1983  (t = 20.34, r = 0.658, eff_df = 545, p_bonf = 6.94e-66, 1/p = >1 million, IF = >1000, overlap n = 545)

In [17]:

fl["best"]                             # the recovered calendar placement

fl["best"] # the recovered calendar placement

Out[17]:

{'min_year': 1433,
 'max_year': 1983,
 'r': 0.6575472355491244,
 't': 20.337143327658065,
 'eff_df': 544.990930993626,
 'p_bonf': 6.93775874208215e-66,
 'one_over_p': 1.4413876832215724e+65,
 'isolation_factor': 5.386473277333847e+65,
 'n': 545,
 'date_warning': None}

Rejoining the placed series¶

With return_rwl=True, xdate_floater hands back the floater already dated at its best-fit years as fl["placed"]. We pass that straight to combine_rwl, which merges it into the reference collection on the union of their years, exactly as dplR's combine.rwl does. Because the placement is correct, the recovered CAM021 lands back on its original years and the rejoined collection matches the one we started with.

(xdate_floater also returns fl["combined"], the same rejoin done for you, when you do not need the two pieces separately.)

In [ ]:

placed = fl["placed"] # the floater, dated at its best-fit years

rejoined = dpl.combine_rwl(reference, placed) # use combine_rwl to join two DataFrames along their year index
rejoined.shape # size of the reference plus the replaced series - back to the whole collection

placed = fl["placed"] # the floater, dated at its best-fit years rejoined = dpl.combine_rwl(reference, placed) # use combine_rwl to join two DataFrames along their year index rejoined.shape # size of the reference plus the replaced series - back to the whole collection

Out[ ]:

(1358, 34)

In [ ]:

# the recovered series lands back on its original years
recovered = rejoined["CAM021"].dropna().values
original = rwl["CAM021"].dropna().values
np.array_equal(recovered, original)  # if True the placement is exact

# the recovered series lands back on its original years recovered = rejoined["CAM021"].dropna().values original = rwl["CAM021"].dropna().values np.array_equal(recovered, original) # if True the placement is exact

Out[ ]:

True

8. Emulating COFECHA¶

COFECHA (Holmes, 1983) is the Fortran program that has been the standard crossdating check for decades. preset="COFECHA" reproduces its exact recipe: a 32-year spline detrend, autoregressive prewhitening using the Burg method, an arithmetic z-scored master, Pearson instead of Spearman correlation, and COFECHA's segment anchoring and critical value.

Unlike the default path, the preset expects to take the raw ring widths and detrends them itself, just as the COFECHA Fortran program did, so there is no need to call detrend first.

In [20]:

cof = dpl.xdate(rwl, preset="COFECHA")  # raw frame; the preset detrends internally

cof = dpl.xdate(rwl, preset="COFECHA") # raw frame; the preset detrends internally

      Number of dated series      : 34
      Master series               : 626-1983, 1358 yrs
      Total rings in all series   : 23276
      Total dated rings checked   : 23227
      Series intercorrelation     : 0.669
      Avg mean sensitivity        : 0.316
      Segments, possible problems : 15
      Mean length of series       : 684.6

Flags for CAM011
  [A] not significant as dated ('best' = lag of highest correlation):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1900-1949     +0    0.03 -0.25  0.17 -0.04  0.08 -0.22 -0.23 -0.02 -0.20  0.07  0.31 -0.23 -0.04 -0.06 -0.00 -0.01  0.22  0.26 -0.14  0.08 -0.13
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1934-1983     -1   -0.02 -0.08  0.14  0.08  0.03 -0.15 -0.06 -0.04 -0.03  0.27  0.17                                                            

Flags for CAM031
  [A] not significant as dated ('best' = lag of highest correlation):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1800-1849     +0    0.03  0.06 -0.10  0.26  0.04 -0.00  0.11 -0.15  0.12 -0.08  0.33 -0.03  0.29 -0.09 -0.23 -0.16 -0.33 -0.09  0.17  0.27  0.02

Flags for CAM051
  [A] not significant as dated ('best' = lag of highest correlation):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1400-1449     +0   -0.15 -0.12  0.10  0.15 -0.27  0.06  0.26 -0.10 -0.34 -0.21  0.30 -0.06 -0.10  0.15  0.13  0.03  0.02  0.11  0.12  0.17 -0.16
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1375-1424     -4   -0.01 -0.25  0.17  0.06 -0.10  0.06  0.33  0.03 -0.04 -0.12  0.17 -0.01 -0.17  0.15 -0.06 -0.13 -0.04  0.13 -0.03  0.29 -0.18

Flags for CAM131
  [A] not significant as dated ('best' = lag of highest correlation):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1800-1849     +0   -0.09 -0.07  0.12  0.19  0.15  0.01 -0.06 -0.09  0.12 -0.20  0.27 -0.09  0.11 -0.09  0.04 -0.24  0.18  0.03  0.13 -0.11 -0.00

Flags for CAM162
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1350-1399     -3   -0.12 -0.07 -0.11 -0.06  0.11 -0.12 -0.01  0.39  0.22  0.13  0.34 -0.05 -0.00 -0.17 -0.01 -0.18 -0.32 -0.16  0.08 -0.16 -0.04

Flags for CAM201
  [A] not significant as dated ('best' = lag of highest correlation):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1325-1374     +0   -0.10 -0.02 -0.07  0.07  0.04 -0.01  0.05  0.03  0.14  0.22  0.29  0.07  0.23  0.17 -0.10 -0.03 -0.26 -0.19 -0.27 -0.20 -0.05
      1375-1424     +0    0.11  0.05  0.06  0.15 -0.09 -0.05 -0.23 -0.18 -0.07 -0.12  0.31  0.18 -0.14 -0.11  0.15 -0.24 -0.06  0.16 -0.00 -0.20  0.11
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1350-1399     +1   -0.08  0.01 -0.13  0.16 -0.11 -0.12 -0.27 -0.04 -0.09  0.27  0.30  0.31 -0.06  0.19  0.12 -0.09 -0.27  0.03 -0.24 -0.26  0.12

Flags for CAM211
  [A] not significant as dated ('best' = lag of highest correlation):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      750-799       +0    0.20 -0.00 -0.03 -0.17 -0.05  0.03 -0.00 -0.13 -0.09 -0.01  0.24 -0.23 -0.09  0.05 -0.16  0.24  0.16  0.00  0.07 -0.04  0.09
  [B] correlates better at another lag ('best' = that lag):
      Segment     best     -10    -9    -8    -7    -6    -5    -4    -3    -2    -1     0    +1    +2    +3    +4    +5    +6    +7    +8    +9   +10
      1000-1049     -9    0.02  0.32 -0.11 -0.16  0.07 -0.08 -0.21 -0.06 -0.03  0.27  0.30 -0.13 -0.14  0.15  0.20 -0.25 -0.29 -0.14  0.06  0.02  0.13
      1025-1074     -1   -0.01 -0.13  0.00 -0.09 -0.04 -0.06 -0.16 -0.19  0.21  0.67  0.26 -0.08  0.00  0.21 -0.06 -0.47 -0.40 -0.11 -0.03  0.06  0.12
      1350-1399     -7    0.02 -0.01  0.21  0.34 -0.13 -0.09 -0.21 -0.11  0.03  0.13  0.29  0.16 -0.13  0.14 -0.01 -0.18 -0.06 -0.20 -0.17 -0.00 -0.03
      1375-1424     -7   -0.20 -0.12  0.06  0.37 -0.14 -0.03 -0.05 -0.19 -0.08  0.06  0.37 -0.04 -0.08  0.01 -0.01 -0.23  0.16 -0.05 -0.19  0.04  0.02

In [21]:

cof["n_problems"]                       # COFECHA-style count of possible problems

cof["n_problems"] # COFECHA-style count of possible problems

Out[21]:

15

9. Batch QA reports¶

xdate_report runs the whole crossdating check on a file and produces a COFECHA-style text report, built for screening many files at once. It accepts a file path or a collection already read into memory.

output= controls where the report goes: "screen" prints it, "file" writes a .txt, and "both" (the default) does both. We use "screen" here so the notebook does not leave files behind.

In [ ]:

rep = dpl.xdate_report(RWL + "ca533.rwl", output="screen", preset="COFECHA") # call the report on a file not yet in memory

rep = dpl.xdate_report(RWL + "ca533.rwl", output="screen", preset="COFECHA") # call the report on a file not yet in memory

ca533.rwl successfully extracted as rwl file with 34 series covering the period from 626 to 1983

COFECHA-style output, reproduced with dplPy 0.8.0 preset="COFECHA" on 13Oct26


      Measurement file name  : ca533.rwl
      Beginning year         : 626
      Ending year            : 1983

      Series intercorrelation: .669
      Avg mean sensitivity   : .316
      Avg standard deviation : .188
      Avg autocorrelation    : .720
      Number dated series    : 34
      Segment length tested  : 50

      Number problem segments: 18
      Pct problem segments   : 1.94


 PART 5:  CORRELATION OF SERIES BY SEGMENTS: 
-----------------------------------------------------------------------------------------------------------------------------------
Correlations of  50-year dated segments, lagged  25 years
Flags:  A = correlation under .3281 but highest as dated;  B = correlation higher at other than dated position
        critical value .3281 for the 50-year window, from pcrit = 0.01, one tailed

 Seq Series  Time_span     625  650  675  700  725  750  775  800  825  850  875  900  925  950  975 1000 1025 1050 1075 1100
                           674  699  724  749  774  799  824  849  874  899  924  949  974  999 1024 1049 1074 1099 1124 1149
 --- -------- ---------   ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
  10 CAM071   1037 1983                                                                                  .67  .63  .63  .72 
  11 CAM072   1114 1983                                                                                                 .78 
  12 CAM081   1081 1983                                                                                            .63  .75 
  13 CAM082    977 1983                                                                        .62  .58  .53  .70  .77  .75 
  20 CAM121   1000 1983                                                                             .54  .61  .45  .49  .72 
  21 CAM122   1000 1983                                                                             .66  .58  .55  .60  .71 
  22 CAM131    695 1970            .60  .57  .66  .52  .67  .74  .50  .59  .77  .81  .63  .61  .63  .53  .48  .44  .64  .77 
  23 CAM132    710 1232                 .69  .65  .48  .60  .62  .56  .75  .82  .73  .53  .61  .68  .60  .54  .60  .69  .75 
  24 CAM141   1030 1970                                                                                  .46  .67  .59  .66 
  27 CAM161   1106 1609                                                                                                 .71 
  28 CAM162    971 1970                                                                   .59  .65  .60  .56  .60  .67  .70 
  33 CAM201    990 1582                                                                        .37  .44  .35  .57  .71  .55 
  34 CAM211    626 1968            .48  .43  .41  .17B .41  .48  .30B .42  .59  .52  .34  .46  .55  .22B .26B .57  .47  .57 
 Av segment correlation    NaN  NaN .54  .56  .57  .39  .56  .61  .45  .59  .73  .69  .50  .57  .58  .52  .50  .58  .63  .70 

 Seq Series  Time_span    1125 1150 1175 1200 1225 1250 1275 1300 1325 1350 1375 1400 1425 1450 1475 1500 1525 1550 1575 1600
                          1174 1199 1224 1249 1274 1299 1324 1349 1374 1399 1424 1449 1474 1499 1524 1549 1574 1599 1624 1649
 --- -------- ---------   ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
   1 CAM011   1530 1983                                                                                  .70  .74  .83  .72 
   2 CAM021   1433 1983                                                              .72  .64  .68  .71  .71  .68  .72  .71 
   3 CAM031   1356 1983                                               .47  .37  .54  .67  .70  .68  .60  .53  .48  .49  .59 
   4 CAM032   1435 1983                                                              .48  .53  .68  .71  .54  .36  .62  .68 
   6 CAM042   1538 1983                                                                                  .63  .60  .64  .66 
   7 CAM051   1247 1983                      .58  .54  .60  .69  .52  .38  .17B .30A .57  .42  .47  .68  .60  .64  .73  .66 
   8 CAM061   1357 1983                                               .74  .75  .80  .78  .80  .80  .75  .63  .69  .82  .83 
   9 CAM062   1525 1983                                                                                  .67  .74  .76  .77 
  10 CAM071   1037 1983  .82  .63  .62  .66  .61  .59  .61  .75  .78  .80  .82  .77  .71  .73  .76  .82  .78  .77  .76  .68 
  11 CAM072   1114 1983  .77  .53  .49  .57  .65  .74  .68  .70  .80  .77  .59  .66  .77  .72  .69  .59  .53  .67  .75  .75 
  12 CAM081   1081 1983  .65  .65  .70  .68  .75  .81  .76  .65  .67  .70  .64  .71  .71  .67  .68  .69  .55  .52  .70  .76 
  13 CAM082    977 1983  .53  .49  .76  .77  .65  .61  .62  .65  .53  .52  .56  .58  .60  .55  .65  .77  .68  .73  .78  .70 
  14 CAM091   1460 1983                                                                   .78  .84  .89  .76  .72  .70  .72 
  15 CAM092   1591 1983                                                                                            .82  .80 
  18 CAM111   1446 1983                                                              .79  .80  .85  .88  .86  .80  .86  .79 
  19 CAM112   1471 1983                                                                   .75  .82  .86  .83  .81  .82  .78 
  20 CAM121   1000 1983  .75  .77  .81  .77  .79  .79  .72  .73  .72  .78  .76  .78  .70  .60  .84  .80  .64  .78  .88  .86 
  21 CAM122   1000 1983  .69  .63  .70  .75  .73  .71  .65  .57  .61  .79  .74  .77  .84  .83  .82  .84  .78  .70  .71  .76 
  22 CAM131    695 1970  .67  .58  .64  .70  .68  .64  .61  .51  .50  .51  .48  .67  .69  .62  .62  .71  .64  .51  .63  .75 
  23 CAM132    710 1232  .72  .67  .74  .69                                                                                 
  24 CAM141   1030 1970  .76  .74  .80  .84  .82  .83  .79  .77  .77  .74  .77  .81  .85  .87  .87  .84  .74  .71  .75  .76 
  25 CAM151   1222 1970                 .56  .59  .65  .60  .72  .66  .37  .41  .68  .63  .70  .70  .71  .79  .81  .84  .78 
  26 CAM152   1221 1449                 .49  .53  .59  .62  .64  .68  .71  .69  .77                                         
  27 CAM161   1106 1609  .77  .65  .62  .73  .70  .68  .68  .71  .46  .35  .48  .41  .46  .45  .23B .33B .60  .58  .52      
  28 CAM162    971 1970  .63  .54  .66  .72  .54  .47  .51  .56  .49  .34B .44  .52  .43  .62  .84  .72  .54  .56  .62  .39 
  29 CAM171   1213 1970                 .58  .49  .42  .34  .57  .57  .47  .52  .58  .78  .79  .62  .66  .62  .64  .82  .78 
  30 CAM172   1174 1970       .50  .49  .62  .62  .56  .46  .59  .52  .41  .55  .60  .57  .62  .56  .62  .64  .65  .77  .82 
  31 CAM181   1190 1970            .76  .68  .59  .71  .68  .67  .75  .61  .60  .73  .77  .85  .74  .66  .70  .68  .70  .74 
  32 CAM191   1180 1970            .75  .76  .73  .67  .59  .72  .79  .72  .80  .84  .71  .57  .66  .77  .54  .65  .76  .76 
  33 CAM201    990 1582  .42  .45  .49  .47  .62  .69  .53  .42  .29A .30B .31A .51  .67  .58  .55  .51  .46  .60           
  34 CAM211    626 1968  .66  .55  .75  .79  .60  .52  .55  .73  .60  .29B .42  .76  .69  .59  .66  .67  .66  .66  .76  .50 
 Av segment correlation   .68  .60  .67  .67  .65  .64  .61  .65  .62  .56  .56  .66  .68  .67  .69  .71  .66  .66  .73  .72 

 Seq Series  Time_span    1625 1650 1675 1700 1725 1750 1775 1800 1825 1850 1875 1900 1925 1950 1975
                          1674 1699 1724 1749 1774 1799 1824 1849 1874 1899 1924 1949 1974 1999 2024
 --- -------- ---------   ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
   1 CAM011   1530 1983  .64  .74  .72  .52  .51  .50  .41  .49  .58  .63  .38  .31A .35  .17      
   2 CAM021   1433 1983  .71  .70  .61  .66  .80  .71  .53  .50  .67  .79  .78  .68  .61  .56      
   3 CAM031   1356 1983  .66  .56  .64  .67  .57  .66  .50  .33A .59  .70  .64  .69  .72  .64      
   4 CAM032   1435 1983  .64  .71  .73  .61  .60  .67  .65  .59  .72  .85  .73  .70  .69  .62      
   5 CAM041   1683 1983            .33  .35  .39  .48  .57  .68  .72  .73  .58  .50  .65  .68      
   6 CAM042   1538 1983  .79  .73  .69  .64  .44  .47  .54  .44  .56  .76  .73  .74  .76  .77      
   7 CAM051   1247 1983  .74  .75  .68  .66  .60  .52  .51  .57  .54  .44  .43  .36  .49  .48      
   8 CAM061   1357 1983  .85  .85  .82  .76  .79  .79  .75  .81  .81  .85  .78  .75  .72  .67      
   9 CAM062   1525 1983  .81  .82  .80  .79  .84  .86  .81  .78  .80  .83  .74  .74  .78  .77      
  10 CAM071   1037 1983  .69  .73  .68  .68  .73  .67  .68  .73  .60  .72  .70  .38  .38  .41      
  11 CAM072   1114 1983  .73  .72  .78  .75  .78  .86  .84  .82  .66  .67  .62  .59  .77  .68      
  12 CAM081   1081 1983  .71  .51  .51  .59  .73  .87  .67  .58  .62  .62  .67  .48  .38  .44      
  13 CAM082    977 1983  .74  .67  .62  .64  .63  .74  .78  .60  .52  .41  .41  .72  .67  .53      
  14 CAM091   1460 1983  .85  .84  .79  .71  .71  .74  .75  .79  .81  .80  .73  .70  .75  .73      
  15 CAM092   1591 1983  .76  .70  .60  .58  .68  .54  .48  .66  .79  .65  .45  .54  .67  .71      
  16 CAM101   1727 1983                      .71  .74  .81  .79  .81  .82  .79  .82  .82  .84      
  17 CAM102   1665 1983       .66  .60  .71  .77  .80  .82  .67  .64  .80  .75  .59  .68  .72      
  18 CAM111   1446 1983  .77  .86  .83  .82  .87  .81  .76  .75  .80  .88  .77  .71  .82  .75      
  19 CAM112   1471 1983  .76  .83  .81  .75  .80  .79  .78  .80  .80  .85  .77  .70  .70  .63      
  20 CAM121   1000 1983  .86  .89  .88  .81  .76  .76  .71  .73  .69  .52  .39B .57  .68  .66      
  21 CAM122   1000 1983  .84  .89  .87  .76  .78  .90  .82  .78  .80  .83  .58  .47  .66  .54      
  22 CAM131    695 1970  .79  .73  .63  .44  .55  .63  .43  .27A .48  .66  .65  .62  .60           
  24 CAM141   1030 1970  .79  .83  .71  .65  .74  .76  .77  .72  .77  .81  .72  .76  .86           
  25 CAM151   1222 1970  .78  .82  .70  .62  .74  .74  .73  .76  .73  .79  .79  .75  .78           
  28 CAM162    971 1970  .42  .52  .51  .68  .77  .67  .57  .62  .61  .69  .75  .55  .58           
  29 CAM171   1213 1970  .78  .72  .50  .39  .55  .56  .57  .49  .49  .59  .61  .57  .59           
  30 CAM172   1174 1970  .82  .80  .71  .70  .80  .68  .69  .78  .74  .79  .79  .74  .77           
  31 CAM181   1190 1970  .79  .80  .80  .65  .54  .45  .38  .42  .44  .58  .65  .47  .54           
  32 CAM191   1180 1970  .74  .68  .82  .69  .68  .69  .75  .81  .77  .85  .80  .73  .80           
  34 CAM211    626 1968  .47  .76  .71  .63  .73  .61  .52  .52  .51  .69  .65  .50  .49           
 Av segment correlation   .74  .74  .69  .65  .69  .69  .65  .64  .67  .72  .66  .62  .66  .62   NaN

 PART 7:  DESCRIPTIVE STATISTICS: 
-----------------------------------------------------------------------------------------------------------------------------------

                                                Corr   //-------- Unfiltered --------\\  //---- Filtered -----\\
                           No.    No.    No.    with   Mean   Max     Std   Auto   Mean   Max     Std   Auto  AR
 Seq Series   Interval   Years  Segmt  Flags   Master  msmt   msmt    dev   corr   sens  value    dev   corr  ()
 --- -------- ---------  -----  -----  -----   ------ -----  -----  -----  -----  -----  -----  -----  -----  --
   1 CAM011   1530 1983    454     18      2     .565   .44   1.36   .222   .702   .344   2.73  1.001   .002   2
   2 CAM021   1433 1983    551     22      0     .681   .42   1.11   .185   .703   .296   2.42  1.001   .005   2
   3 CAM031   1356 1983    628     25      1     .627   .35   1.03   .214   .809   .374   2.98  1.001   .003   2
   4 CAM032   1435 1983    549     22      0     .661   .29    .85   .163   .665   .438   2.58  1.001  -.003   5
   5 CAM041   1683 1983    301     12      0     .549   .53   1.38   .223   .701   .264   2.68  1.002   .000   3
   6 CAM042   1538 1983    446     18      0     .641   .44   3.03   .348   .885   .268   2.51  1.001   .006   3
   7 CAM051   1247 1983    737     30      2     .554   .27   1.32   .140   .711   .313   2.49  1.001  -.001   4
   8 CAM061   1357 1983    627     25      0     .790   .46   1.09   .202   .510   .448   2.64  1.001   .007   2
   9 CAM062   1525 1983    459     18      0     .800   .44    .92   .188   .529   .430   2.26  1.001   .005   3
  10 CAM071   1037 1983    947     38      0     .709   .25    .62   .109   .578   .408   2.22  1.001  -.005   5
  11 CAM072   1114 1983    870     35      0     .718   .31   1.22   .163   .735   .385   3.68  1.001   .002   2
  12 CAM081   1081 1983    903     36      0     .657   .33    .82   .124   .719   .248   2.50  1.001  -.001   3
  13 CAM082    977 1983   1007     40      0     .634   .29    .76   .114   .770   .238   2.83  1.000   .002   3
  14 CAM091   1460 1983    524     21      0     .768   .53   1.54   .255   .630   .404   2.35  1.001   .000   1
  15 CAM092   1591 1983    393     16      0     .723   .35    .98   .226   .559   .687   2.82  1.001   .002   2
  16 CAM101   1727 1983    257     10      0     .806   .57   1.26   .260   .709   .330   2.27  1.002   .004   2
  17 CAM102   1665 1983    319     13      0     .747   .60   1.43   .261   .677   .346   2.38  1.002  -.001   2
  18 CAM111   1446 1983    538     22      0     .809   .63   1.42   .249   .625   .294   2.92  1.001   .007   4
  19 CAM112   1471 1983    513     21      0     .781   .57   1.27   .211   .580   .294   2.43  1.001   .006   4
  20 CAM121   1000 1983    984     39      1     .737   .26    .59   .106   .594   .358   2.29  1.001  -.004   4
  21 CAM122   1000 1983    984     39      0     .738   .27    .73   .109   .649   .319   2.75  1.001  -.000   5
  22 CAM131    695 1970   1276     51      1     .611   .55   1.20   .198   .788   .202   3.24  1.000   .002   3
  23 CAM132    710 1232    523     21      0     .648   .40    .99   .148   .809   .191   3.26  1.001  -.004   2
  24 CAM141   1030 1970    941     37      0     .755   .63   1.39   .204   .748   .190   2.39  1.001   .006   3
  25 CAM151   1222 1970    749     30      0     .702   .45   1.64   .273   .772   .382   3.41  1.001  -.002   3
  26 CAM152   1221 1449    229      9      0     .643   .53   1.25   .195   .691   .236   2.21  1.002  -.003   4
  27 CAM161   1106 1609    504     20      2     .563   .34    .90   .149   .794   .243   2.55  1.001  -.002   2
  28 CAM162    971 1970   1000     40      1     .596   .40   1.04   .184   .840   .241   2.61  1.001  -.003   2
  29 CAM171   1213 1970    758     30      0     .594   .45   1.54   .210   .803   .232   3.23  1.001  -.009   2
  30 CAM172   1174 1970    797     32      0     .659   .48   1.98   .249   .850   .227   2.82  1.001   .005   2
  31 CAM181   1190 1970    781     31      0     .654   .28    .80   .149   .807   .309   3.11  1.001   .002   2
  32 CAM191   1180 1970    791     31      0     .759   .37   2.54   .336   .881   .395   2.59  1.001   .006   2
  33 CAM201    990 1582    593     24      3     .505   .47   1.49   .181   .710   .242   3.21  1.001   .011   3
  34 CAM211    626 1968   1343     51      5     .555   .36   1.10   .182   .682   .388   2.67  1.000  -.000   4
 --- -------- ---------  -----  -----  -----   ------ -----  -----  -----  -----  -----  -----  -----  -----  --
 Total or mean:          23276    927     18     .669   .40   1.21   .188   .720   .316   2.75  1.001   .001

  [1/1] ca533: 34 series, 18 problem segments
1 of 1 file(s) reported OK

In [ ]:

nm046 = dpl.readers(RWL + "nm046.rwl")
rep2 = dpl.xdate_report(nm046, name="nm046", output="screen")   # or, call a collection already in memory

nm046 = dpl.readers(RWL + "nm046.rwl") rep2 = dpl.xdate_report(nm046, name="nm046", output="screen") # or, call a collection already in memory

nm046.rwl successfully extracted as rwl file with 8 series covering the period from 1681 to 1969

COFECHA-style output, reproduced with dplPy 0.8.0 on 13Oct26


      Measurement file name  : nm046
      Beginning year         : 1681
      Ending year            : 1969

      Series intercorrelation: .813
      Avg mean sensitivity   : .477
      Avg standard deviation : 1.076
      Avg autocorrelation    : .584
      Number dated series    : 8
      Segment length tested  : 50

      Number problem segments: 0
      Pct problem segments   : 0.00


 PART 5:  CORRELATION OF SERIES BY SEGMENTS: 
-----------------------------------------------------------------------------------------------------------------------------------
Correlations of  50-year dated segments, lagged  25 years
Flags:  A = correlation not significant but highest as dated;  B = correlation higher at other than dated position
        critical value from pcrit = 0.05, one tailed

 Seq Series  Time_span    1700 1725 1750 1775 1800 1825 1850 1875 1900
                          1749 1774 1799 1824 1849 1874 1899 1924 1949
 --- -------- ---------   ---- ---- ---- ---- ---- ---- ---- ---- ----
   1 644011   1681 1969                      .88  .83  .78  .86  .84 
   2 644021   1795 1969                      .84  .78  .76  .88  .88 
   3 644031   1816 1969                           .86  .80  .83  .75 
   4 644041   1829 1950                                .87  .78  .75 
   5 644051   1801 1968                           .79  .71  .82  .84 
   6 644061   1881 1969                                          .70 
   7 644071   1882 1969                                          .92 
   8 644081   1886 1969                                          .76 
 Av segment correlation    NaN  NaN  NaN  NaN .86  .82  .78  .84  .80 

 PART 7:  DESCRIPTIVE STATISTICS: 
-----------------------------------------------------------------------------------------------------------------------------------

                                                Corr   //-------- Unfiltered --------\\  //---- Filtered -----\\
                           No.    No.    No.    with   Mean   Max     Std   Auto   Mean   Max     Std   Auto  AR
 Seq Series   Interval   Years  Segmt  Flags   Master  msmt   msmt    dev   corr   sens  value    dev   corr  ()
 --- -------- ---------  -----  -----  -----   ------ -----  -----  -----  -----  -----  -----  -----  -----  --
   1 644011   1681 1969    289      5      0     .834   .70   2.41   .433   .396   .603   2.80   .490  -.007   9
   2 644021   1795 1969    175      5      0     .844  1.02   2.96   .595   .454   .536   2.39   .465  -.001   6
   3 644031   1816 1969    154      4      0     .823  1.28   3.49   .685   .637   .403   1.88   .365  -.020   9
   4 644041   1829 1950    122      3      0     .788  1.94   6.08  1.179   .730   .365   2.13   .363   .026   4
   5 644051   1801 1968    168      4      0     .848  1.50   3.68   .764   .467   .474   2.62   .457   .003   1
   6 644061   1881 1969     89      1      0     .755  2.79   9.98  2.373   .847   .410   2.47   .431   .054   3
   7 644071   1882 1969     88      1      0     .890  2.32   5.45  1.253   .385   .552   2.70   .486   .031   1
   8 644081   1886 1969     84      1      0     .720  1.81   4.96  1.330   .756   .474   1.97   .410  -.080   6
 --- -------- ---------  -----  -----  -----   ------ -----  -----  -----  -----  -----  -----  -----  -----  --
 Total or mean:           1169     24      0     .813  1.67   4.88  1.076   .584   .477   2.37   .433   .001

  [1/1] nm046: 8 series, 0 problem segments
1 of 1 file(s) reported OK

10. Integrity checks with dpl.check¶

Crossdating asks whether every ring is on its correct year. dpl.check asks a different question: does the data make sense at all? It screens a collection for problems that are not dating errors and returns them as a table of findings, one row per problem.

Each finding carries:

• series -- the series it concerns, blank for a whole-collection or file-level finding. In crossdating you think series-first: which cores are fine and which need a look.
• check -- a stable id for the kind of problem, matching dplR's rwl.check ids, such as RWL_INTERNAL_NA or RWL_SERIES_OUTLIER.
• severity -- error, warning, or note, so you can triage at a glance.
• group -- the family the check belongs to: structure, series, values, zeros, crossdating, file, or provenance.
• year_from, year_to, n -- where the problem sits in the series and how many rings it spans.
• value -- a number attached to the finding, such as a correlation or a lag.
• message -- a plain-English description.

The checks cover missing values inside a series, duplicated cores, a constant value used as a not-measured placeholder (a real ITRDB hazard, like someone entering 1 for a ring that was never measured), implausible means or units, locally absent rings, file-format defects, and the crossdating screen itself.

check and the dating tools complement each other. Where xdate and series_corr locate a dating error, check flags the same series from a different angle. Run it on the collection with the simulated missing ring, and the corrupted series turns up as a poor-fit crossdating outlier.

In [24]:

findings = dpl.check(miss, file="co021 (simulated error)")
findings[findings["group"] == "crossdating"][["series", "check", "severity", "message"]]

findings = dpl.check(miss, file="co021 (simulated error)") findings[findings["group"] == "crossdating"][["series", "check", "severity", "message"]]

Out[24]:

	series	check	severity	message
0	644222	RWL_SERIES_OUTLIER	warning	correlates with the collection at r = 0.494 wh...

In [25]:

clean = dpl.check(RWL + "ca533.rwl")   # a well-formed file
clean[["series", "check", "severity", "year_from", "year_to", "message"]].head(8)

clean = dpl.check(RWL + "ca533.rwl") # a well-formed file clean[["series", "check", "severity", "year_from", "year_to", "message"]].head(8)

Out[25]:

	series	check	severity	year_from	year_to	message
0	CAM031	RWL_CONSECUTIVE_ZERO	note	1578.0	1580.0	3 consecutive absent rings, 1578-1580
1	CAM061	RWL_CONSECUTIVE_ZERO	note	1654.0	1655.0	2 consecutive absent rings, 1654-1655
2	CAM062	RWL_CONSECUTIVE_ZERO	note	1547.0	1548.0	2 consecutive absent rings, 1547-1548
3	CAM062	RWL_CONSECUTIVE_ZERO	note	1654.0	1655.0	2 consecutive absent rings, 1654-1655
4	CAM062	RWL_CONSECUTIVE_ZERO	note	1857.0	1858.0	2 consecutive absent rings, 1857-1858
5	CAM071	RWL_CONSECUTIVE_ZERO	note	1497.0	1498.0	2 consecutive absent rings, 1497-1498
6	CAM071	RWL_CONSECUTIVE_ZERO	note	1578.0	1579.0	2 consecutive absent rings, 1578-1579
7	CAM071	RWL_CONSECUTIVE_ZERO	note	1655.0	1656.0	2 consecutive absent rings, 1655-1656

In [26]:

clean.groupby(["severity", "check"]).size()   # how many of each kind, across the file

clean.groupby(["severity", "check"]).size() # how many of each kind, across the file

Out[26]:

severity  check               
note      RWL_CONSECUTIVE_ZERO    22
          RWL_LOW_DEPTH            1
          RWL_ZERO_RING            1
dtype: int64

Where to go next¶

That is the full crossdating toolkit: a collection-wide number (interseries_corr), a per-series test (xdate), a visual overview (xdate_plot), a per-series drill-down (series_corr), floating-series placement and rejoining (xdate_floater, combine_rwl), COFECHA emulation (preset="COFECHA"), batch reports (xdate_report), and a data-integrity screen (dpl.check).

• For the detrending step behind the indices, see the detrending and standardization guide.
• For how dplPy's crossdating compares with COFECHA and dplR, see the crossdating background note.
• For building chronologies from a crossdated, detrended collection, see the chronologies guide.