Comparison of Calibration/Verification v5.0 to Calibration/Validation v3.4.
Scott M. Duke-Sylvester
E. Jane Comiskey
The Calibration/Verification run (CalVer v5.0), produced by
the South Florida Water Management Model (SFWMM) version 5.0, differs in several
the Calibration/Validation run (CalVal v3.4) from version 3.4 of the SFWMM.
There has been a shift in terminology from Validation to Verification;
however these two terms are used interchangeably in this discussion.
The most notable difference is the inclusion of the years from 1996 through
2000 and the omission of the years 1979 and 1980 in CalVer v5.0.
Because of this difference, the comparison provided here addresses only the period
of the overlap between the two models (1981 -- 1995).
The scenarios also differ in how they partition the years into calibration and verification (or
In the CalVal v3.4, 1979 through 1990 are calibration years, while 1991 through 1995 are
Calibration years are those years of gauging station data used as a basis for
parameter estimation within the SFWMM.
The Validation years are those years of gauging station data used to assess the
performance of the SFWMM.
In CalVer v5.0, the calibration phase is divided into two segments, the first from
1984 through 1990 and the second from 1991 through 1995.
While these periods are contiguous in time, their division reflects a shift
in water management practices.
A more detailed explanation of this division is provided by the SFWMD.
The verification phase in SFWMM v5.0 is also divided into two segments, the first
from 1981 through 1993 and the second from 1996 through 2000.
This comparison will focus on the divisions used in CalVer v5.0.
CalVer v5.0 and CalVal v3.4 both provide spatially explicit estimates of water
depth on a daily basis for approximately 1600 2x2 mile cells.
This results in approximately 8.7 million water depth values for each scenario.
Comparing such large structured data sets is challenging.
Data need to be summarized in some way to make the comparison tractable and
to make patterns in the data apparent.
Summarizing the data necessarily emphasizes one aspect of the data over another.
A particular approach can emphasize the spatial or temporal aspect of the data,
and can also emphasize patterns at different spatial or temporal scales.
To avoid forming a biased impression of the data, the data should be analyzed in
several different ways.
However, a very large number of alternative data comparisons contributes
little to understanding.
We compare CalVer v5.0 and CalVal v3.4 using three approaches that provide
useful insights into the differences between these data.
First, we compare the scenarios using mean annual ponding depths computed for the
entire study area.
That is, for each year (1981 -- 1995) we compute the mean ponding depth across
space and time.
Each annual average is the mean of approximately 1600 2x2 mile cells over
365 days of data (366 for leap years).
The time series of means are plotted in Figure 1.
Mean ponding for CalVer v5.0 is plotted in green, mean ponding for CalVal v3.4 is plotted
This approach summarizes spatial and temporal variation in water depths within each year
and highlights inter-year differences between the two scenarios.
Mean ponding depths range between 50 mm and 275 mm.
Based on the mean ponding water depths CalVer v5.0 is drier than CalVal v3.4 for
most years between 1981 and 1995.
In only three years (1983, 1984 and 1995) is CalVer v5.0 wetter than CalVal v3.4.
Both scenarios show a cycle of increased ponding followed by a decrease in ponding
from 1981 to 1985.
This is followed by another cycle of increasing and decreasing ponding depth from
1985 to 1989.
Finally, there is a general increase in mean ponding depth from 1989 through 1995 (See Figure 1).
Generally the two scenarios track each other well through these patterns.
This pattern is not especially surprising since the two scenarios are calibrated and validated using
the same or similar gauging station data.
Percent difference ( computed as ABS( CalVerv5.0 - CalVerv3.4))/CalVerv3.4 * 100 and
plotted in blue in Figure 1 ) ranges between 0.5% and 31%.
In seven of the fifteen years (1984 -- 1986, 1989 -- 1992) the difference between the two
scenarios is greater than 10%, with the largest difference, 31%, occurring in 1990.
The second approach to comparing scenarios applies linear regression to provide
insight into the difference between data sets.
In this approach, water depth data from each scenario are paired based on
location (2x2 mile cell) and plotted against each other.
One such scatter plot is created for each day, resulting in 5478 scatter plots, each displaying
approximately 1600 data points.
For each scatter plot, we compute the slope, intercept and r2 value for a
least squares line fitted to the data.
Linear regression is used to summarize daily spatial differences between the two scenarios.
The r2 values characterize the level of correlation between the two data sets
and determines whether or not the slope and intercept provide additional information.
The magnitude of the slope provides information about the difference in
variation in water depths between the two scenarios, while the sign indicates
whether or not the scenarios vary directly or inversely.
The intercepts reflect any overall shift in water depth between the
The slopes, intercepts and r2 values are computed for
each day and are plotted in separate time series graphs (Figures 2-4).
The time series graphs allow us to see the temporal variation in the differences
between the two scenarios on a daily basis.
In contrast to the other two approaches used here, this approach allows us to
look at intra-year variation in the differences between the two scenarios.
In the analysis reported here, CalVal v3.4 data are arbitrarily assigned to the
x-axis and the CalVer v5.0 data are assigned to the y-axis.
The r2 values provide an estimate of the amount of variation in the scatter plots
that is explained by a linear model.
The r2 values for the comparison between CalVer v5.0 and CalVal v3.4 range between
0.83 and 0.96 with a mean of 0.92, standard deviation of 0.019 and are slightly
negatively skewed (Figure 5).
In the time series plot, a strong seasonal cycle of increasing and
decreasing r2 values is apparent, with higher values occurring in the dry season and
lower values occurring in the wet season.
This indicates that the scenarios tend to be more similar during the
dry season than in the wet season.
No long term trend in r2 values is apparent.
The relatively large r2 values indicate that a strong linear
relationship between the two scenarios and supports the meaningful
interpretation of slopes and intercepts.
Slope values for the scatter plots range between 1.01 and 0.82 with a mean
of 0.93, standard deviation of 0.027 and are slightly negatively skewed (Figure 6).
A time series plot of the slopes (Figure 3) shows a short term seasonal cycle
in slopes, with lower slopes during the wet season and higher values (nearer to
1.0) during the dry season.
Some longer term (2-3 year) cycles of increased and
decreased slopes are seen, but these
cycles are weak and do not follow a readily
recognizable pattern over
the period of comparison.
Positive slope values near one indicate a similar level of variation in the
Slopes greater than one indicate greater variation in water depths
in CalVer v5.0 relative to CalVal v3.4.
Slopes less than one indicate less variation in water depths in CalVer v5.0 relative to
Negative slopes would indicate an inverse relationship between the two scenarios.
Intercept values for the comparison of CalVer v5.0 and CalVal v3.4 range between
-140.6 mm and 102.9 mm, with a mean of -12.54 mm and a standard deviation of 44.84 mm (Figure 7).
Negative intercept values indicate a shift toward lower water depths in CalVer v5.0
relative to CalVal v3.4, while positive intercepts indicate a shift toward
deeper water levels in CalVer v5.0.
Variation in the intercept with time shows a seasonal cycle, with
dry season intercepts being larger than wet season intercepts.
During the periods 1981 -- 1985, 1986 -- 1989 and 1990 -- 1995
a pattern of increasing intercepts is seen, with a sharp return to lower intercept values
These trends of increasing intercepts indicate that CalVer v5.0 water levels
become progressively deeper relative to CalVal v3.4, followed by a sharp return
to lower water levels.
The final comparison of CalVer v5.0 and CalVal v3.4 is based on a visual examination
of the spatial variation in mean annual water depths.
In this comparison, each scenario is summarized by computing
mean annual water depths for each 2x2 mile cell to create
One map layer is subtracted from the other to create a third layer
that represents the difference in mean annual water depth between
the two scenarios.
These three map layers are presented together in the standard ATLSS
three panel map format for each year (Figures 8-22),
allowing us to see spatial patterns of increased or decreased water depth.
In each three panel map, CalVer v5.0 appears in the left hand panel, CalVal v3.4 appears
in the right hand panel.
The difference is computed as
(CalVer v5.0 - CalVal v3.4) and appears in the center panel.
In the left and right hand panels, each 2x2 mile cell is color-coded
based on the local mean annual water depths.
Dark blue to green colors are used in 2x2 mile cells where mean annual water depths
are negative (below ground surface), yellow represents depths at or near zero, and
orange to red colors represent positive (above ground surface) mean annual water depths.
In the difference map, colors from dark cyan to white
indicate areas where differences are negative, while colors from white to yellow
indicate areas where differences are positive.
Negative differences indicate locations where CalVer v5.0 mean annual water levels
are lower relative to CalVal v3.4, positive differences indicate locations where
CalVer v5.0 water levels are relatively higher.
This comparison emphasizes the spatial pattern of inter-year differences between the
In comparing the spatial pattern of differences between the two scenarios we will
focus on the ATLSS study area.
The ATLSS study area includes the Everglades National Park (ENP),
Big Cypress National Preserve (BCNP) the Water Conservation Areas (WCA-2, WCA-3),
Loxahatchee N.W.R. and surrounding natural areas.
We omit any discussion of differences in the urban areas along the east coast of
south Florida, and the agricultural areas (Everglades Agricultural Area or EAA)
to the south of Lake Okeechobee.
Starting in 1981, CalVer v5.0 has lower mean annual water depth relative to
CalVal v3.4 for most areas and higher water depths
for a few scattered areas including portions of Long Pine Key in the Everglades
National Park (ENP), the northernmost portion of Water Conservation Area (WCA) 3
and most of WCA-2A.
The years from 1982 through 1985 are similar to 1981, with lower CalVer v5.0 water depths
for most areas.
Higher water levels are seen under CalVer v5.0 in only a few areas.
However, negative differences tend to become progressively smaller over these years, and
positive differences between the scenarios are seen in more locations.
In the following year, 1986, differences between
the scenarios are negative in most areas, indicating a return to lower CalVer v5.0 water depths relative to
CalVal v3.4, with positive water differences in a few scattered locations.
For the following two year, the magnitude of negative water differences
decreases again, and positive differences occur in a larger number of locations.
In 1989 and 1990, negative differences again dominate most the study area,
with only a few scattered locations having positive water differences.
Over the following years, negative differences appear to be come smaller in
magnitude, and positive differences are found more frequently.
This pattern mirrors closely the pattern of increasing intercepts observed
in Figure 4.
In addition to the overall pattern of differences described above there are a
number of patterns at smaller spatial extent.
During all years compared (1981 -- 1995), water depths in CalVer v5.0
in the area around I-75 in WCA-3A are lower relative to CalVal v3.4.
During the period 1986 through 1995, mean annual water depths in A.R.M. Loxahatchee
National Wildlife Refuge (Loxahatchee) are lower in CalVer v5.0 relative to CalVal v3.4.
The comparisons presented here indicate that CalVer v5.0 and CalVal v3.4 differ
in a number of ways.
Based on mean annual ponding (Figure 1) there are several years in which
the scenarios differ by 10% to 30%.
This means that over the entire study water depths are between 16mm and 26 mm
deeper (or shallower, see figure 1 for direction) in CalVer v5.0 relative to CalVal v3.4.
This can have a significant effect on ATLSS models sensitive to changes in water depths.
Differences between the CalVer v5.0 and CalVal v3.4 are also seen in the time series plots in
First, differences between the scenarios tend to be larger during the wet season.
As a result, model predictions based on these two scenarios for
species that respond to wet season hydrology patterns are more
likely to be different than those for other models.
This point is important since the seasonal signal can contribute to
the magnitude of differences in model predictions based
on these two scenarios.
Second, there appears to be a consistent pattern of increased water depth for CalVer v5.0,
relative to CalVal v3.4.
We would expect, therefore, to see a commensurate pattern in model predictions for most or
all of the ATLSS models.