Training 17: Reading the verification report and deviations

Before relying on a calculation type, find out what it was checked against. Everything below is in the app, under Settings > Verification, and on the public site.

Settings, Verification: every example and value with the app's result

The numbers, read carefully #

This release: 8,110 values compared in 2,262 examples, 1,867 of them published and 395 hand calculations from the cited equations: 7,714 within tolerance, 396 recorded deviations.

Those are counts of tests, not a grade for the program. Weigh them this way:

A published example is someone else's worked calculation. Agreement is real evidence, for the case that example covers. Most are single cases in one set of conditions, many written to an older edition.
A hand calculation is ours, from the cited equations. It catches arithmetic and programming errors. It does not catch a misreading of the code, because the same person read it twice.
A table cell read back (a load table, a shape table) checks that one value.
The percentage on the page mostly reflects how many easy comparisons exist for a type. Read the examples, not the percentage.

The statement in each chapter #

Every calculation chapter opens its Verification examples section with a status line. For the steel beam:

Verification status: 168 values compared in 37 examples (37 published, 0 hand calculations from the cited equations): 164 within tolerance, 4 recorded deviations. Code basis: partial. Only some provisions were read in the target edition; the rest rests on older editions, examples or summaries.

Code basis says how the module was written: read (the provisions it uses were read in the target edition), partial, older edition, from knowledge (no document on hand) or not determined (no record). ILX Studio plans to verify every module against the purchased references once the module list is complete.

Types with little behind them #

No published example, hand calculations only (6): Concrete stair, Gabion retaining wall, Gravity retaining wall, Masonry bearing, Segmental retaining wall, Wood stair.

Thin, 2 published examples or fewer (15): Basement (restrained) retaining wall, Combined footing, Corbel, Glulam and SCL beam, Guard or barrier post, Load chasedown, Mat foundation, Monorail beam, One-way slab, Post-tensioned slab strip, Prestressed masonry wall, Seismic site parameters and spectrum, Soldier pile wall, Strap footing, Unreinforced masonry wall.

Being on this list does not make a type wrong. It means less stands behind it, so check its results more closely.

Reading a deviation #

A deviation is a published value the program does not reproduce within tolerance, kept on the record with the reason. They are not all alike. On the Verification page, select the status Documented deviation and read each one. Ask of each:

Who differs, and which way? Real examples from this release's record:

Steel beam, F.8B. The example takes a simpler, conservative section; the program follows the specification's wording and reads a little higher: 46.3 against 45.4 kip-ft. A difference of method, stated.
Steel column, E.14A. The example rounds Lc/r up to 127 before finding the stress. Unrounded it is 126.6, so the program reads 52.8 against 52.3 kips. Harmless.
Strap footing. The one free example is a software output with unstated soil weights. None of its four pressures matches within tolerance: differences of up to 6 % that nobody can trace. That is a weak check, and the chapter says so.
Sheet pile wall, Fig. 4-43. The program's cantilever moment is 8 % below the printed value. It is on the unsafe side. It comes from equilibrium of the stated pressures, and no line-by-line example is on hand. If this case governs you, check it yourself: see the chapter.

The last one is the kind to look for: a program value on the unsafe side of a published one.

What verification does not cover #

Inputs outside the examples, combinations of options no example uses, and whatever a type's Limitations leave out. Passing every example does not make a calculation right for your structure. See Limits and your responsibility.

The whole method, with the tolerances, is in Verification.