Scenario — Evidence to Offer

Part 2 of the Scenario series picks up exactly where your job analysis ended. The SecOps Analyst role you defined is now live: 1,800 people applied, 180 were hired, and those hires are being rated and paid. Your job in this app: audit the hiring system (adverse impact, validity, reliability, incremental validity, utility, decision errors), then follow the chain one link further — build the BARS that rates the people you hired, and connect those ratings to compensation. That last stretch is a full rehearsal of your assignment.

Quick stats

Meridian

Meridian is a fictional cybersecurity SaaS company (~1,200 employees) hiring for a Security Operations Analyst role. The selection system is a multi-hurdle compensatory model:

How to use this app

• Hover any dotted-underlined term for definition + formula + cautions.
• Press ⌘K (or Ctrl-K) to open the command palette and jump anywhere.
• Press ⌘1–⌘9 (or Ctrl-1–9) to switch tabs from the keyboard.
• Click any dotted-underlined term to jump to its full glossary entry — a “Back” button returns you to exactly where you were.
• Use the simulators to manipulate inputs and watch outputs update live.
• Each simulator has a Reset button to restore default Meridian data.

Key terms (hover for definitions)

This app covers ValidityValidityThe degree to which evidence and theory support inferences from test scores.r_xy = cov(X, Y) / (σ_X · σ_Y)Watch: Concurrent designs underestimate operational validity due to range restriction. Always correct (Thorndike Case II/III)., ReliabilityReliabilityConsistency of measurement; the proportion of observed score variance attributable to true score variance.α = (k/(k-1)) · (1 − Σσ²_i / σ²_total)Watch: α assumes tau-equivalence; for congeneric measures, use McDonald's ω., Adverse impactAdverse impactA substantially different rate of selection that works to the disadvantage of members of a protected class.IR = (hire rate_focal) / (hire rate_reference)Watch: 4/5 rule and statistical tests can disagree. Small n inflates 4/5 false positives; large n inflates statistical false positives. Report both., utility (BCG)Utility (BCG model)Dollar value gained from a selection system, relative to a baseline procedure.ΔU = N·T·SR_z·r·SD_y − N·CWatch: SD_y estimation: 40%-of-salary heuristic (Schmidt et al.), global estimation, CREPID. Always run sensitivity analysis., Incremental validityIncremental validityThe unique variance in the criterion explained by a predictor beyond predictors already in the model.ΔR² = R²_full − R²_reducedWatch: Order matters in hierarchical regression — typically enter from cheapest/most-validated to most-expensive/novel., Spearman-Brown prophecySpearman-Brown prophecyPredicts reliability when test length (or rater count) is increased by a factor k.ρ_kk = kρ / (1 + (k−1)ρ)Watch: Diminishing returns: doubling raters from 4 to 8 buys less than doubling from 1 to 2., Selection ratioSelection ratioProportion of applicants hired (n hired / n applied). Lower SR = more selective = greater utility, ceteris paribus.SR = n_hired / n_appliedWatch: Combined with low base rate of success, utility may not justify the assessment cost — Taylor-Russell tables show this region., Range restrictionRange restrictionReduction in predictor variance because only selected applicants enter the criterion sample, biasing r downward.ρ = r · (S_X/s_X) / √(1 − r² + r²·(S²_X/s²_X))Watch: Concurrent validity studies on incumbents are especially restricted; predictive studies on applicants less so., Differential predictionDifferential predictionWhen a predictor's regression slope or intercept differs across subgroups, leading to systematic over- or under-prediction.Y' = a + b·X (test if a or b differs by group)Watch: Power to detect slope differences is typically < 0.30 in field studies; pool data or use Bayesian methods., and other graduate I/O psychometric constructs. The full glossary is on the Glossary tab.

Adverse Impact

Adverse impact analysis asks whether a selection procedureDecision ruleThe procedure for combining assessment scores into a hire/no-hire decision.Composite = Σ w_i · z_iWatch: Unit weights perform robustly when sample sizes are small (Dawes, 1979); regression weights overfit. produces substantially different selection rates across protected classes. The Four-fifths ruleFour-fifths ruleEEOC heuristic: a focal group's selection rate below 80% of the reference group's rate suggests adverse impact.Pass if (IR ≥ 0.80) for every focal groupWatch: With small subgroup n, the IR is high-variance — don't act on one quarter's data alone. is a screening heuristic, not a legal definition. Pair it with Fisher's exact testFisher's exact testExact test of independence in a 2×2 contingency table; appropriate when expected cell counts are small.P = Π C(n_i, k_i) / C(N, K)Watch: Two-tailed vs one-tailed: courts typically expect two-tailed for fairness inquiries. for small samples and the 2-SD test (per Hazelwood) for larger ones.

Meridian audit finding

Two subgroups failed 4/5: Asian (IR=0.60, n=294) and Non-binary (IR=0.58, n=47). Only the first is actionable — the second has too small n for stable inference. Fisher exact p for age (the deliberately-baked-in issue) = 0.43 — not significant despite the bias, illustrating that the 4/5 rule and statistical tests answer different questions.

Reading the three tests — in plain language

Interactive simulator

Manipulate the focal and reference group hire rates and sample sizes. Watch how the 4/5 ratio, Fisher exact p, and 2-SD z move together — or apart. Try to find configurations where they disagree.

Validity

Validity is the central concept in selection psychometrics. The Standards (AERA/APA/NCME, 2014) treat content, criterion, and construct evidence as types of evidence, not types of validity. The composite predictor in Meridian shows r = 0.54 with manager-rated 6-month performance — attenuated by range restrictionRange restrictionReduction in predictor variance because only selected applicants enter the criterion sample, biasing r downward.ρ = r · (S_X/s_X) / √(1 − r² + r²·(S²_X/s²_X))Watch: Concurrent validity studies on incumbents are especially restricted; predictive studies on applicants less so. because only hires enter the criterion sample.

Range restriction, in plain language

Validity by predictor

How to read this table

What the numbers say, stage by stage

Reliability

The Meridian composite shows Cronbach's αCronbach's αInternal-consistency reliability estimator; lower bound under tau-equivalence.α = (k/(k-1)) · (1 − Σσ²_i / σ²_T)Watch: α > 0.70 acceptable, > 0.80 good, > 0.90 excellent — but very high α (> 0.95) may indicate redundancy. = 0.88. Single-rater interview r = 0.49; with two raters and Spearman-Brown prophecySpearman-Brown prophecyPredicts reliability when test length (or rater count) is increased by a factor k.ρ_kk = kρ / (1 + (k−1)ρ)Watch: Diminishing returns: doubling raters from 4 to 8 buys less than doubling from 1 to 2., this rises to 0.65.

Interactive simulator

How many raters do you need? Move the single-rater r slider and watch the Spearman-Brown curve update. The diminishing-returns pattern is the central insight: doubling raters from 1→2 has much greater payoff than 4→8.

Incremental Validity

Incremental validityIncremental validityThe unique variance in the criterion explained by a predictor beyond predictors already in the model.ΔR² = R²_full − R²_reducedWatch: Order matters in hierarchical regression — typically enter from cheapest/most-validated to most-expensive/novel. quantifies the unique variance explained by each new predictor beyond predictors already present. ΔR² depends both on the predictor's zero-order validity AND its uniqueness (lack of overlap with the existing battery).

Meridian hierarchical regression

Interactive simulator

Toggle predictors on and off. Watch the composite R² rebuild from scratch. Notice that the FINAL R² with all predictors active is invariant to entry order — but the ΔR² PATH depends entirely on order.

Utility (Brogden-Cronbach-Gleser)

The BCG modelUtility (BCG model)Dollar value gained from a selection system, relative to a baseline procedure.ΔU = N·T·SR_z·r·SD_y − N·CWatch: SD_y estimation: 40%-of-salary heuristic (Schmidt et al.), global estimation, CREPID. Always run sensitivity analysis. translates validity and selection ratio into dollar value. With composite r = 0.54, SR = 0.10 (mean z_selected = 1.75), SD_ySD_yStandard deviation of job performance expressed in dollars; the BCG utility model's most contested input.≈ 0.40 × annual salary (heuristic)Watch: Report a range, not a point estimate. Sensitivity-test at 0.20× and 0.60× salary. = $30K, and tenure = 3.2 years, Meridian's system delivers ≈ $16.3M/year vs random, ≈ $10.3M/year vs an unstructured-interview baseline.

A worked example, in plain language

Interactive simulator

The model is sensitive to its inputs. Use the sliders to test how ΔU responds to validity, selection ratio, SD_y, tenure, hire volume, and cost. The rule of practice: report a range, not a point.

False Positives & False Negatives

At Meridian's 10% selection ratio, the system achieves 99% precision but only 20% recall. That's not a flaw — it's the mathematical consequence of selectivity. The optimal cutoff depends on asymmetric FP / FN costs.

Meridian confusion matrix

Reading the four boxes (one year of Meridian hiring: 1,800 applicants, a 10% bar, and — for teaching purposes — perfect hindsight about who would have succeeded):

178 true positives — hired, and would succeed. 2 false positives — hired, but would fail. Only 2 misses among 180 hires is the 99% precision from the headline: the strict bar almost never lets a bad hire through.

722 false negatives — rejected, yet would have succeeded. This is the price of that strict bar: 900 applicants would have succeeded, and a 10% selection ratio only had room for 178 of them — the 20% recall.

898 true negatives — rejected, and would have failed. The quiet box where the system did its job.

The simulator below is the live version of this table — drag the cutoff and watch people flow between the four boxes.

Interactive simulator

Move the cutoff slider and watch the four cells re-distribute. Set FP and FN costs to reflect realistic hiring scenarios — bad-hire cost in a security role vs missed-talent cost in a shortage market — and find the cost-minimizing cutoff.

BARS Lab — From Critical Incidents to a Rating Scale

In the job analysis you sorted eight critical incidents onto 1–5 effectiveness anchors. You may not have realized it, but that sorting was the first step of building a BARSBehaviorally Anchored Rating Scale (BARS)A performance rating scale whose levels are defined by concrete, observable job behaviors derived from critical incidents (Smith & Kendall, 1963), rather than trait adjectives.CIT incidents → retranslation → scaled anchors (1–5)Watch: Anchors must describe behavior, not traits. “Poor attitude” is unratable; “leaves shift without documenting open incidents” is not.. Here you finish the job: assemble a five-level BARS for the SecOps Analyst's Teamwork & Communication dimension, stress-test your anchor writing, and then use the scale to rate three real vignettes.

Why BARS — and why it must come from the job analysis

BARS (Smith & Kendall, 1963) replaces trait adjectives (“dependable,” “team player”) with observable behaviors scaled from ineffective to exemplary. Because the anchors are derived from critical incidents collected during job analysis, a BARS inherits the JA's content validity: every anchor is traceable to documented job behavior. That traceability is what makes appraisal-based decisions — merit pay, promotion, termination — defensible under the same UGESPUniform Guidelines (UGESP, 1978)29 CFR §1607. Applies to any procedure used as a basis for an employment decision — including performance appraisals that drive pay and retention, not just hiring tests.Watch: Courts have treated appraisals that feed layoff or pay decisions as “selection procedures.” JA-grounded anchors are the defense. logic you used to audit the hiring system.

Anatomy of a good anchor

Each level of a BARS must describe the same behavior domain at increasing effectiveness, using observable actions. The classic failure modes:

Step 1 — Rebuild the scale

The SME panel wrote seven anchors for Teamwork & Communication (SecOps Analyst), then shuffled them. Assign each anchor the level you think it describes, 1 (Unsatisfactory) to 5 (Exemplary). Levels can repeat — a real BARS pools several incidents per level, so two anchors can both land on a 3. If two anchors feel indistinguishable, that itself is a finding: ambiguous anchors get rewritten.

Step 2 — Write one anchor yourself

Part A of the assignment asks you to write five anchors, not sort them. Warm up here: draft a Level 3 (Meets Expectations) anchor for Teamwork & Communication. The checker — programmers call this a “linter,” a tool that scans a draft for common flaws the way spell-check scans for typos — flags the classic pitfalls. It is a coach, not a grader.

Step 3 — Apply your BARS to three vignettes

Now use the scale. Rate each SecOps analyst 1–5 and write a 2–3 sentence rationale that cites behaviors from the vignette matched to anchor language — exactly what Part B of the assignment requires. Commit a rating and rationale before revealing the calibration panel's answer.

Where this goes next

Pay Link — Compa-Ratio & Merit Decisions

A rating that goes nowhere is theater. This tab connects your BARS ratings to compensation using the compa-ratioCompa-ratioAn employee's pay expressed as a fraction of the pay-range midpoint for the role. The standard diagnostic for position-in-range.CR = current pay ÷ range midpointWatch: A compa-ratio is only as meaningful as the midpoint behind it — midpoints must come from defensible market pricing of the JA-defined role. method and a merit adjustment framework — the exact machinery of the assignment Part C.

Compa-ratio in one minute

Every role has a pay range built around a market-anchored midpoint — the target rate for a fully proficient performer. The compa-ratio locates a person in that range: CR = pay ÷ midpoint. CR ≈ 1.00 means at market; below ~0.90 suggests underpayment (or a new hire still growing into the role); above ~1.10 suggests premium pay that should be justified by sustained premium performance. Merit decisions read rating and compa-ratio together: the same rating earns a different increase depending on where the person already sits in the range.

Interactive simulator

The merit adjustment framework

Meridian uses the same framework your assignment provides (Table 4). Ratings gate the increase band; compa-ratio informs where in the band to land.

Merit worksheet — the three analysts you rated

Your BARS Lab ratings carry over (editable below). For each analyst: compute the compa-ratio yourself, then choose a merit % inside the band the framework allows for that rating. New salary and new compa-ratio update live. SecOps midpoint: $85,000.

Before any of this touches a paycheck: the four fairness checks

Part D of the assignment asks four reflection questions. They are not decoration — each one names a real failure mode of merit-pay systems. Meet them here first:

Mission Hand-Off — From Ratings to Raises

This is the bridge out of the simulation and into your assignment, “Applying BARS and Compensation Analytics.” Everything you practiced on the SecOps Analyst you will now do solo on the Electrical Engineering Technician (EET) case that accompanies your assignment. Same method — new role. That transfer is the point.

Your three-act arc

The assignment, decoded

Deliverable: an informative 10–12 slide PowerPoint plus a Word document with your speaker notes. Four parts, each rehearsed in this app:

The EET numbers you'll work with

From the assignment (Table 3): role midpoint $55,000.

Recognize these three? They are behavioral cousins of SA-01, SA-02, and SA-03. Your ratings may or may not transfer — read the EET vignettes on their own evidence.

Self-check: your EET compa-ratios

Compute each compa-ratio to 2 decimal places and verify before it goes on a slide. This checks your arithmetic only — it never shows the answer.

A 10–12 slide blueprint that earns its length

Part D warm-up (exports with your launch kit)

Draft two or three sentences on the question you find hardest: calibration before pay, uncalibrated-rating risk, pay-equity monitoring, or transparency. Starting now makes slide 10 an edit instead of a blank page.

Take your work with you

Exercises

Seven scripted exercises pairing with the simulators and the Part-2 modules. Work through these as you go; your instructor will discuss solutions.

Glossary

Graduate-level reference. Every term shown as a hover-tooltip elsewhere in the app is detailed here with its formula and key cautions.