AI training programs
don't fail on strategy.
They fail on execution.
I'm Lauren McDonald. I design the training systems, evaluation frameworks, and onboarding infrastructure that make AI data programs actually run. I've worked inside these programs across operations, quality management, and instructional design. I know where they break.
The person behind
the frameworks.
I came up through the operational side of AI data programs, moving across quality management, program operations, and eventually into instructional design and enablement. That path shaped how I approach this work. I don't start with slide decks. I start with what's actually breaking and work backwards from there.
My background in psychology shapes how I design. I think a lot about how people actually learn, where attention drops off, and what makes feedback land instead of sting. I pair that with a real appetite for the operational side: the data analysis, the process design, the systems thinking that makes training scale beyond one good session.
I founded Learning Craft AI because I kept seeing the same gap: organizations scaling AI programs fast without the training infrastructure to support them. This portfolio, including the live platform you can explore, exists because I think the work should speak before the resume does. If something here resonates, I'd love to talk.
What I Design
Original frameworks and sample documents representing the full range of what I build from contributor onboarding to QA calibration systems. Scroll down to read each one in full.
Week-by-Week Guide
Contributor Onboarding Playbook
A structured week-by-week program guide for AI contributor onboarding milestones, benchmarks, roles, and escalation design.
AI Evaluation Rubric Framework
Multi-dimension scoring rubric for RLHF and human evaluation dimensions, anchor examples, calibration protocol, and common scoring errors.
Training Gap Analysis Framework
Structured methodology for identifying and prioritizing training gaps current vs. target state mapping, root cause classification, and remediation roadmap.
QA Calibration SOP
End-to-end SOP for running calibration sessions session design, IRR tracking, facilitator checklist, and escalation protocol.
Learning Craft AI Course Platform
A fully designed and deployed gamified learning platform XP system, badge framework, admin course builder, and leaderboard, built from scratch.
B2B Consulting Learning Craft AI
Strategic and hands-on consulting for organizations building AI contributor programs from audits to full program builds.
Contributor Onboarding Playbook
A structured week-by-week guide for onboarding contributors to AI data programs covering orientation, workflow training, quality benchmarks, and milestone checkpoints.
This playbook supports consistent, scalable onboarding for contributors joining AI data programs. It provides a structured pathway from day-one orientation through full independent performance, with clear milestone checkpoints and quality expectations at each stage. It is intended for program managers, operations leads, and instructional designers responsible for contributor ramp-up.
Platform orientation, program documentation review, introduction to core task types. Focus on understanding why evaluation work matters and how contributor quality connects to model performance.
In-depth rubric training with anchor examples and practice sets. Live calibration sessions to align scoring understanding and establish inter-rater reliability baselines.
Live task work under supervised conditions. Individual feedback focused on error patterns, quality consistency, and escalation behavior.
Independent work against SLA and quality benchmarks. Formal milestone review at end of Week 4. Contributors who don't meet benchmarks receive a targeted remediation plan.
| Milestone | Timing | Success Criteria | If Not Met |
|---|---|---|---|
| Platform & orientation complete | End of Week 1 | 100% module completion and documentation review | Extended access + manager check-in |
| Rubric calibration baseline | End of Week 2 | IRR ≥ 0.70 on practice sets; live calibration attendance | Additional practice + 1:1 rubric coaching |
| Supervised practice quality gate | Mid Week 3 | Quality score ≥ threshold; error rate below program limit | Targeted feedback loop + supervised extension |
| Independent readiness review | End of Week 4 | Sustained quality and throughput at benchmarks over 5-day window | Formal remediation plan with defined timeline |
Contributors should be trained on when and how to escalate. Clear escalation paths reduce inconsistent decision-making and help capture edge cases that improve rubric documentation over time.
© Lauren McDonald · learningcraftai.com · Sample not for redistribution or reuse
Portfolio · Private Share Only
AI Evaluation Rubric Framework
A multi-dimension scoring framework for RLHF and human evaluation dimensions, scoring scale, anchor examples, calibration protocol, and common errors.
| Score | Label | Definition | When to Use |
|---|---|---|---|
| 4 | Excellent | Fully meets expectations with no meaningful weaknesses | You would not change anything about this aspect |
| 3 | Good | Mostly meets expectations; minor issues that don't significantly impact quality | Small improvements possible but response is generally solid |
| 2 | Fair | Partially meets expectations; noticeable weaknesses affect usefulness | Clear problems but still contains something valuable |
| 1 | Poor | Fails to meet expectations; significant problems present | Wrong, harmful, incoherent, or completely misses the mark |
© Lauren McDonald · learningcraftai.com · Sample not for redistribution or reuse
Portfolio · Private Share Only
Training Gap Analysis Framework
A structured methodology for identifying, mapping, and prioritizing training gaps from data collection through remediation planning.
Current performance vs. target benchmarks across five quality dimensions. Gap size and impact drive prioritization order.
© Lauren McDonald · learningcraftai.com · Sample not for redistribution or reuse
Portfolio · Private Share Only
QA Calibration SOP
End-to-end SOP for running calibration sessions session design, IRR tracking, facilitator checklist, and escalation protocol for AI evaluation programs.
Calibration ensures contributors are interpreting and applying rubrics consistently. Without it, scoring drift occurs reviewers develop slightly different mental models of what a score means, and inter-rater reliability declines. This SOP defines how sessions should be structured, run, and documented.
| Type | Audience | Frequency | Duration |
|---|---|---|---|
| Onboarding Calibration | All new contributors before independent work begins | Once, during Week 2 | 60–90 min |
| Ongoing Calibration | Active contributors on all programs | Weekly or biweekly | 30–45 min |
| Drift Correction | Contributors flagged for IRR below threshold | Within 5 business days of flag | 30 min 1:1 |
| Rubric Update Calibration | All active contributors on rubric change | Within 48 hrs of update | 45–60 min |
| Level | Threshold | Action if Below |
|---|---|---|
| Cohort | ≥ 0.75 | Run additional session within the week; review rubric for clarity issues |
| Individual contributor | ≥ 0.70 | Schedule 1:1 drift correction; provide targeted anchor examples for divergent dimensions |
| Individual repeated failure | 2 sessions below 0.65 | Escalate to program manager; suspend independent access pending remediation plan |
© Lauren McDonald · learningcraftai.com · Sample not for redistribution or reuse
Portfolio · Private Share Only
Learning Craft AI
Course Platform
A fully designed and deployed gamified learning platform built from scratch — demonstrating instructional design principles inside a real, working system. Browse the screenshots below then launch it yourself.
The demo is open — no account needed. Use the credentials below to explore the full learning experience including lessons, knowledge checks, and the XP system.
Launch Demo →