A pilot is one of the few genuine ways to learn how a product behaves in your environment, with your actual faculty and students, before you commit to it. It is also remarkably easy to run a pilot that produces nothing but a warm feeling: a handful of enthusiastic early adopters try a tool, report that they liked it, and the pilot becomes a formality on the way to a decision that was already made.
A pilot designed to produce a real signal looks different. Here is how to set one up.
Start with the actual question
Before anything else, write down the specific thing the pilot is meant to answer. Not “should we buy this,” but something narrower and genuinely testable. Can our identity team integrate this in the time the supplier estimates? Do faculty outside the early-adopter group find the grading workflow usable without one-on-one hand-holding? Does the accessibility of the authoring interface hold up when a non-specialist builds a course in it? Does the tool reduce the time instructors spend on a task, or does it just move that time somewhere else?
The question you write down determines everything else about the design.
Choose the courses on purpose
The instinct is to recruit volunteers, and volunteers skew toward faculty already comfortable with new tools and motivated to make them work. That tells you the best-case scenario, which you may already have guessed.
A more informative pilot includes real range: a couple of enthusiastic adopters, a couple of capable-but-skeptical faculty, and a course or two resembling the hard cases. High enrollment, a lab or studio format, a discipline with specific needs. Pay participants for the extra work, which also earns you the right to ask more of them in feedback.
Support it the way the real thing would be supported
If the pilot gives participants a direct line to a supplier engineer and daily hand-holding, you are testing a product plus a white-glove service that will not exist once you are at scale. Support the pilot the way the institution would genuinely support the product, the normal help desk, the normal documentation, so the result reflects the real deployment, not a showcase version of it.
Decide what you will measure, and how
For each pilot question, name the actual evidence, for integration and configuration: real hours logged by your staff against the supplier’s estimate, plus the issues encountered and how they got resolved, for faculty experience: a short structured survey and a debrief conversation, from all participants, not just the ones who volunteer feedback unprompted, for student experience: a brief survey in the pilot sections, plus support tickets related to the tool, for accessibility: an actual test of the parts faculty and students touched, with real assistive technology, for a specific claim, say, whether it reduces grading time, ask participants to estimate before and during, and be honest that it is still an estimate.
Set the length and the ending in advance
A term is usually the right length. Long enough for the novelty to wear off and for the tool to face a real grading cycle, short enough to inform a decision on schedule. Decide in advance what result would mean yes, what would mean no, and what would mean a second look. Writing that down before the pilot runs is what keeps the eventual conclusion honest.
Read it as one input, not the whole answer
A pilot answers what a demo and reference calls cannot. How the product behaves here. It does not answer everything. Cost, contract terms, long-term supplier direction, and fit with your governance process are separate lines of inquiry entirely. A well-run pilot is one strong input into a fuller evaluation, and its value lies in how specific it is, not in a general sense of whether people seemed to like it.
Jack Wrightmann is a consultant at Wrightmann Education Technologists. Part of the Technology Procurement series.
References
- EDUCAUSE. (2025). Higher Education Community Vendor Assessment Toolkit (HECVAT). https://www.educause.edu/higher-education-community-vendor-assessment-toolkit
- 1EdTech Consortium. Learning Tools Interoperability (LTI) and OneRoster. https://www.1edtech.org/standards