Most NZ teams trying AI for the first time make the same mistake: they buy licences, tell everyone it’s available, and wait for adoption to happen. Three months later, two people are using it enthusiastically and the rest have quietly gone back to how they always worked.
A well-designed AI pilot changes this outcome. Here’s how to run one.
What a Good AI Pilot Is (and Isn’t)
A pilot is not:
- Giving your whole team AI access and seeing what happens
- A vendor demo with a couple of follow-up sessions
- One enthusiastic person showing their colleagues cool tricks
A good pilot is a structured 4–6 week experiment with a specific group, specific use cases, defined success metrics, and a deliberate plan for what happens when it works.
Step 1: Choose the Right Team
Your pilot group should be 6–15 people. Big enough to generate useful data; small enough to manage properly.
Ideal pilot team characteristics:
- High writing volume — teams that produce a lot of documents, emails, or reports see the clearest early gains
- Willing participants — a pilot with resistant participants produces bad data. Start with people who are genuinely curious.
- A visible champion — someone in the group (ideally a respected peer, not just management) who will model the behaviour and talk about what’s working
- Representative of your broader workforce — if the pilot is all tech-savvy early adopters, the results won’t tell you much about wider rollout
Step 2: Define 3–5 Specific Use Cases
The biggest predictor of pilot success is use case specificity. “Use AI for your work” fails. “Use AI for these three specific tasks” works.
Good pilot use cases are:
- High frequency — tasks people do multiple times per week
- Currently time-consuming — there’s a meaningful before/after to measure
- Not high-stakes — avoid starting with tasks where AI errors have serious consequences
- Writing-adjacent — first drafts, summaries, correspondence, documentation
Examples by team type:
- HR: Job description drafting, interview question sets, employee correspondence
- Marketing: First draft blog posts, social media content, email campaigns
- Operations: Meeting minutes, procedure documentation, supplier communications
- Legal: Client correspondence drafting, research summaries, document review notes
- Finance: Management account commentary, client correspondence, policy documentation
Step 3: Build the Foundation Before Day One
Don’t give people AI access and then train them. Train them first — specifically, on the skill that determines output quality: context engineering.
Before the pilot starts, participants need to know:
- How to set up rich context (not just prompts)
- How to build reusable templates for the pilot use cases
- What not to put into AI tools (data privacy basics)
- How to evaluate whether AI output is good enough to use
A half-day workshop before the pilot period handles all of this. It’s the single highest-leverage investment in a pilot’s success.
Step 4: Measure from the Start
Decide what you’re measuring before the pilot begins. The simplest useful metrics:
- Task time: How long do the pilot use cases take before vs. during the pilot? Self-reported weekly is fine.
- Usage frequency: How many times per week is each participant using AI for the pilot tasks? (Not overall — specifically the defined use cases.)
- Output quality: Participants rate the quality of AI-assisted outputs vs. their normal outputs on a simple 1–5 scale.
- Confidence: Weekly pulse — “How confident are you using AI for your core work?” (1–10). Watch this trend upward.
A five-minute weekly check-in (Friday afternoons work well) captures all of this. Don’t make measurement burdensome or you won’t get honest data.
Step 5: Create Visibility
Adoption is partly social. Build in mechanisms that make AI use visible:
- A dedicated Slack/Teams channel where people share what’s working
- Brief weekly share-out (5 minutes in a team meeting) — “what worked this week?”
- A shared document where good prompts and templates get collected
The templates document becomes a library that outlasts the pilot. By week 4, it’s already more valuable than anything a vendor could give you.
Step 6: End with a Decision, Not a Drift
Week 6 is decision week. With 5 weeks of data, you can answer:
- Is adoption happening? (If not — why? Was it the use cases, the training, or the tool?)
- What are the time savings? Are they significant enough to justify broader rollout?
- What use cases worked best? What should the wider rollout focus on first?
- What do we need to do differently at scale?
A pilot that ends with a clear decision and a rollout plan is a success, even if the numbers were modest. A pilot that just… fades out is a waste of the goodwill you spent to run it.
The Most Common Pilot Failure Mode
Pilots fail most often not because AI doesn’t work, but because adoption never really happened — and nobody intervened early enough to find out why.
Build in a week 2 check-in with each participant. Ask: are you using it? What’s blocking you? Most problems in week 2 are fixable. By week 5 they’ve become habits of non-use that are much harder to shift.
GenAI Training NZ designs and delivers AI pilot programmes for NZ organisations. If you’d like help structuring your pilot — including the training component — get in touch. Or start with an AI Roadmap Workshop to establish your baseline.
Also see: Context Engineering | AI Training for Teams | AI for Leadership Teams
