How to Evaluate an ML Engineer When You Are Not Technical

You can evaluate an ML engineer without being able to read their code, as long as you evaluate the right things. Founders and operators hiring their first AI engineer usually try to judge technical depth directly, lose, and fall back on credentials and confidence. Both are weak predictors. What predicts success in an applied machine learning role is a short list of behaviors that a non-technical interviewer can verify: production ownership, clear reasoning about trade-offs, honesty about failure, and the ability to explain their work to you. This guide, built on Vintti AI's screening process for dedicated AI/ML engineers, gives you a script for all four.

Why Non-Technical Evaluation Usually Fails

Most non-technical hiring managers evaluate the wrong layer: vocabulary instead of outcomes. A candidate who talks fluently about transformers, embeddings and fine-tuning sounds impressive and may never have kept a model running in production for a month. The inverse profile, quieter and more concrete, is usually the better hire.

Three failure patterns show up again and again in searches Vintti AI supports:

• Credential anchoring. A master's degree or a big-company logo gets read as proof of production skill. It proves exposure, and exposure is different from ownership.
• Confidence anchoring. Polished storytelling gets read as competence. In ML, the honest candidates hedge more, because the work is probabilistic.
• Delegated judgment without structure. Asking a friendly engineer to "chat with them for 30 minutes" produces a vibe, and a vibe is a weak signal unless the conversation is scripted.

The fix is to stop trying to judge the code and start judging what the code did in the world.

Scope note: this guide applies to applied AI/ML roles (models and LLM features shipped inside a product). Research roles that publish papers require a different evaluation and a technical panel.

The Five Signals You Can Verify Without Code

Five signals separate production ML engineers from tutorial-level profiles, and none of them require you to read Python.

If a candidate scores clearly on four of the five, they are worth a paid working session. If they score on two or fewer, stop there regardless of how the conversation felt.

The 45-Minute Interview Script

A structured 45-minute conversation gives a non-technical interviewer more signal than an unstructured technical screen. Here is the script Vintti AI recommends to founders who interview shortlisted candidates directly.

Minutes 0-15: the production story

Ask for one system they took to production and stay on it for the full 15 minutes. Useful prompts, in order:

1. "What did the system do for users, and how did you know it was working?"
2. "Walk me through what happened between the model working on your laptop and working for customers."
3. "What broke after launch? What did you change?"
4. "What would you do differently if you rebuilt it?"

What a strong answer sounds like: specific numbers (latency, error rates, cost per request), named tools, a concrete incident, and at least one thing they regret. What a weak answer sounds like: a description of the model architecture with no users in it.

Minutes 15-30: trade-offs and judgment

1. "Tell me about a time you argued against using machine learning for a problem."
2. "When did you pick the boring solution over the interesting one?"
3. "How do you decide when a model is good enough to ship?"

Senior engineers answer the first question instantly, because talking teams out of unnecessary ML is a large part of the job. Candidates who have never argued against ML have usually never owned the consequences of shipping it.

Minutes 30-40: how they would work with you

1. "Explain how a large language model decides what to say next, as if I were a customer."
2. "If I gave you our product and a week, what would you want to know before proposing anything?"
3. "How would you keep me informed on a project that is going badly?"

The first question tests translation skill directly. You are the judge: if you understood the explanation and could repeat it, they pass. The other two test whether they ask about data, users and constraints before proposing architecture.

Minutes 40-45: their questions

Strong candidates ask about your data, your existing systems, who reviews their work and how success is measured. Candidates who only ask about stack and tooling are optimizing for their resume rather than your outcome.

The Working Session You Can Run With a Proxy

Even without an in-house ML lead, you can run a live working session by borrowing one technical proxy for two hours. The proxy can be an advisor, a senior backend engineer, a fractional CTO or a screener from a staffing partner. Their job is to observe technical correctness; yours is to observe everything else.

The format Vintti AI uses in its own screening, which you can replicate:

• A scoped exercise close to your real work, never a generic algorithm puzzle: wire a model into a small API, design an evaluation for one of your features, debug a deliberately broken inference pipeline.
• 90 minutes, screen shared, thinking out loud. You are watching the working process: the questions they ask before starting, how they handle ambiguity, whether they test as they go.
• A five-minute debrief where the candidate explains what they did and what they would need to make it production-ready.

Your proxy answers one question afterwards: "Would you trust this person to touch our production system with light supervision?" You answer another: "Did I understand what they did and why?" Both answers need to be yes.

Reference Checks That Actually Predict Performance

Reference calls predict ML hiring outcomes when you ask about operations, and fail when you ask about talent. Everyone's references say the candidate is smart. Ask instead:

• "What did they own end to end, and what happened to it after they left?"
• "How did they behave during an incident?"
• "Did they ever push back on a project? Were they right?"
• "Who fixed their production issues, them or someone else?"

Backchannel references (a former colleague you find independently) carry more weight than the three names the candidate hands you. Vintti AI runs both types before a profile reaches a client shortlist.

Red Flags a Non-Technical Interviewer Can Catch

Several red flags are visible to any interviewer who listens for them.

• Every project story ends at "the model achieved 94% accuracy". Accuracy is an input; user or business impact is the output.
• No incident, ever. Production systems fail; engineers who have never seen one have never owned one.
• Cannot name a single thing they would redo.
• Explains everything at the same level of abstraction, whether talking to you or to an engineer.
• Blames data, product or infrastructure for every failure. Some blame is fair; universal blame is a pattern.
• Vague about their individual contribution on team projects ("we built", never "I built").

One red flag is a conversation. Three is a no.

What the Hire Costs, and Why It Changes the Bar

Evaluation rigor should scale with the cost of a mistake, and the cost depends heavily on where you hire. Yearly compensation from the Vintti AI Roles Benchmark:

A mis-hire at US senior rates, discovered at month six with recruiting fees and ramp time included, easily passes $150,000. The same mistake with a nearshore hire costs a fraction of that, which changes a practical decision: at nearshore rates you can afford a paid two-week trial project as the final evaluation step, and a paid trial beats every interview technique in this guide. The full dataset by role and level is published in the Vintti AI salary benchmark.

The Evaluation Checklist

Copy this into your hiring doc.

• Wrote the first quarter's deliverables before the first interview
• Production story: specific users, specific numbers, a specific incident
• Candidate argued against ML at least once in their career
• Candidate explained an ML concept and I could repeat it
• Candidate asked about our data and constraints before proposing solutions
• Live working session run with a technical proxy, close to our real work
• Proxy answered yes to "would you trust them in production with light supervision"
• Two references asked operational questions, one of them backchannel
• Fewer than three red flags from the list above
• Decision made within two weeks of the first interview

(Note: Vintti AI is a nearshore staffing and recruiting firm. We are not an Employer of Record (EOR), a managed service provider, or a BPO. We source, screen and place dedicated AI/ML engineers, as long-term embedded roles or project-based placements, and your company employs or contracts them directly. While Latin America is our core talent pool, we also recruit US-based and global candidates when the role calls for it.)

Frequently Asked Questions

Can a non-technical founder really evaluate an ML engineer?

Yes, for the dimensions that predict success in applied roles: production ownership, judgment, honesty and communication. Pair your interview with a technical proxy for one working session and you cover the rest. What you cannot do alone is grade code quality, and you do not need to.

‍

What is the single best interview question for an ML engineer?

"Tell me about a time you argued against using machine learning." Senior applied engineers answer immediately with a specific case. It tests judgment, business literacy and ownership at once.

‍

Should I use a take-home test?

A short, paid, realistic one, yes. Unpaid multi-day take-homes filter out the best candidates, who have other offers. A 90-minute live session with a proxy usually gives more signal than any take-home.

‍

How does Vintti AI screen AI/ML engineers before I see them?

Production stories told in depth, a live working session on a realistic exercise, an English evaluation for engineering conversations and reference checks, before a profile reaches your shortlist. You then run the interview in this guide on a pre-vetted list: a shortlist in about 7 days, first interviews within 1-2 weeks, and a completed hire in 2-4 weeks.

‍

Which role do I actually need: ML engineer, LLM developer or evals engineer?

If the job is training, fine-tuning or ML infrastructure, an AI/ML engineer. If it is building on top of foundation models with retrieval and orchestration, an LLM integration developer. If the pain is quality and regressions in an LLM feature already in production, an evals engineer. The explainer What Does an AI/ML Engineer Do? covers the boundaries in detail.

‍

Full-time or project-based?

Both work. Vintti AI places dedicated embedded engineers and project-based placements; in either model the engineer works only for you, which makes a paid trial project a natural first step.

‍

Evaluating an ML engineer without a technical background comes down to evaluating outcomes instead of vocabulary: what they shipped, what broke, what they chose not to build, and whether they can explain all of it to you. Run the script, borrow a proxy for one session, check references operationally, and decide fast.

‍

Hire a pre-screened AI/ML engineer - start with a conversation