AI interview questions by skill
Vibe coding
What does vibe coding mean in practice, and how does it differ from code completion? (Easy)
An agent returns 200 lines that run without error. What do you check before committing? (Medium)
How do you scope a task so an agent is likely to succeed, and what would you not hand it? (Hard)
A team adopts agents, velocity rises, and defect rate rises with it. What changed and what do you fix? (Extra hard)
Prompt engineering
What makes a prompt reliable rather than lucky? (Easy)
When do few-shot examples help, and when do they hurt? (Medium)
A prompt works on your test cases and fails in production. How do you diagnose it? (Hard)
How do you version and test prompts across a team the way you would code? (Extra hard)
RAG
What problem does retrieval-augmented generation solve? (Easy)
Why does chunking strategy matter so much? (Medium)
Your RAG system answers wrongly. How do you tell whether retrieval or generation is at fault? (Hard)
How do you evaluate a RAG pipeline end to end? (Extra hard)
LLM fundamentals
What is a token, and why does it matter to you as a developer? (Easy)
What does temperature control, and when would you set it low? (Medium)
Why do models hallucinate, and which mitigations actually work? (Hard)
A feature is accurate at 2k tokens of context and degrades at 60k. What is happening? (Extra hard)
Vector databases
What does a vector database store and query? (Easy)
What does approximate nearest neighbour search trade away? (Medium)
Why is metadata filtering harder than it looks? (Hard)
Recall is good offline and poor in production. What do you look at? (Extra hard)
LLM evaluation
Why can you not judge an AI feature by trying it a few times? (Easy)
What goes into a useful eval set? (Medium)
What are the failure modes of using a model as the judge? (Hard)
How do you catch a regression caused by the provider rather than by your change? (Extra hard)
AI output verification
Why does AI-generated code need a different kind of review from a change written by a colleague? (Easy)
What defects appear most often in AI-generated code? (Medium)
How do you verify generated code in an area you do not know well? (Hard)
An AI-authored change passes review and tests, then causes an incident. What does the postmortem change? (Extra hard)
LLM API integration
What should wrap every call to a hosted model? (Easy)
Why stream a model response, and what does streaming complicate? (Medium)
How do you get structured output you can reliably parse? (Hard)
A model call sits inside a database transaction. What is wrong with that? (Extra hard)
LLM output validation
Why validate model output before using it? (Easy)
A reply should be JSON and arrives wrapped in a code fence. How do you handle it? (Medium)
What should happen when validation still fails after retries? (Hard)
How do you validate free-text output that has no schema? (Extra hard)
AI cost control
What drives the cost of an LLM feature? (Easy)
What are the cheapest wins on an expensive AI feature? (Medium)
How do you set a per-request budget you can actually enforce? (Hard)
A feature is profitable in aggregate and loses money on a minority of users. What do you do? (Extra hard)
AI UI prototyping
What does a model do well when building an interface? (Easy)
What is usually missing from a generated interface? (Medium)
How do you take a generated prototype to production without inheriting its problems? (Hard)
Generated interfaces look internally consistent but drift from the design system. Why, and what fixes it? (Extra hard)
AI accessibility review
What can a model reliably catch in an accessibility review? (Easy)
What can it not catch, and why? (Medium)
A model reports that an element passes contrast. Why verify before accepting it? (Hard)
How do you fold AI assistance into accessibility work without weakening it? (Extra hard)
AI product scoping
What makes a problem a poor fit for an AI feature? (Easy)
How do you define a quality bar for a probabilistic feature? (Medium)
How do you design for the failure case rather than the demo? (Hard)
A model feature demos brilliantly and is quietly abandoned by users. What went wrong? (Extra hard)
AI test generation
What is the fastest useful thing to generate tests for? (Easy)
What should you give a model so it generates good tests? (Medium)
How do you review a large generated test suite efficiently? (Hard)
Where does generated testing help least? (Extra hard)
AI-assisted incident response
What is a model useful for during an incident? (Easy)
What is the specific danger of using one mid-incident? (Medium)
How do you use it without letting it drive? (Hard)
How would you evaluate whether AI assistance actually improved incident response? (Extra hard)
AI-assisted testing
What are models genuinely good at in testing, and what stays human? (Easy)
Why is it risky to generate tests from the implementation? (Medium)
How do you use a model to raise coverage without lowering test quality? (Hard)
A generated suite reports 95 percent coverage and misses a whole class of bug. How? (Extra hard)
MLOps fundamentals
What has to be versioned to reproduce a model result? (Easy)
What is model drift, and how would you detect it? (Medium)
How do you deploy a new model version safely? (Hard)
Offline accuracy improves but the online metric does not move. What are the usual causes? (Extra hard)
On-device AI
Why run a model on the device instead of calling a server? (Easy)
What does quantisation trade away? (Medium)
What constrains a model on a phone beyond available memory? (Hard)
How do you decide between on-device, server, or both? (Extra hard)
Prompt specification
Why treat a prompt as a specification rather than a message? (Easy)
What must a prompt state if its reply is going to be parsed? (Medium)
A prompt is improved for clarity and the feature starts showing empty lists. Why? (Hard)
How do you stop a prompt and its parser drifting apart over time? (Extra hard)
AI agents
What makes something an agent rather than a chat completion? (Easy)
What belongs in a good tool definition? (Medium)
How do you keep an agent loop from running away? (Hard)
An agent passes every scripted test and fails in production. What kind of failure is that, and how do you find it? (Extra hard)
Model Context Protocol
What problem does MCP solve? (Easy)
What is the difference between a tool and a resource in MCP? (Medium)
What are the security concerns with connecting a model to an MCP server? (Hard)
When would you not expose a capability through MCP? (Extra hard)
LangChain
What does LangChain give you that calling the model API directly does not? (Easy)
What does a retriever do in a chain, and where does it sit? (Medium)
When does the abstraction cost more than it saves? (Hard)
A LangChain application gets slower and more expensive after an upgrade with no code change. Where do you look? (Extra hard)
Embeddings
What is an embedding? (Easy)
Why can embeddings from two different models not be compared? (Medium)
How do you choose an embedding model for a retrieval task? (Hard)
Retrieval quality drops after re-indexing with a newer version of the same embedding model. What happened? (Extra hard)
Hugging Face
What is on the Hugging Face Hub and why does it matter? (Easy)
What do you read in a model card before using a model? (Medium)
How do you decide between running a Hub model locally and using a hosted inference endpoint? (Hard)
A team ships a Hub model and later discovers a licence problem. What should have caught it? (Extra hard)
Model fine-tuning
When is fine-tuning the wrong first move? (Easy)
What is LoRA and why is it used instead of full fine-tuning? (Medium)
How do you know a fine-tuned model has overfitted? (Hard)
A fine-tuned model scores better on the evaluation set and users prefer the base model. How? (Extra hard)
AI safety
What is prompt injection? (Easy)
How can a model leak data through its output? (Medium)
How do you build a boundary between your instructions and content from the world? (Hard)
An agent with tool access reads an attacker-controlled document. What is the worst case, and what limits it? (Extra hard)
Responsible AI
Where does bias in a model come from? (Easy)
How would you measure whether a model treats groups fairly? (Medium)
What should a team answer before a model makes a decision about a person? (Hard)
A hiring model shows no difference in outcomes across groups and is still discriminating. How? (Extra hard)
AI code review
What is a model reliably good at when reviewing code? (Easy)
Where does an AI reviewer quietly agree with a mistake? (Medium)
How do you review an AI-authored change differently from a human one? (Hard)
A team adds AI review, approval times fall, and escaped defects do not change. What is happening? (Extra hard)
Machine learning
What is the difference between supervised and unsupervised learning? (Easy)
Why do you need a validation set as well as a test set? (Medium)
A model reports 98 percent accuracy. Why might that be meaningless? (Hard)
A model performs well in evaluation and badly in production. What are the usual causes? (Extra hard)
Deep learning
What does backpropagation do? (Easy)
What is regularisation for, and name two forms of it? (Medium)
What did transformers change about sequence modelling? (Hard)
Training loss decreases and then suddenly explodes to NaN. What do you check? (Extra hard)
Natural language processing
What is tokenisation and why does it matter? (Easy)
Which classic NLP tasks are now better served by a general LLM, and which are not? (Medium)
How do you evaluate a summarisation system? (Hard)
An extraction pipeline that worked on English documents fails on documents in another language. What are the causes beyond vocabulary? (Extra hard)
Computer vision
What is the difference between classification, detection and segmentation? (Easy)
Why does data augmentation matter more in vision than in many other domains? (Medium)
What changes when a multimodal model reads an image and text in one context? (Hard)
A detection model that performed well in testing misses objects in deployment. What are the common causes? (Extra hard)
scikit-learn
What does a scikit-learn pipeline give you? (Easy)
What is cross-validation and when would you not use plain k-fold? (Medium)
How do you avoid leakage when scaling and encoding features? (Hard)
A grid search reports a best score that the model cannot reproduce on new data. What went wrong? (Extra hard)
Model serving
Why is serving a model different from serving an ordinary web request? (Easy)
What is dynamic batching and what does it trade? (Medium)
How do you handle cold starts when autoscaling model servers? (Hard)
p50 latency is fine and p99 is ten times worse. What are the likely causes in a model server? (Extra hard)
MLflow
What does experiment tracking record, and why? (Easy)
What is a model registry for? (Medium)
What has to be logged for a run to be reproducible, beyond parameters and metrics? (Hard)
Two runs with identical logged parameters produce different results. How do you find out why? (Extra hard)
Semantic search
What can semantic search find that keyword search cannot? (Easy)
Where does keyword search still beat semantic search? (Medium)
How does hybrid search combine the two, and why does it usually win? (Hard)
How do you measure whether a search change actually improved results? (Extra hard)
Feature stores
What problem does a feature store solve? (Easy)
What is train/serve skew and how does a feature store prevent it? (Medium)
What is point-in-time correctness and why does it matter for training data? (Hard)
A model trained on feature store data still shows skew in production. Where do you look? (Extra hard)
Data pipelines
What is the difference between batch and streaming ingestion? (Easy)
Why must a pipeline step be idempotent? (Medium)
How do you handle a schema change in an upstream source without breaking consumers? (Hard)
A pipeline runs green every day and a dashboard has been wrong for a month. What was missing? (Extra hard)
pandas
What is the difference between loc and iloc? (Easy)
Why does a groupby followed by a merge sometimes multiply your rows? (Medium)
A dataframe that fits on disk will not fit in memory. What are your options? (Hard)
What is the SettingWithCopyWarning telling you, and why is the bug worse than the warning? (Extra hard)
NumPy
Why is a Python loop over a NumPy array usually the wrong answer? (Easy)
What is broadcasting? (Medium)
Why does the dtype of an array matter for both correctness and speed? (Hard)
A function modifies an array it was given, and the caller did not expect it. How does that happen? (Extra hard)