Core GenAI Theory Questions
50 foundational questions frequently asked by leading AI companies, each with a concise model answer.
01 | Transformers & LLM Fundamentals
T1. Walk through the Transformer architecture, from input tokens to the final output.
Text is split into tokens, and each token ID is mapped to a learned embedding vector, with positional information added. The vectors pass through a stack of identical layers. Each layer applies multi-head self-attention, which lets every token gather information from other tokens, followed by a position-wise feed-forward network; both sub-layers use residual connections and layer normalization. After the final layer, a linear projection maps each vector to scores over the vocabulary, and softmax turns those scores into a probability distribution for the next token. During generation, a token is sampled, appended to the input and the process repeats.
T2. What advantages led Transformers to replace RNNs?
RNNs process tokens one at a time, so training cannot be parallelized across a sequence and information from distant tokens fades through many steps (vanishing gradients). Transformers process all positions at once, connect any two tokens directly through attention and map efficiently onto GPUs. That combination of parallel training, better long-range modelling and predictable scaling with more data and parameters is why they replaced RNNs for language tasks.
T3. Explain self-attention and the mathematical intuition behind it.
Self-attention lets each token decide how much to draw from every other token. Each token produces a query (what it is looking for), a key (what it offers) and a value (the information it carries). The dot product of a query with every key measures relevance; dividing by the square root of the key dimension keeps the values in a range where softmax does not saturate; softmax turns the scores into weights that sum to one; and the output is the weighted sum of the values. Intuitively, each token builds a context-aware representation of itself from the most relevant parts of the sequence.
Key formula: Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V
T4. What are Query, Key and Value vectors, and how do they interact?
They are three different linear projections of each token's embedding, learned during training. The query represents what a token is searching for, the key represents what a token can be matched on, and the value is the content that gets passed along when a match occurs. Query-key dot products produce attention weights, and those weights are applied to the values. Separating the roles lets the model learn different notions of 'relevance' and 'content'.
T5. Why does multi-head attention outperform a single attention head?
A single attention head produces one weighted average per token, so it can capture only one type of relationship at a time. Multi-head attention runs several heads in parallel, each with its own projections into a smaller subspace, so different heads can specialize in different patterns, such as syntax, coreference or positional proximity. Their outputs are concatenated and projected back, giving richer representations at roughly the same computational cost.
T6. Why does GPT apply masked (causal) self-attention?
GPT is trained to predict the next token, so a position must not see tokens that come after it, otherwise the task would be trivial and the model would fail at generation time. The causal mask sets attention scores for future positions to negative infinity before the softmax, so each token attends only to itself and earlier tokens. This also allows training on every position of a sequence in parallel while matching how generation works one token at a time.
T7. What would happen if you removed positional encoding from a Transformer?
Attention on its own is order-agnostic: shuffling the input tokens shuffles the outputs in the same way, so without positional information the model treats the input largely as a bag of tokens and cannot distinguish 'dog bites man' from 'man bites dog'. Positional encodings (sinusoidal, learned or rotary such as RoPE) inject order. One nuance worth mentioning: decoder-only models with a causal mask can infer some positional information implicitly, but explicit position signals still work far better in practice.
T8. Why are learned embeddings used instead of one-hot vectors?
One-hot vectors are as long as the vocabulary (often 50,000 to 200,000 entries), extremely sparse and all equally distant from each other, so they encode no similarity between words. Learned embeddings are dense, low-dimensional vectors trained so that tokens used in similar contexts end up close together, which helps the model generalize. Technically an embedding lookup is equivalent to multiplying a one-hot vector by a learned matrix, but the lookup is far more efficient.
T9. Explain the next-token prediction training objective.
The model is trained to predict each token from the tokens before it, by minimizing the cross-entropy between its predicted distribution and the actual next token across huge text corpora. This self-supervised objective needs no labels, scales to trillions of tokens and forces the model to learn grammar, facts and reasoning patterns because all of them help predict what comes next.
Key formula: Loss = − Σₜ log P(xₜ | x₁ … xₜ₋₁)
T10. Why are decoder-only models well suited to text generation?
Decoder-only models are trained with exactly the same objective they use at inference time, next-token prediction, so training and generation match. The architecture is simple, uses every token of training data as a target, scales predictably and supports efficient generation through KV caching. Encoder-only models such as BERT are better suited to classification and embeddings, and encoder-decoder models remain strong for some sequence-to-sequence tasks such as translation.
02 | Tokenization
T11. How does Byte Pair Encoding (BPE) tokenization work?
Byte Pair Encoding starts with individual characters or bytes as the vocabulary. It repeatedly counts the most frequent adjacent pair of symbols in the training corpus and merges it into a new symbol, adding it to the vocabulary, until the target vocabulary size is reached. Common words become single tokens, while rare words are split into meaningful subword pieces. Byte-level BPE, used by GPT models, starts from bytes so it can encode any text without unknown tokens.
T12. Why doesn't GPT simply split text into whole words?
Word-level vocabularies would be enormous, still miss new or rare words (out-of-vocabulary problems), handle morphology poorly (run, running, runner become unrelated) and break down for languages without spaces and for code. Subword tokenization balances vocabulary size and sequence length: frequent words stay whole and anything else can be built from pieces.
T13. In what ways does tokenization influence inference cost?
Providers charge per input and output token, context windows are measured in tokens and latency grows with the number of tokens processed and generated. A tokenizer that splits text into more tokens therefore increases cost, slows responses and leaves less room in the context window. Numbers, code, unusual formatting and non-English text often consume more tokens than expected.
T14. Why do different LLMs produce different token counts for the same text?
Each model family has its own tokenizer, trained on a different corpus with a different algorithm and vocabulary size. A larger vocabulary or one trained on more multilingual or code data produces fewer tokens for the same text. That is why identical text can have noticeably different token counts, and therefore cost and context usage, across providers.
T15. What problems arise when tokenizing multilingual text?
Tokenizers trained mostly on English split many other languages, especially non-Latin scripts, into far more tokens per word. Those languages then cost more, fit less content into the context window, run slower and can see lower quality because words are fragmented into less meaningful pieces. This is sometimes called a 'token tax' on non-English languages.
03 | Prompt Engineering
T16. What are the qualities of an effective prompt?
An effective prompt states the task clearly and specifically, provides the necessary context, defines the role and audience where helpful, specifies the output format and constraints, and includes examples when the format or style is subtle. It separates instructions from data with clear delimiters, tells the model what to do when information is missing, and is tested and iterated on against a set of real inputs rather than judged on one example.
T17. Compare zero-shot and few-shot prompting.
Zero-shot prompting gives only instructions and relies on the model's existing ability, which works well for common, clearly described tasks. Few-shot prompting adds a handful of input-output examples, which helps with unusual formats, domain conventions or subtle classification boundaries. Few-shot costs more tokens and can bias the model toward the examples, so choose examples that are diverse and representative.
T18. In which situations can Chain-of-Thought prompting hurt performance?
Chain-of-thought can hurt on simple tasks, where it adds latency and cost and can lead the model to overthink an easy answer. It may hurt where the reasoning generated is plausible but wrong, which is more common with smaller models, and in latency-sensitive applications. It is also often unnecessary with reasoning models that already think internally before answering. Use it selectively for genuinely multi-step problems.
T19. What is Self-Consistency prompting and when is it useful?
Self-consistency samples several independent reasoning paths for the same question at a non-zero temperature and selects the most common final answer by majority vote. Because correct reasoning tends to converge on the same answer while errors scatter, this improves accuracy on arithmetic, logic and multi-step questions. The trade-off is cost and latency that grow with the number of samples.
T20. Explain ReAct prompting using a concrete example.
ReAct (Reason + Act) interleaves reasoning with tool use. The model writes a thought about what it needs, chooses an action such as a search or API call, receives an observation and repeats until it can answer. Example: asked 'What is the population of the city where the Eiffel Tower is located?', the model thinks it needs the city, searches and observes 'Paris'; it then thinks it needs Paris's population, searches again, observes the figure and gives the final answer. This grounds reasoning in real data and underpins most agent frameworks.
04 | Hallucinations
T21. What causes LLMs to hallucinate?
LLMs generate the most plausible continuation, not verified facts. Hallucinations come from gaps or errors in training data, outdated knowledge, ambiguous prompts, training that rewards fluent and confident answers, sampling randomness, and missing or poor-quality context in RAG systems. The model has no built-in mechanism to check its outputs against reality.
T22. Is it possible to eliminate hallucinations entirely? Why or why not?
Not entirely. Generation is probabilistic and models compress knowledge imperfectly, so some error rate will always remain. Hallucinations can be reduced dramatically through grounding, verification, constrained outputs and tool use, and their impact can be limited through detection, citations, abstention and human review. The realistic goal is an error rate that is acceptable for the use case, with safeguards proportional to the risk.
T23. Which techniques would you use to reduce hallucinations in production?
Ground answers with good retrieval, instruct the model to use only the provided context and to say when it does not know, require citations, lower temperature for factual tasks, use tools for exact facts and calculations, validate outputs with claim-level checks or a verifier model, apply confidence thresholds with abstention, and route high-risk cases to humans. Monitor hallucination rates continuously in production.
T24. How does the temperature setting relate to hallucination?
Temperature divides the model's logits before softmax. A higher temperature flattens the distribution, so lower-probability tokens get sampled more often; that increases diversity but also the chance of drifting into unsupported content. A lower temperature makes outputs more deterministic. Low temperature does not eliminate hallucinations, though: a model can be confidently wrong when the most likely continuation is itself incorrect.
T25. What is the difference between factuality and fluency?
Fluency is how natural, grammatical and coherent text sounds; factuality is whether it is true and supported by evidence. LLMs are optimized heavily for fluency, so false statements can sound just as convincing as true ones. That is why fluency should never be used as a proxy for correctness and why factuality needs separate evaluation.
05 | Retrieval-Augmented Generation (RAG)
T26. Describe an end-to-end RAG pipeline.
Offline: ingest documents, parse and clean them, split them into chunks, generate embeddings and store them with metadata in a vector index (often alongside a keyword index). Online: take the user query, optionally rewrite it, retrieve candidates with vector or hybrid search, filter by permissions and metadata, rerank, assemble the best chunks into a prompt with instructions, generate the answer with citations and optionally verify it. Monitoring and evaluation run across both halves.
T27. RAG or fine-tuning: which would you pick, and why?
Choose RAG when the model needs knowledge, especially proprietary, frequently changing or permission-controlled information that must be cited. Choose fine-tuning when the model needs behaviour: a consistent format, a domain style, a narrow task or shorter prompts. In practice, start with prompting and RAG, fine-tune only when behaviour gaps persist, and combine both when you need specialized behaviour grounded in current facts.
T28. Why does chunking matter so much in RAG?
Chunking determines what can be retrieved. Chunks that are too large dilute embeddings and waste context with irrelevant text; chunks that are too small lose surrounding meaning; boundaries that cut through sentences, tables or sections break the information apart. Good chunking directly improves retrieval precision, recall and answer quality, which makes it one of the highest-leverage RAG decisions.
T29. How would you choose a chunk size for PDF documents?
There is no universal size. A common starting point is around 300 to 800 tokens with 10 to 20% overlap, split along the document's structure (headings, sections, paragraphs) and keeping tables intact. Prepend document and section titles to each chunk. Use smaller chunks for dense reference material and larger ones for narrative text, respect the embedding model's input limit and tune the size on an evaluation set.
T30. What is reranking and why is it necessary?
Reranking is a second retrieval stage. A first stage such as vector or hybrid search quickly retrieves a broad set of candidates; a more accurate but slower model, usually a cross-encoder that reads the query and each document together, then reorders them by true relevance. It is needed because embedding similarity is approximate, and reranking significantly improves which chunks make it into the limited context window.
T31. Compare hybrid search with pure vector search.
Vector search finds semantically similar text even when words differ, but can miss exact terms such as product codes, names and acronyms. Keyword search such as BM25 excels at exact matches but misses paraphrases. Hybrid search runs both and fuses the results, commonly with Reciprocal Rank Fusion, and is more robust across query types, which is why it is the default in most production RAG systems.
Key formula: RRF score(d) = Σ 1 / (k + rankᵢ(d)), commonly k = 60
T32. What qualities define a good embedding model?
It retrieves well on your domain and query types (check benchmarks such as MTEB, then test on your own data), supports the languages you need, has a suitable maximum input length and dimension size for your cost and latency budget, and is affordable and fast enough to embed your corpus. For specialized domains, models fine-tuned on in-domain data often outperform larger general models.
T33. How does weak retrieval lead to hallucinations?
When retrieval misses the relevant evidence or returns misleading chunks, the model still tries to answer, falling back on its parametric knowledge or stretching irrelevant context to fit the question. Weak retrieval therefore directly produces confident but unsupported answers. Fixing retrieval is often the single most effective way to reduce RAG hallucinations.
T34. How would you evaluate a RAG system end to end?
Evaluate the components and the whole. For retrieval, use labelled query-document pairs to measure recall@k, precision and ranking quality. For generation, measure faithfulness to the retrieved context, answer relevance and correctness against reference answers. End to end, use golden datasets, LLM-as-judge scoring calibrated with human review, and online signals such as user feedback, resolution and escalation rates.
T35. Which metrics matter most when measuring RAG quality?
Context recall (did we retrieve the needed evidence), context precision (how much of the retrieved context is relevant), faithfulness or groundedness (is the answer supported by the context), answer relevance and correctness, citation accuracy, and operational metrics such as latency and cost per query. Frameworks such as RAGAS package several of these.
06 | Embeddings & Vector Databases
T36. What are sentence embeddings?
Sentence embeddings are fixed-length dense vectors that represent the meaning of a whole sentence or passage, so that semantically similar texts are close together in vector space. They are produced by models trained, often with contrastive learning, to pull matching pairs together and push unrelated pairs apart, and they power semantic search, clustering, deduplication and RAG retrieval.
T37. Why is cosine similarity often preferred over Euclidean distance?
Cosine similarity measures the angle between vectors and ignores their length, so it captures direction of meaning rather than magnitude, which can vary with text length or other artefacts. Most embedding models are trained with cosine-based objectives. For normalized vectors the two measures give the same ranking, because squared Euclidean distance equals 2 − 2 × cosine similarity.
Key formula: cos(a, b) = (a · b) / (‖a‖ ‖b‖)
T38. Explain Approximate Nearest Neighbor (ANN) search.
Exact nearest-neighbour search compares the query to every vector, which becomes too slow at millions or billions of vectors. ANN algorithms build index structures (graphs, clusters, quantized codes) that find very close neighbours much faster by searching only part of the space. They trade a small amount of recall for large gains in speed and memory, and that trade-off is tuned with index parameters.
T39. Compare HNSW and IVF indexes.
HNSW builds a multi-layer graph of vectors; searches navigate from sparse upper layers down to dense lower layers. It gives high recall and low latency and supports incremental inserts, but uses more memory. IVF clusters vectors with k-means and searches only the nearest clusters (controlled by nprobe); it needs training, uses less memory especially when combined with product quantization (IVF-PQ), and its recall depends on how many clusters are searched. HNSW suits latency-sensitive workloads that fit in memory; IVF-PQ suits very large, memory-constrained collections.
T40. Why should embeddings be normalized?
Normalizing embeddings to unit length makes the dot product equal to cosine similarity, so faster inner-product indexes can be used. It also removes magnitude effects so comparisons depend only on direction, keeps scores in a consistent range for thresholds and is expected by many embedding models and vector databases.
07 | Fine-Tuning & Alignment
T41. What is instruction tuning?
Instruction tuning is supervised fine-tuning of a pretrained model on many examples of instructions paired with high-quality responses, across varied tasks. It teaches the model to follow instructions and respond helpfully instead of merely continuing text, and it is the first step that turns a base model into an assistant.
T42. Explain RLHF step by step.
Step one: supervised fine-tuning on high-quality demonstrations. Step two: collect human preferences by having people compare pairs of model responses, and train a reward model to predict which response humans prefer. Step three: optimize the model with reinforcement learning (typically PPO) to maximize the reward model's score, with a KL penalty that keeps it close to the supervised model so it does not exploit the reward model or drift into degenerate text.
T43. What is Direct Preference Optimization (DPO), and how does it differ from RLHF?
DPO uses the same preference data as RLHF (prompts with a preferred and a rejected response) but skips the separate reward model and reinforcement learning loop. It optimizes the policy directly with a classification-style loss that increases the likelihood of preferred responses relative to rejected ones, measured against a frozen reference model. It is simpler, more stable and cheaper to train, though RLHF can be more flexible for some objectives.
Key formula: L_DPO = − log σ( β [ log πθ(y_w|x)/π_ref(y_w|x) − log πθ(y_l|x)/π_ref(y_l|x) ] )
T44. Compare LoRA and QLoRA.
LoRA freezes the original weights and learns small low-rank update matrices for selected layers, training only a tiny fraction of parameters while keeping quality close to full fine-tuning. QLoRA adds 4-bit quantization of the frozen base model (NF4), along with techniques such as double quantization and paged optimizers, so large models can be fine-tuned on a single GPU. QLoRA uses much less memory; the trade-offs are somewhat slower training and a small potential quality cost.
Key formula: W' = W + ΔW = W + B·A, where B ∈ ℝ^(d×r), A ∈ ℝ^(r×k), r ≪ d
T45. Why isn't fine-tuning always the right solution?
Fine-tuning needs high-quality labelled data, costs time and money, and creates a model you must evaluate, host and maintain. It does not add up-to-date or citable knowledge well, can degrade general capabilities and needs repeating as base models improve. Many problems are solved more cheaply with better prompts, RAG or tools, so fine-tune only after those options are exhausted.
08 | Production GenAI
T46. Which techniques would you use to lower LLM latency?
Stream responses to cut perceived latency; use smaller or faster models for simpler steps; shorten prompts and cap output length; cache responses, retrieval results and shared prompt prefixes; run independent calls in parallel and reduce sequential LLM calls; optimize self-hosted inference with batching, quantization and speculative decoding; and keep services in the same region to minimize network hops.
T47. How would you cut token costs in production?
Route requests to the cheapest model that meets the quality bar; trim system prompts, examples and retrieved context; summarize long histories; cap output length; cache exact, semantic and prefix-level repeats; use batch APIs for non-urgent work; fine-tune smaller models for high-volume narrow tasks; and attribute costs per feature so the biggest spenders are visible.
T48. Which guardrails would you put in place before deployment?
Input guardrails: prompt-injection and jailbreak detection, PII detection and redaction, topic and policy filters, and rate limits. Output guardrails: content safety moderation, groundedness checks, PII and secret leakage checks, format and schema validation. System guardrails: least-privilege tool access, human approval for risky actions, audit logging and a tested fallback or refusal behaviour.
T49. How would you monitor hallucinations once a system is live?
Sample production responses and score them automatically for faithfulness to retrieved context using NLI models or calibrated LLM judges; extract and verify claims in high-risk domains; have humans review a regular sample to calibrate the automated scores; collect user feedback with reasons; and track the hallucination rate on dashboards by feature, with alerts when it rises.
T50. Design a production-ready architecture for an enterprise chatbot.
Channels (web, mobile, chat apps) connect through an API gateway with authentication and rate limiting to an orchestration service. The orchestrator applies input guardrails, classifies intent and routes to RAG over a permission-aware knowledge base, to tools and APIs for transactions, or to deterministic flows. An LLM gateway handles model routing, caching and failover. Output guardrails check responses before delivery. Supporting components include conversation memory, human handoff, observability and tracing, an evaluation pipeline, and CI/CD for
0 Comments