NLP Interview Questions

Foundational Concepts & Text Preprocessing

1. What is Natural Language Processing (NLP)?

NLP is a subfield of Artificial Intelligence (AI), computer science, and linguistics concerned with the interactions between computers and human (natural) languages. It focuses on enabling computers to understand, interpret, process, and generate human language in a way that is valuable. The ultimate objective is to bridge the gap between human communication and computer understanding.

2. What are the major tasks/applications of NLP?

Major tasks include: * Text Classification: Assigning categories to text (e.g., sentiment analysis, spam detection, topic labeling). * Named Entity Recognition (NER): Identifying and classifying named entities (persons, organizations, locations, dates) in text. * Machine Translation: Translating text from one language to another. * Sentiment Analysis: Determining the emotional tone (positive, negative, neutral) expressed in a piece of text. * Text Summarization: Generating a concise summary of a longer document. * Question Answering: Providing answers to questions posed in natural language, often based on a given context. * Part-of-Speech (POS) Tagging: Assigning grammatical parts of speech (noun, verb, adjective) to each word. * Language Modeling: Predicting the probability of a sequence of words. * Speech Recognition: Converting spoken language into text. * Text Generation: Creating human-like text (e.g., chatbots, story writing).

3. Explain Tokenization.

Tokenization is the process of breaking down a stream of text (like a sentence or paragraph) into smaller units called tokens. These tokens are typically words, punctuation marks, or sometimes sub-words or characters. It’s a fundamental first step in most NLP pipelines. For example, “NLP is fascinating!” might be tokenized into [“NLP”, “is”, “fascinating”, “!”].

4. What is the difference between Stemming and Lemmatization? Give examples. 

Both are techniques used for text normalization to reduce words to their base or root form. * Stemming: A heuristic process that chops off the ends of words (suffixes) to get a base form called a “stem”. It’s often faster but can result in non-dictionary words. Example: “running”, “ran” -> “run”; “studies”, “studying” -> “studi”. * Lemmatization: A more sophisticated process that uses vocabulary and morphological analysis (considering the word’s part of speech) to return the base dictionary form of a word, known as the “lemma”. It’s generally more accurate but slower. Example: “running”, “ran” -> “run”; “studies”, “studying” -> “study”; “better” -> “good”.

5. What are Stop Words and why are they often removed?

Stop words are common words in a language (e.g., “a”, “an”, “the”, “is”, “in”, “on”, “and”) that appear frequently but often carry little semantic meaning relevant to the core content of the text. They are often removed during preprocessing to: * Reduce the dimensionality of the data (fewer unique tokens). * Improve the performance of some models (like Bag-of-Words based classifiers) by focusing on more meaningful words. * Save computational resources and time. * However, removal isn’t always beneficial, especially for tasks where sentence structure or subtle meanings are important (e.g., some sentiment analysis, language modeling).

Text Representation & Embeddings

6. Explain Bag-of-Words (BoW). What are its limitations? 

Bag-of-Words is a simple text representation model. It represents a piece of text as an unordered collection (a “bag”) of its words, disregarding grammar and word order but keeping track of frequency (term frequency). A document is represented as a vector where each dimension corresponds to a word in the vocabulary, and the value is typically the count of that word in the document. * Limitations: * Loses word order and syntactic information. (“Man bites dog” vs. “Dog bites man” look similar). * Doesn’t capture semantic meaning (synonyms are treated as different words). * Vocabulary can become very large (high dimensionality). * Doesn’t inherently account for word importance (common words might dominate).

7. What is TF-IDF? How does it improve upon simple BoW counts?

TF-IDF stands for Term Frequency-Inverse Document Frequency. It’s a numerical statistic used to reflect how important a word is to a document within a collection or corpus. It improves upon simple term frequency (TF) by down-weighting words that are common across many documents. * Term Frequency (TF): Measures how frequently a term appears in a specific document. (Often normalized). * Inverse Document Frequency (IDF): Measures how important a term is across the entire corpus. It’s calculated as log(Total number of documents / Number of documents containing the term). Rare words get a high IDF score, common words get a low score. * TF-IDF Score: TF * IDF. Words that are frequent in a specific document but rare across documents get a high score, indicating they are characteristic of that document.

8. What are Word Embeddings? Why are they useful? A8:

Word Embeddings are dense, low-dimensional vector representations of words, typically learned from large amounts of text data. Unlike sparse representations like BoW or TF-IDF, embeddings capture semantic relationships between words. Words with similar meanings tend to have similar vectors (closer in the vector space). * Usefulness: * Capture semantic similarity and relationships (e.g., vector(‘King’) – vector(‘Man’) + vector(‘Woman’) ≈ vector(‘Queen’)). * Provide dense, lower-dimensional input features for downstream ML models, improving performance and efficiency. * Can be pre-trained on massive datasets and then used for tasks with smaller datasets (transfer learning).

9. Explain Word2Vec (mention Skip-gram and CBOW).

Word2Vec is a popular technique (developed at Google) to learn word embeddings from text. It uses shallow neural networks. There are two main architectures: * Continuous Bag-of-Words (CBOW): Predicts the current target word based on its surrounding context words. It treats the context words as a “bag” (order doesn’t matter) and averages their vectors to predict the target. Generally faster and better for frequent words. * Skip-gram: Predicts the surrounding context words given the current target word. It works well with small amounts of training data and represents rare words well. It’s generally slower than CBOW. * Both models learn embeddings as a byproduct of the prediction task.

10. What is GloVe (Global Vectors for Word Representation)? How does it differ from Word2Vec?

GloVe is another popular word embedding technique developed at Stanford. It aims to combine the strengths of global matrix factorization methods (like Latent Semantic Analysis – LSA) and local context window methods (like Word2Vec). * Difference: Word2Vec focuses on local context windows (predicting words based on nearby words). GloVe constructs a large co-occurrence matrix that captures how frequently words co-occur across the entire corpus. It then uses matrix factorization techniques (specifically, weighted least squares regression) on this global co-occurrence data to learn word vectors such that their dot product relates to their probability of co-occurrence. GloVe directly leverages global statistics.

11. What is Cosine Similarity and why is it often used with word embeddings?

Cosine Similarity is a metric used to measure the similarity between two non-zero vectors in an inner product space. It calculates the cosine of the angle between the vectors. The value ranges from -1 (exactly opposite) to 1 (exactly the same), with 0 indicating orthogonality (uncorrelated). * It’s often used with word embeddings because it measures orientation rather than magnitude. Word embedding vectors represent direction in semantic space. Even if two words appear with different frequencies (affecting vector magnitude in some representations), their semantic similarity is better captured by the angle (direction) between their embedding vectors. 

Traditional Machine Learning Models for NLP

12. How can Naive Bayes be used for text classification (e.g., spam detection)?

Naive Bayes is a probabilistic classifier based on Bayes’ theorem with a “naive” assumption of conditional independence between features (words). For text classification: 1. Represent each document as a vector of word counts or TF-IDF scores (features). 2. Calculate the prior probability of each class (e.g., P(Spam), P(Not Spam)). 3. Calculate the likelihood of each word appearing given a class (e.g., P(word | Spam)). This is learned from the training data. 4. For a new document, use Bayes’ theorem to calculate the posterior probability of each class given the words in the document: P(Class | Document) ∝ P(Class) * Π P(word_i | Class). The independence assumption allows multiplying the individual word probabilities. 5. Assign the class with the highest posterior probability. * It’s simple, fast, and works surprisingly well for text tasks despite the unrealistic independence assumption.

13. Can you use models like SVM or Logistic Regression for text classification? How? 

Yes, SVM (Support Vector Machines) and Logistic Regression are very effective for text classification. 1. Preprocessing: Clean the text (lowercase, remove punctuation, stop words, stemming/lemmatization). 2. Feature Extraction: Convert the text into numerical vectors using methods like Bag-of-Words, TF-IDF, or even pre-trained word/document embeddings. TF-IDF is very common here. 3. Model Training: Train the SVM or Logistic Regression classifier using the labeled document vectors. * SVM: Finds an optimal hyperplane that separates the data points (documents) of different classes with the maximum margin. Works well in high-dimensional spaces (common with TF-IDF). * Logistic Regression: Models the probability of a document belonging to a particular class using a logistic (sigmoid) function. Outputs probabilities, which can be useful. 4. Prediction: Use the trained model to predict the class of new, unseen documents.

Sequence Models (RNNs, LSTMs, GRUs)

14. What is a Recurrent Neural Network (RNN)? Why is it suitable for sequence data like text?

An RNN is a type of neural network designed to work with sequential data. Unlike feedforward networks, RNNs have connections that form directed cycles, allowing them to maintain an internal state or “memory”. This memory captures information about previous elements in the sequence, which influences the processing of subsequent elements. * Suitability for Text: Text is inherently sequential; the meaning often depends on the order of words. RNNs can process words one by one, updating their hidden state at each step to incorporate information from earlier words, making them suitable for tasks like language modeling, machine translation, and sentiment analysis where context matters.

15. What are the problems of Vanishing and Exploding Gradients in RNNs?

These are major challenges when training deep RNNs, particularly on long sequences: * Vanishing Gradients: During backpropagation, gradients (error signals) are multiplied by weights at each time step. If these weights (or their derivatives) are consistently small (less than 1), the gradients can shrink exponentially as they propagate backward through time. This means the network struggles to learn long-range dependencies, as the influence of earlier inputs on later outputs becomes negligible. * Exploding Gradients: Conversely, if the weights (or derivatives) are consistently large (greater than 1), the gradients can grow exponentially, leading to huge updates in weights and numerical instability (often resulting in NaN values). This makes training difficult or impossible. Gradient clipping is a common technique to mitigate this.

16. How do LSTMs (Long Short-Term Memory) and GRUs (Gated Recurrent Units) address the vanishing gradient problem? 

LSTMs and GRUs are specialized types of RNNs designed to overcome the vanishing gradient problem and better capture long-range dependencies. They introduce gating mechanisms: * LSTMs: Use three main gates (input, forget, output) and a separate cell state. * Forget Gate: Decides which information to throw away from the cell state. * Input Gate: Decides which new information to store in the cell state. * Output Gate: Decides what to output based on the cell state (filtered). * The cell state acts like a conveyor belt, allowing information to flow through relatively unchanged unless explicitly modified by the gates, thus preserving gradients over longer sequences. * GRUs: A simpler variant with two gates (reset and update) and no separate cell state. * Reset Gate: Determines how to combine the new input with the previous memory. * Update Gate: Decides how much of the previous memory to keep and how much new information to add. * These gates learn to control the flow of information, selectively remembering relevant past information and forgetting irrelevant parts, enabling learning of dependencies across longer time steps.

17. What is the difference between an Encoder-Decoder architecture and a standard RNN?

Standard RNN: Processes an input sequence and typically produces an output at each time step (e.g., POS tagging) or a single output after the entire sequence (e.g., sentiment classification). Input and output lengths are often related. * Encoder-Decoder Architecture (Sequence-to-Sequence): Designed for tasks where the input and output sequences can have different lengths (e.g., machine translation, summarization). * Encoder: An RNN (often LSTM or GRU) reads the entire input sequence and compresses it into a fixed-size context vector (the final hidden state or a combination of states), capturing the essence of the input. * Decoder: Another RNN takes the context vector from the encoder as its initial state and generates the output sequence step-by-step, often using the previously generated output as input for the next step.

Attention Mechanism & Transformers

18. What is the Attention Mechanism in NLP? Why was it introduced?

The Attention Mechanism allows a model (typically a decoder in seq2seq) to selectively focus on different parts of the input sequence when generating each part of the output sequence. Instead of relying solely on a single fixed-size context vector from the encoder (which becomes a bottleneck for long sequences), the decoder can “look back” at the encoder’s hidden states from all input time steps. * Why Introduced: To overcome the limitation of the fixed-size context vector in standard encoder-decoder models. This vector struggles to capture all necessary information from long input sequences, leading to performance degradation. Attention provides a way for the decoder to access relevant parts of the input directly, improving performance significantly, especially in tasks like machine translation with long sentences.

19. Explain the concept of Self-Attention (as used in Transformers).

Self-Attention (or intra-attention) is an attention mechanism that relates different positions of a single sequence to compute a representation of that sequence. Unlike traditional attention relating decoder states to encoder states, self-attention allows the model to weigh the importance of all other words in the same input sentence when encoding a specific word. It calculates how much focus each word should place on every other word (including itself) within the sentence. This helps capture dependencies and relationships between words regardless of their distance, overcoming limitations of RNNs in handling long-range dependencies efficiently.

20. What is Multi-Head Attention?

Multi-Head Attention is a key component of the Transformer model. Instead of computing a single attention function, it runs the attention mechanism multiple times (“heads”) in parallel with different, learned linear projections of the queries, keys, and values. The outputs from each head are then concatenated and linearly transformed to produce the final output. * Benefit: It allows the model to jointly attend to information from different representation subspaces at different positions. A single attention head might focus on one type of relationship (e.g., syntactic), while another focuses on a different type (e.g., semantic proximity). This provides a richer representation compared to single-head attention.

21. Describe the high-level architecture of the Transformer model.

The Transformer, introduced in the paper “Attention Is All You Need,” is based entirely on attention mechanisms, dispensing with recurrence and convolutions. Its architecture consists of: * Encoder Stack: Composed of multiple identical layers. Each layer has two sub-layers: 1. Multi-Head Self-Attention mechanism. 2. Position-wise Fully Connected Feed-Forward Network. * Residual connections and layer normalization are applied around each sub-layer. * Decoder Stack: Also composed of multiple identical layers. Each layer has three sub-layers: 1. Masked Multi-Head Self-Attention mechanism (masked to prevent attending to future positions during generation). 2. Multi-Head Attention mechanism over the output of the encoder stack (cross-attention). 3. Position-wise Fully Connected Feed-Forward Network. * Residual connections and layer normalization are applied. * Positional Encoding: Since the model contains no recurrence, positional encodings (using sine/cosine functions or learned embeddings) are added to the input embeddings to give the model information about the position of words in the sequence.

22. Why are Transformers considered significant in NLP? 

Transformers are significant because: * Parallelization: Unlike RNNs which process sequences sequentially, the self-attention mechanism allows processing all words in parallel, leading to much faster training times on modern hardware (GPUs/TPUs). * Long-Range Dependencies: Self-attention directly connects all words, making it much better at capturing long-range dependencies compared to RNNs, which struggle due to vanishing gradients. * State-of-the-Art Performance: They achieved state-of-the-art results on numerous NLP tasks (translation, language understanding) upon release and formed the basis for subsequent large language models. * Scalability: Their architecture scales well to extremely large datasets and model sizes, enabling the development of models like BERT and GPT.

Pre-trained Models & Large Language Models (LLMs)

23. What is Transfer Learning in NLP? 

Transfer Learning in NLP involves using knowledge gained from training a model on a large, general dataset (like Wikipedia or a massive web crawl) and applying it to a different, often smaller, and more specific task. Typically, a large language model is pre-trained on a task like masked language modeling or next-token prediction. The learned representations (embeddings and contextual understanding) from this pre-trained model are then used as a starting point for a downstream task (like sentiment analysis or NER) by adding a small task-specific layer and fine-tuning the model on the target task’s data. This significantly reduces the need for large labeled datasets for the specific task and often improves performance.

24. Explain BERT (Bidirectional Encoder Representations from Transformers). How is it pre-trained?

BERT is a highly influential pre-trained language representation model based on the Transformer encoder architecture. Its key innovation is its bidirectionality – it considers both the left and right context simultaneously when generating representations for each word. * Pre-training Tasks: 1. Masked Language Model (MLM): During pre-training, some percentage (e.g., 15%) of the input tokens are randomly masked (replaced with a [MASK] token). The model’s objective is to predict the original masked tokens based on the surrounding unmasked context from both directions. This forces the model to learn rich contextual representations. 2. Next Sentence Prediction (NSP): The model receives pairs of sentences (A, B) and predicts whether sentence B is the actual next sentence that follows A in the original text or just a random sentence. This helps the model understand relationships between sentences, beneficial for tasks like Question Answering and Natural Language Inference. (Note: Later models like RoBERTa found NSP less critical than MLM).

25. What are the differences between BERT and GPT models?

Architecture: BERT uses the Transformer encoder stack. GPT (Generative Pre-trained Transformer) models primarily use the Transformer decoder stack. * Pre-training Objective: BERT uses Masked Language Modeling (MLM) and Next Sentence Prediction (NSP), allowing it to learn bidirectional representations. GPT uses a standard Language Modeling objective (predicting the next word given the previous words), making it inherently unidirectional (autoregressive) and well-suited for text generation. * Directionality: BERT is deeply bidirectional. GPT is unidirectional (left-to-right). * Fine-tuning: BERT is typically fine-tuned by adding a small classification layer on top and training on task-specific data, excelling at understanding tasks (classification, NER, QA). GPT models excel at generation tasks and can also be fine-tuned, but they popularized few-shot/zero-shot learning via prompting, especially the larger variants (GPT-3, GPT-4).

26. What is Fine-Tuning a pre-trained model?

Fine-tuning is the process of taking a pre-trained language model (like BERT or GPT) that has already learned general language patterns from a massive dataset, and further training it on a smaller, task-specific labeled dataset. Typically, a task-specific head (e.g., a linear layer for classification) is added on top of the pre-trained model’s core layers. During fine-tuning, the weights of the entire model (or just the top layers) are updated using the task-specific data to adapt the model’s general knowledge to the nuances of the target task.

27. What are Large Language Models (LLMs)? Give examples.

LLMs are deep learning models characterized by their massive size (billions or even trillions of parameters) and pre-training on vast amounts of text data. They exhibit remarkable abilities in understanding and generating human-like text and can perform a wide range of NLP tasks, often with minimal or no task-specific training (zero-shot or few-shot learning via prompting). * Examples (as of early 2025): OpenAI’s GPT-4, Google’s Gemini family, Anthropic’s Claude series, Meta’s Llama series, Mistral AI models.

28. What is Prompt Engineering?

Prompt Engineering is the art and science of designing effective inputs (“prompts”) to guide Large Language Models (LLMs) towards generating desired outputs or performing specific tasks accurately. Since LLMs are often used in a zero-shot or few-shot setting without fine-tuning, the way a task is described or exemplified in the prompt significantly impacts the quality and relevance of the model’s response. It involves crafting clear instructions, providing relevant context, and sometimes including examples (few-shot prompting).

29. Discuss some ethical considerations related to LLMs.

Ethical concerns include: * Bias: LLMs are trained on vast internet data, which contains societal biases (gender, race, religion, etc.). These biases can be reflected or amplified in the model’s outputs. * Misinformation & Disinformation: LLMs can generate convincing but false or misleading text (“hallucinations”) at scale, potentially spreading misinformation. * Toxicity & Harmful Content: They can generate toxic, hateful, or otherwise harmful content if not properly safeguarded. * Job Displacement: Automation of tasks previously done by humans (writing, customer service). * Environmental Impact: Training massive models requires significant computational resources and energy. * Copyright & Ownership: Issues related to training data ownership and generated content originality. * Privacy: Potential for models to memorize and reveal sensitive information from training data.

Specific NLP Tasks

30. How would you approach a Sentiment Analysis task?

Data Collection & Labeling: Gather text data (e.g., reviews, tweets) and label it as positive, negative, or neutral. 2. Preprocessing: Clean the text (lowercase, remove URLs/handles, possibly punctuation/stop words depending on the model, handle emojis/negations carefully). 3. Feature Extraction: * Traditional: TF-IDF vectors. * Modern: Word embeddings (Word2Vec, GloVe) or contextual embeddings from pre-trained models (BERT, RoBERTa). 4. Model Selection: * Traditional: Naive Bayes, SVM, Logistic Regression (with TF-IDF). * Deep Learning: LSTMs/GRUs, CNNs (work well for text too), or fine-tuning a pre-trained Transformer model (often state-of-the-art). 5. Training: Train the chosen model on the labeled data. Handle class imbalance if necessary (e.g., using over/under-sampling, SMOTE, or adjusting class weights). 6. Evaluation: Evaluate using metrics like accuracy, precision, recall, F1-score (especially important for imbalanced datasets), and confusion matrix. 7. Deployment: Deploy the trained model for inference on new text.

31. Explain Named Entity Recognition (NER). What models are commonly used?

A31: NER is the task of identifying and classifying named entities in text into predefined categories such as Person, Organization, Location, Date, Time, Monetary Value, etc. For example, in “Apple Inc. is headquartered in Cupertino.”, NER would identify “Apple Inc.” as ORGANIZATION and “Cupertino” as LOCATION. * Common Models: * Rule-based/Dictionary: Using predefined lists and grammatical rules (less robust). * Traditional ML: Conditional Random Fields (CRFs) often built on top of features extracted by models like SVMs or Maximum Entropy models. * Deep Learning: Bidirectional LSTMs (or GRUs) often combined with a CRF layer on top (BiLSTM-CRF) are very popular. The BiLSTM captures contextual information, and the CRF layer helps ensure valid sequences of tags (e.g., I-PERSON cannot follow B-ORGANIZATION). Fine-tuning pre-trained models like BERT is now often state-of-the-art.

32. What is Topic Modeling? Mention a common algorithm.

Topic Modeling is an unsupervised machine learning technique used to discover abstract “topics” that occur in a collection of documents. It helps understand the hidden thematic structure in large text corpora without pre-defined labels. It assumes each document is a mixture of topics, and each topic is a mixture of words. * Common Algorithm: Latent Dirichlet Allocation (LDA). LDA is a generative probabilistic model. It models documents as distributions over topics and topics as distributions over words. Given a corpus, LDA algorithms try to infer the hidden topic structure (which words are associated with which topics, and which topics are associated with which documents).

33. How does Machine Translation work at a high level (mention seq2seq and Transformers)?

Machine Translation aims to translate text from a source language to a target language. * Statistical MT (Older): Relied on learning statistical models from large bilingual corpora (parallel texts). * Neural MT (NMT – Current Standard): * Seq2Seq with RNNs/LSTMs: An encoder RNN reads the source sentence and produces a context vector. A decoder RNN uses this context vector to generate the target sentence word by word. Attention mechanisms were added to improve handling of long sentences. * Transformer-based NMT: This is the current state-of-the-art. It uses the Transformer architecture (encoder-decoder stacks with self-attention and cross-attention) which allows for better parallelization and capturing long-range dependencies, leading to superior translation quality compared to RNN-based models.

34. What are challenges in Text Summarization? (Abstractive vs. Extractive)

Text Summarization aims to create a short, coherent summary of a longer text. Challenges include: maintaining factual consistency, ensuring coherence and readability, capturing the main points, and avoiding redundancy. * Extractive Summarization: Selects important sentences or phrases directly from the original text and concatenates them to form a summary. Easier to implement and generally preserves factual accuracy but can lack coherence. Algorithms often involve scoring sentences based on features like TF-IDF, position, or graph-based centrality (e.g., TextRank). * Abstractive Summarization: Generates new sentences that capture the essence of the original text, potentially using words not present in the source. More human-like but much harder. Requires deeper language understanding and generation capabilities. Typically uses seq2seq models with attention or Transformer-based architectures (like BART, T5, GPT). Prone to “hallucinations” (generating incorrect facts).

Evaluation Metrics

35. Explain Precision, Recall, and F1-Score. When is F1-Score particularly useful?

These are common metrics for evaluating classification tasks: * Precision: Measures the accuracy of positive predictions. Of all instances the model predicted as positive, what fraction were actually positive? Formula: TP / (TP + FP) (TP=True Positives, FP=False Positives). High precision means fewer false positives. * Recall (Sensitivity): Measures how many of the actual positive instances the model correctly identified. Of all actual positive instances, what fraction did the model predict as positive? Formula: TP / (TP + FN) (FN=False Negatives). High recall means fewer false negatives. * F1-Score: The harmonic mean of Precision and Recall. Formula: 2 * (Precision * Recall) / (Precision + Recall). It provides a single score that balances both concerns. * Usefulness of F1: F1-Score is particularly useful when dealing with imbalanced datasets, where one class is much more frequent than the other(s). Accuracy can be misleading in such cases (e.g., predicting the majority class always might give high accuracy but is useless). F1 gives a better measure of performance by considering both false positives and false negatives.

36. What is Accuracy? When might it be a misleading metric?

Accuracy is the proportion of total predictions that were correct. Formula: (TP + TN) / (TP + TN + FP + FN) (TN=True Negatives). It measures overall correctness. * Misleading: Accuracy can be misleading, primarily in imbalanced datasets. For example, if 95% of emails are not spam, a model that always predicts “not spam” will achieve 95% accuracy but fail completely at the actual task of identifying spam. In such cases, Precision, Recall, F1-score, or AUC are better metrics.

37. What are BLEU and ROUGE scores used for?

These metrics are commonly used for evaluating the quality of generated text, particularly in machine translation and text summarization. * BLEU (Bilingual Evaluation Understudy): Primarily used for machine translation. It measures how closely the machine-generated translation matches one or more high-quality human reference translations. It calculates precision based on n-gram overlap (unigrams, bigrams, trigrams, etc.) between the candidate and reference translations, incorporating a brevity penalty to discourage overly short translations. Higher BLEU scores indicate better similarity to references. * ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Primarily used for text summarization (and sometimes machine translation). It measures recall based on n-gram overlap between the generated summary and human reference summaries. Different variants exist (ROUGE-N for n-gram recall, ROUGE-L for longest common subsequence, ROUGE-S for skip-bigrams). Higher ROUGE scores indicate better coverage of the content in the references.

38. What is Perplexity in the context of language models?

Perplexity is a common metric for evaluating the performance of language models. It measures how well a probability model predicts a sample. Lower perplexity indicates that the language model is better at predicting the sequence of words in a test set. It’s calculated as the exponential of the cross-entropy loss. Intuitively, it can be thought of as the (geometric) average branching factor of the model – the weighted average number of choices the model has when predicting the next word. A perplexity of 10 means the model is, on average, as confused as if it had to choose uniformly among 10 words at each step.

Practical Considerations & Challenges

39. How would you handle typos or misspelled words in NLP?

A39: * Spell Checkers/Correctors: Use libraries (like pyspellchecker, TextBlob, SymSpell) to identify and correct misspelled words based on edit distance (e.g., Levenshtein distance) and language models. * Character-level Embeddings/Models: Use models that operate at the character level (e.g., character CNNs, FastText embeddings which incorporate subword info) as they are inherently more robust to misspellings and OOV words. * Fuzzy Matching: Use fuzzy string matching techniques if comparing against known lists or dictionaries. * Data Augmentation: Intentionally introduce spelling errors into the training data to make the model more robust. * Ignore (Sometimes): If typos are rare and the model (especially large pre-trained ones) is robust enough, explicitly handling them might not be necessary or could introduce its own errors.

40. How do you deal with Out-Of-Vocabulary (OOV) words?

OOV words are words encountered during testing/inference that were not seen during training (and thus don’t have a representation in the model’s vocabulary). Handling methods include: * UNK Token: Represent all OOV words with a special (unknown) token. Simple, but loses information. * Character Embeddings: Build representations from characters, allowing the model to create vectors for unseen words. * Subword Tokenization: Use techniques like Byte Pair Encoding (BPE), WordPiece (used by BERT), or SentencePiece that break words into smaller subword units. Rare/OOV words can often be represented as a sequence of known subwords, mitigating the OOV problem significantly. This is standard in modern Transformer models. * FastText Embeddings: FastText explicitly learns embeddings for character n-grams, allowing it to construct vectors for OOV words based on their subword parts.

41. How can you handle imbalanced datasets in text classification?

Resampling Techniques: * Oversampling: Duplicate instances from the minority class (can lead to overfitting). Techniques like SMOTE (Synthetic Minority Over-sampling Technique) generate synthetic samples rather than just duplicating. * Undersampling: Remove instances from the majority class (can lead to loss of information). * Algorithmic Approaches: * Cost-Sensitive Learning: Assign higher misclassification costs to the minority class during model training (e.g., using class_weight parameters in libraries like scikit-learn). * Ensemble Methods: Use ensemble techniques like bagging or boosting, potentially combined with resampling (e.g., BalancedRandomForest, RUSBoost). * Use Appropriate Metrics: Focus on metrics like Precision, Recall, F1-Score, AUC-PR (Area Under Precision-Recall Curve), or Matthews Correlation Coefficient (MCC) instead of accuracy. * Generate More Data: If feasible, collect more data, especially for the minority class. * Anomaly Detection: Frame the problem as anomaly detection if the minority class is extremely rare and represents abnormal behavior.

42. What is the difference between Syntactic and Semantic Analysis in NLP?

Syntactic Analysis (Parsing): Focuses on the grammatical structure of a sentence. It analyzes how words are arranged according to grammatical rules to form valid sentences. Tasks include Part-of-Speech (POS) tagging and constituency/dependency parsing (identifying phrases, clauses, and relationships like subject-verb-object). It deals with form and structure. * Semantic Analysis: Focuses on understanding the meaning of words, phrases, and sentences, and their relationships. It deals with ambiguity, context, and the intended message. Tasks include Word Sense Disambiguation (WSD), sentiment analysis, coreference resolution (identifying mentions that refer to the same entity), and relationship extraction.

43. How can you incorporate domain knowledge into an NLP model? 

Feature Engineering: Manually create features based on domain-specific dictionaries, ontologies, or rules (e.g., identifying medical terms, financial concepts). * Data Augmentation/Generation: Use domain knowledge to generate more relevant training data or augment existing data. * Pre-training/Fine-tuning: Pre-train or fine-tune language models on domain-specific corpora (e.g., BioBERT for biomedical text, FinBERT for financial text). * Hybrid Approaches: Combine rule-based systems capturing domain knowledge with machine learning models. * Knowledge Graph Integration: Inject knowledge from domain-specific knowledge graphs into the model’s embeddings or architecture.

44. Explain what Regular Expressions (Regex) are and give an NLP use case.

Regular Expressions are sequences of characters that define a search pattern. They are powerful tools for pattern matching within strings. * NLP Use Cases: * Data Cleaning: Removing HTML tags, URLs, email addresses, phone numbers, or special characters from text. * Tokenization: Defining custom rules for splitting text into tokens (e.g., splitting based on specific punctuation). * Feature Extraction: Extracting specific patterns like dates, times, monetary values, or product codes. * Simple Information Extraction: Finding mentions of specific patterns without complex models (e.g., extracting all capitalized words as potential proper nouns, though NER models are better).

45. What are N-grams? How are they used?

N-grams are contiguous sequences of N items (words, characters, etc.) from a given sample of text or speech. * Unigram (N=1): Single words (“the”, “cat”). * Bigram (N=2): Sequences of two words (“the cat”, “cat sat”). * Trigram (N=3): Sequences of three words (“the cat sat”, “cat sat on”). * Uses: * Language Modeling: Predicting the next word based on the preceding N-1 words (traditional N-gram language models). * Feature Extraction: Using counts or presence/absence of specific N-grams as features for text classification models (e.g., using bigrams can capture some local word order information missed by BoW). * Evaluation Metrics: Used in metrics like BLEU and ROUGE to compare generated text with reference text. * Spell Correction, Text Completion.

Advanced & Emerging Topics

46. What is Zero-shot and Few-shot learning in the context of LLMs?

These refer to the ability of large pre-trained models to perform tasks they were not explicitly trained for: * Zero-shot Learning: The model performs a task based only on a natural language description (the prompt) without seeing any examples of that specific task during its fine-tuning phase (though it learned related patterns during pre-training). Example: Prompting GPT-4 to classify movie review sentiment without ever fine-tuning it on sentiment analysis. * Few-shot Learning: The model is given a few examples (typically 1 to ~32) of the task within the prompt itself, in addition to the task description. It uses these examples as conditioning to understand the desired output format and perform the task on a new input. Example: Providing 3 examples of sentence pairs and their entailment status before asking the model to classify a new pair.

47. Briefly explain Vector Databases and their relevance to NLP/LLMs.

Vector Databases are databases specifically designed to store, manage, and search high-dimensional vector embeddings efficiently. * Relevance: LLMs and other NLP models often represent text (words, sentences, documents) as dense vectors (embeddings). To perform tasks like semantic search (finding documents similar in meaning, not just keywords), recommendation, or Retrieval-Augmented Generation (RAG), we need to quickly find vectors similar to a query vector among millions or billions of stored vectors. Vector databases use specialized indexing algorithms (like HNSW, IVF, LSH) to perform Approximate Nearest Neighbor (ANN) searches much faster than exact searches, making these applications feasible at scale.

48. What is Retrieval-Augmented Generation (RAG)?

A48: RAG is an approach to improve the factual consistency and relevance of LLM outputs, especially for knowledge-intensive tasks. Instead of relying solely on the knowledge implicitly stored in the LLM’s parameters (which can be outdated or inaccurate), RAG first retrieves relevant documents or passages from an external knowledge source (like a document collection indexed in a vector database) based on the input prompt/query. It then provides these retrieved documents as additional context to the LLM along with the original prompt, enabling the model to generate a response that is grounded in the retrieved information. This helps reduce hallucinations and allows the model to access up-to-date or domain-specific knowledge.

49. How might you evaluate the “trustworthiness” or “safety” of an LLM’s output?

Evaluating trustworthiness/safety is complex and multi-faceted: * Factuality/Accuracy: Check against known facts or ground truth sources. Use metrics like factual consistency or develop specific benchmarks (e.g., TruthfulQA). RAG can help improve this. * Bias Detection: Use bias benchmarks (like BBQ, BOLD) or probing techniques to measure social biases in outputs. Evaluate fairness across different demographic groups. * Toxicity Detection: Use classifiers trained to detect hate speech, toxicity, or harmful content in the generated text. Evaluate against safety benchmarks (e.g., ToxiGen). * Robustness: Test how the model’s output changes with slight perturbations in the input (adversarial attacks). * Calibration: Assess whether the model’s confidence scores reflect its actual accuracy. * Hallucination Detection: Develop methods or metrics to identify when the model generates plausible but fabricated information. * Human Evaluation: Ultimately, human judgment is often required to assess nuanced aspects of safety, bias, and helpfulness through structured evaluations or red-teaming exercises.

50. Where do you see the field of NLP heading in the next few years (as of 2025)?

(This requires a forward-looking, opinionated answer based on current trends) * Continued Scaling (but with Efficiency Focus): While models will likely continue to grow, there will be increased focus on efficiency, smaller specialized models, quantization, and techniques like Mixture-of-Experts (MoE) to manage computational costs. * Multimodality: Integration of text with other modalities like images, audio, and video will become more seamless and powerful (e.g., models understanding charts in documents, generating video descriptions). * Improved Reasoning & Factuality: Significant research effort will target improving the logical reasoning capabilities of LLMs and reducing hallucinations, potentially through better architectures, training methods, and integration with external knowledge sources (like RAG and knowledge graphs). * Personalization & Customization: Easier ways to adapt large models to specific domains, tasks, or individual user preferences without full retraining. * Enhanced Agentic Capabilities: LLMs acting more like agents, capable of using tools, planning complex tasks, and interacting with external systems. * Ethics, Safety, and Alignment: Increased focus on developing robust methods for ensuring LLMs are safe, unbiased, controllable, and aligned with human values. Regulation will likely play a growing role. * On-device NLP: More powerful NLP capabilities running directly on edge devices (phones, laptops) for privacy and latency benefits.