Flow Diagram of Data Lakehouse
While Data Lake is excels for Machine Learning , Data warehouse is used for Business Intelligence , Data Lakehouse excels at both.
Data Lakehouse supports :
Flow Diagram of Data Warehouse
Flow Diagram of Data Lake
Medallion Architecture
Kafka for Fan Out Design Pattern
where one stream is converted to multiple allowing more than one consumer
Machines performing tasks that normally require human intelligence.
Subset of AI where systems learn patterns from data.
Uses neural networks with multiple layers.
Creates new content such as:
| Term | Meaning |
|---|---|
| Training | Model learns from data |
| Inference | Model makes predictions using learned knowledge |
Examples:
Text is broken into small units called tokens.
Example:
I love AWS
May become:
[I] [love] [AWS]
Model generates incorrect information while sounding confident.
Amount of information an LLM can consider at once.
Large pretrained model that can be adapted for many tasks.
Examples:
Retraining model with domain-specific data.
Instead of retraining:
Benefits:
| RAG Step | Purpose | Model Type | Example Models |
|---|---|---|---|
| 1. Document Ingestion | Read PDFs, DOCX, HTML, Images | OCR / Document AI | Tesseract, LayoutLM, Donut |
| 2. Chunking | Split documents into passages | Rule-based / NLP | Sentence Splitter, Recursive Text Splitter |
| 3. Text → Embeddings | Convert chunks into vectors | Embedding Model (Encoder-only Transformer) | BERT, Sentence-BERT, E5, BGE |
| 4. Vector Storage | Store embeddings | Vector Database | FAISS, Milvus, Weaviate, Pinecone |
| 5. Query → Embedding | Convert user query to vector | Same Embedding Model | BGE, E5, SBERT |
| 6. Retrieval | Find nearest chunks | ANN Search Algorithm | HNSW, IVF, Flat Search |
| 7. Re-ranking (Optional) | Improve retrieved results | Cross Encoder | MonoBERT, Cohere Rerank, BGE Reranker |
| 8. Context Construction | Build prompt with retrieved chunks | Prompt Builder | Template Engine |
| 9. Answer Generation | Generate final answer | Decoder-only LLM | GPT-4o, Claude Sonnet, Llama 3, Mistral |
| 10. Citation Generation (Optional) | Show sources | LLM / Metadata Layer | GPT-4o, Claude |
| Step | Transformer Type |
|---|---|
| Embedding Generation | Encoder-only |
| Re-ranking | Encoder-only (Cross Encoder) |
| Answer Generation | Decoder-only |
| Translation (optional) | Encoder-Decoder |
| Summarization (optional) | Encoder-Decoder |
| OCR Understanding | Encoder or Encoder-Decoder |
| Multimodal RAG | Vision Encoder + LLM Decoder |
| Transformer Family | Example Models | Used For |
|---|---|---|
| Encoder-only | BERT, RoBERTa, SBERT, E5, BGE | Embeddings, Retrieval |
| Decoder-only | GPT, Llama, Claude, Mistral, Qwen | Generation |
| Encoder-Decoder | T5, FLAN-T5, BART | Summarization, Translation |
| Vision Encoder | ViT, CLIP Vision Encoder | Image Embeddings |
| Vision-Language | LLaVA, Qwen-VL, GPT-4o | Multimodal RAG |
| Layer | Common Choice |
|---|---|
| Chunking | LangChain Recursive Splitter |
| Embeddings | BGE-large, E5-large |
| Vector DB | FAISS, Milvus |
| Retrieval | HNSW |
| Re-ranker | BGE-Reranker |
| Generator | GPT-4o, Claude, Llama 3 |
| Orchestration | LangChain, LlamaIndex |
Documents
↓
Chunking
↓
Encoder Model
(BERT / E5 / BGE)
↓
Embeddings
↓
Vector DB
↓
End
User Query
↓
Encoder Model
(BERT / E5 / BGE)
↓
Similarity Search
↓
Top K Chunks
↓
Cross Encoder Re-ranker
(Optional)
↓
Prompt Construction
↓
Decoder LLM
(GPT / Claude / Llama)
↓
Final AnswerUser Memory Pipeline (Mem0)
User Conversations
↓
Mem0
(Extracts Important Facts)
↓
Memory Store
(Vector DB / PostgreSQL / Redis)
↓
EndQuery-Time Retrieval Pipeline
User Query
↓
Encoder Model
(BERT / E5 / BGE)
├─────────────────────────────┐
│ │
▼ ▼
Similarity Search Mem0 Memory Retrieval
(Vector DB) (Relevant User Memories)
│ │
▼ ▼
Top K Chunks Relevant Memories
│ │
└──────────────┬──────────────┘
▼
Cross Encoder Re-ranker
(Optional, for documents)
▼
Prompt Construction
(Documents + User Memory + Query)
▼
Decoder LLM
(GPT / Claude / Llama)
▼
Final Answer
▼
Mem0 Memory Update
(Store new preferences/facts if needed)
| Task | Model Type |
|---|---|
| Create Embeddings | Encoder-only |
| Retrieve Documents | Vector Search |
| Re-rank Results | Cross Encoder |
| Generate Answer | Decoder-only LLM |
| Summarize Documents | Encoder-Decoder |
| Multimodal Retrieval | CLIP / Vision Encoder |
| Multimodal Generation | GPT-4o / Gemini / Qwen-VL |
Translate this sentence to French.
Provide one example.
Provide multiple examples.
Ask model to reason step-by-step.
At a high level, an LLM is a Transformer-based neural network that converts input text into tokens, processes them through multiple Transformer blocks, and predicts the next token repeatedly to generate text.
Input Text
│
▼
Text Tokenization
│
▼
Token IDs (Integers)
│
▼
Token Embedding Layer
│
▼
Positional Encoding/Embedding
│
▼
N Transformer Decoder Blocks
│
▼
Final Hidden Representation
│
▼
Linear (Output Projection)
│
▼
Softmax Layer
│
▼
Probability for Every Token
│
▼
Select Next Token (Sampling)
│
▼
Append to Input & Repeat
| Component | Purpose |
|---|---|
| Tokenizer | Converts text into tokens |
| Embedding Layer | Converts token IDs into dense vectors |
| Positional Encoding | Gives the model information about token order |
| Transformer Decoder Blocks | Learns relationships between tokens |
| Multi-Head Self-Attention | Determines which words are important |
| Feed Forward Network (FFN) | Learns complex patterns |
| Residual Connections | Prevent information loss |
| Layer Normalization | Stabilizes training |
| Output Linear Layer | Maps hidden vectors to vocabulary logits |
| Softmax | Converts logits into probabilities |
| Sampling Strategy | Chooses the next token |
The tokenizer splits text into tokens.
Example:
Input
I love machine learning
Tokenizer
["I", "love", "machine", "learning"]
Convert to IDs
[52, 908, 3210, 4567]
The neural network only understands numbers.
Each token ID is mapped to a dense vector.
Example
Vocabulary
I → 52
love → 908
machine → 3210
Embedding
52
↓
[0.21, -0.84, 1.02, ..., 0.67]
Instead of one integer,
every token becomes a high-dimensional vector.
Typical embedding size
| Model | Embedding Dimension |
|---|---|
| GPT-2 Small | 768 |
| Llama 7B | 4096 |
| GPT-3 | 12288 |
Attention alone has no concept of sequence order.
Without positional information,
these two sentences appear identical.
Dog bites man
Man bites dog
Positional embeddings add information like:
Dog Position 0
bites Position 1
man Position 2
Final embedding
Token Embedding
+
Position Embedding
This is where most computation happens.
Modern LLMs stack many identical decoder blocks.
Examples:
| Model | Decoder Blocks |
|---|---|
| GPT-2 Small | 12 |
| GPT-3 175B | 96 |
| Llama 3 8B | 32 |
| DeepSeek-R1 671B | Hundreds of expert layers (MoE architecture) |
Each block contains:
Input
│
▼
LayerNorm
│
▼
Masked Multi-Head Attention
│
▼
Residual Connection
│
▼
LayerNorm
│
▼
Feed Forward Network
│
▼
Residual Connection
│
▼
Output
This is the core innovation of Transformers.
Every word looks at every previous word.
Sentence
The animal didn't cross the road because it was tired.
When processing "it"
attention may focus on
animal
road
tired
The model learns that "it" refers to animal.
Each token produces three vectors.
Embedding
↓
Query (Q)
Key (K)
Value (V)
Meaning
| Vector | Purpose |
|---|---|
| Query | What am I looking for? |
| Key | What information do I contain? |
| Value | Information to pass forward |
Attention score
Q × Kᵀ
↓
Similarity Score
↓
Softmax
↓
Attention Weights
↓
Weighted Sum of Values
Formula
Attention(Q,K,V)
=
softmax(QKᵀ / √dk)V
Instead of one attention calculation,
multiple attention heads run in parallel.
Example
Head 1
Grammar
----------------
Head 2
Subject
----------------
Head 3
Object
----------------
Head 4
Long-range dependency
Outputs are concatenated.
After attention,
every token passes through the same small neural network independently.
Typical structure
Linear
↓
Activation (GELU, SwiGLU, etc.)
↓
Linear
Purpose
Instead of replacing the input,
the block adds the original input back.
Output
=
Input
+
Attention Output
Benefits
Keeps activations stable during training.
Without it,
training becomes unstable as models get deeper.
Final hidden vector
↓
Linear layer
↓
Vocabulary size
Example vocabulary
50,000 words
Output
Dog 3.2
Cat 8.9
Car 1.4
Paris 10.1
These are logits (unnormalized scores).
Softmax converts logits into probabilities.
Example
Paris 0.72
London 0.12
Berlin 0.08
Rome 0.08
Probabilities sum to 1.
The next token is selected.
Methods include:
| Method | Description |
|---|---|
| Greedy | Choose the highest probability token |
| Beam Search | Explore multiple candidate sequences simultaneously |
| Top-k Sampling | Sample only from the top k most probable tokens |
| Top-p (Nucleus) Sampling | Sample from the smallest set of tokens whose cumulative probability exceeds p |
| Temperature Sampling | Adjust randomness by scaling logits before Softmax |
Generated token
↓
Append to prompt
↓
Run the decoder again
↓
Predict the next token
This repeats until an end-of-sequence token or another stopping condition is reached.
The original Transformer introduced in the paper "Attention Is All You Need" had two parts:
Encoder
↓
Decoder
LLMs such as GPT, Llama, Qwen, and DeepSeek use only the decoder stack with masked (causal) self-attention, so each token can attend only to itself and previous tokens. This enables autoregressive next-token prediction.
Input:
The capital of France is
Processing:
Text
│
▼
Tokenizer
│
▼
Token IDs
│
▼
Embeddings + Position Embeddings
│
▼
32–100+ Transformer Decoder Blocks
│
▼
Hidden Representation
│
▼
Linear Projection
│
▼
Softmax
│
▼
Predicted Token: "Paris"
│
▼
Append "Paris" to the input
│
▼
Predict the next token
| Component | Function |
|---|---|
| Tokenizer | Converts text to token IDs |
| Embedding Layer | Converts token IDs to dense vectors |
| Positional Embedding | Encodes token order |
| Masked Multi-Head Self-Attention | Captures relationships with previous tokens |
| Feed Forward Network | Learns nonlinear transformations |
| Residual Connections | Preserve information and improve gradient flow |
| Layer Normalization | Stabilizes training |
| Linear Output Layer | Projects hidden states to vocabulary logits |
| Softmax | Produces probabilities over the vocabulary |
| Sampling | Selects the next token for generation |
Modern LLMs extend this core architecture with optimizations such as Rotary Positional Embeddings (RoPE), Grouped Query Attention (GQA) or Multi-Query Attention (MQA), Mixture of Experts (MoE), FlashAttention, KV caching, and quantization, but the fundamental decoder-only Transformer pipeline remains the sam
Avoid bias.
Understand why model produced output.
Protect user data.
Model behaves reliably.
Users know AI is involved.
Evaluation
| Model Group | Evaluation | Description of the Evaluation |
|---|---|---|
| Agentic Models | AgentBench | Evaluates autonomous task execution, planning, tool usage, and multi-step reasoning. |
| Agentic Models | GAIA | Tests real-world assistant capabilities like searching, tool calling, and reasoning. |
| Agentic Models | SWE-bench | Measures ability to solve real GitHub issues by editing codebases. |
| Bi-Encoder | BEIR | Evaluates embedding-based retrieval across multiple datasets and domains. |
| Bi-Encoder | MTEB | Measures embedding quality across retrieval, clustering, classification, and reranking tasks. |
| Bi-Encoder | MS MARCO | Evaluates dense retrieval and passage ranking performance. |
| Cross-Encoder | BEIR | Measures pairwise query-document relevance scoring quality. |
| Cross-Encoder | MS MARCO | Evaluates reranking precision for query-passage relevance. |
| Cross-Encoder | TREC Deep Learning Track | Measures ranking quality for search relevance tasks. |
| Decoder-only | GSM8K | Evaluates arithmetic and multi-step mathematical reasoning. |
| Decoder-only | HellaSwag | Measures commonsense reasoning and next-sentence prediction. |
| Decoder-only | HumanEval | Evaluates code generation correctness using executable unit tests. |
| Decoder-only | MMLU | Measures broad knowledge and reasoning across many academic domains. |
| Decoder-only | MT-Bench | Tests instruction following and conversational quality. |
| Decoder-only | Needle-in-a-Haystack | Measures ability to retrieve specific information from long contexts. |
| Decoder-only | TruthfulQA | Tests factual consistency and resistance to hallucinations. |
| Encoder-decoder | BLEU | Measures overlap between generated text and reference text, mainly for translation. |
| Encoder-decoder | ROUGE | Evaluates summarization quality based on n-gram overlap. |
| Encoder-decoder | SQuAD | Measures extractive question-answering accuracy. |
| Encoder-only | GLUE | Evaluates language understanding tasks like sentiment, entailment, and similarity. |
| Encoder-only | MTEB | Measures embedding performance across multiple NLP tasks. |
| Encoder-only | STS-B | Measures how well embeddings capture sentence similarity. |
| Encoder-only | SuperGLUE | Harder version of GLUE for advanced reasoning tasks. |
| Long-context Models | InfiniteBench | Evaluates memory retention and reasoning over very long contexts. |
| Long-context Models | LongBench | Tests summarization, retrieval, and reasoning on long documents. |
| Long-context Models | Needle-in-a-Haystack | Measures retrieval accuracy from large contexts. |
| Multimodal Models | MMBench | Evaluates image understanding and multimodal reasoning. |
| Multimodal Models | MMMU | Measures multimodal reasoning across academic and professional domains. |
| Multimodal Models | MMVet | Tests advanced visual reasoning and perception. |
| RAG Systems | CRUD-RAG | Measures retrieval robustness and update handling in RAG pipelines. |
| RAG Systems | RAGAS | Evaluates faithfulness, context precision, context recall, and answer relevance in RAG. |
| Reward Models | RewardBench | Evaluates preference model quality and alignment performance. |
| Tool-use Models | ToolBench | Measures correctness in tool selection, API usage, and tool chaining. |
Most important service for the exam.
Provides access to foundation models from:
Features:
Enterprise chatbot over company data.
Developer coding assistant.
Build, train and deploy ML models.
Sagemaker Inference
Sagemaker Pipeline
- Batch mode saves cost by preventing GPU under utilization.
AWS secures:
Customer secures:
Identity and access management.
Data:
Spam or Not Spam
Positive / Negative
Long article -> Short summary
Customer support
Generate Java/Python code
Invoice extraction
| Service | Purpose |
|---|---|
| Bedrock | Generative AI |
| SageMaker AI | Build/train ML models |
| Comprehend | NLP analysis |
| Rekognition | Image analysis |
| Textract | Document extraction |
| Transcribe | Speech to text |
| Polly | Text to speech |
| Translate | Translation |
| Topic | Remember |
|---|---|
| Generative AI | Creates new content |
| Foundation Model | Large pretrained model |
| Hallucination | Confident wrong answer |
| RAG | Retrieve + Generate |
| Bedrock | Managed GenAI platform |
| SageMaker | ML lifecycle |
| Guardrails | Safety controls |
| Fine-Tuning | Retrain model |
| Inference | Model prediction |
| Token | Small text unit |
There are several ways to represent the position of tokens in a Transformer. Although people often say "embeddings like RoPE," RoPE is actually a positional encoding/embedding technique, not a semantic embedding like a word embedding.
Below are the major types of positional embeddings/encodings used in Transformers.
| Method | Learnable | Extrapolates to Longer Sequences | Used In |
|---|---|---|---|
| Absolute Positional Embedding (APE) | Yes | ❌ No | GPT-2, BERT |
| Sinusoidal Positional Encoding | No | ✅ Yes | Original Transformer |
| Relative Positional Embedding (RPE) | Usually Yes | Better than APE | T5, Transformer-XL |
| Rotary Positional Embedding (RoPE) | No | ✅ Yes | Llama, Qwen, DeepSeek, GPT-NeoX |
| ALiBi (Attention with Linear Biases) | No | ✅ Excellent | BLOOM, MPT |
| xPos | No | ✅ Better than RoPE | Long-context models |
| Dynamic RoPE | No | ✅ Improved long-context support | Some Llama variants |
| NTK-aware RoPE Scaling | No | ✅ Extends context window | Llama long-context adaptations |
| YaRN (Yet another RoPE extensioN) | No | ✅ Very good | Long-context fine-tuned LLMs |
| LeX (Length Extrapolation) | Varies | ✅ Designed for long context | Research models |
Each position has its own learnable vector.
Example
Position 0 → [0.1, 0.3, ...]
Position 1 → [0.4, 0.8, ...]
Position 2 → [0.2, 0.9, ...]
Final input
Token Embedding
+
Position Embedding
Advantages
Disadvantages
Used in:
Introduced in the original Transformer paper.
Uses sine and cosine functions.
Formula
PE(pos,2i)=sin(pos/10000^(2i/d))
PE(pos,2i+1)=cos(pos/10000^(2i/d))
Advantages
Disadvantages
Used in:
Instead of absolute positions,
the model learns the relative distance.
Example
Dog is 2 words before cat.
The model learns
Distance = +2
instead of
Dog at position 5
Cat at position 7
Advantages
Used in:
The most popular method in modern LLMs.
Instead of adding a positional vector,
RoPE rotates the Query and Key vectors by an angle based on position.
Embedding
↓
Query
↓
Rotate by position angle
↓
Attention
Advantages
Used in:
Instead of embeddings,
add a linear bias directly to attention scores.
Attention Score
+
Distance Bias
Farther tokens receive a progressively larger penalty.
Advantages
Used in:
An extension of RoPE.
Designed to improve long-sequence stability.
Instead of using a fixed rotation,
it rescales rotations.
Advantages
Standard RoPE uses fixed frequencies.
Dynamic RoPE adjusts frequencies dynamically for longer contexts.
Advantages
Originally developed for extending Llama's context window.
Idea
Stretch the RoPE frequencies.
Example
Original
4096 tokens
↓
Scaled
32768 tokens
No retraining is required in many implementations.
Advantages
Improves on NTK scaling.
Combines interpolation and scaling techniques.
Advantages
Research methods that explicitly optimize for longer contexts.
Goal
Train models that naturally generalize beyond the training context length.
| Model | Positional Method |
|---|---|
| GPT-2 | Absolute Positional Embedding |
| BERT | Absolute Positional Embedding |
| Original Transformer | Sinusoidal Encoding |
| Transformer-XL | Relative Positional Encoding |
| T5 | Relative Positional Encoding |
| DeBERTa | Relative Position Bias |
| GPT-NeoX | RoPE |
| Llama 1/2/3 | RoPE |
| Qwen | RoPE |
| DeepSeek | RoPE |
| Mistral | RoPE |
| BLOOM | ALiBi |
| MPT | ALiBi |
In addition to positional embeddings, LLMs use several other embedding types.
| Embedding Type | Purpose |
|---|---|
| Token Embedding | Represents each token as a dense vector |
| Positional Embedding/Encoding | Represents token order (e.g., RoPE, ALiBi, APE) |
| Segment (Token Type) Embedding | Distinguishes sentence A from sentence B (used in BERT) |
| Word Embedding | Maps words/subwords to vectors |
| Character Embedding | Represents characters instead of words |
| Sentence Embedding | Represents an entire sentence with a single vector |
| Document Embedding | Represents an entire document |
| Instruction Embedding | Encodes task or instruction information in some architectures |
| Multimodal Embedding | Maps text, images, audio, etc., into a shared embedding space |
| Method | Basic Idea | Best For | Limitation |
|---|---|---|---|
| Absolute Positional Embedding | Add a learned vector for each position | Short, fixed-length sequences | Cannot extrapolate well |
| Sinusoidal Encoding | Use deterministic sine/cosine functions | General sequence modeling | Less expressive than learned methods |
| Relative Positional Embedding | Encode distances between tokens | Better relative reasoning | More complex attention computation |
| RoPE | Rotate Query/Key vectors based on position | Modern LLMs with long context | Standard RoPE still has finite context limits |
| ALiBi | Add a linear distance bias to attention scores | Efficient long-context inference | May underperform RoPE on some benchmarks |
| xPos / Dynamic RoPE / NTK Scaling / YaRN | Variants that improve or extend RoPE | Very long-context LLMs (32K–1M+ tokens) | Additional implementation complexity |
Today, RoPE has become the de facto standard for decoder-only LLMs (Llama, Qwen, Mistral, DeepSeek), while ALiBi is valued for its simplicity and excellent extrapolation, and RoPE extensions such as NTK-aware scaling and YaRN are commonly used to extend context windows without retraining the entire model
A2A (Agent-to-Agent) is a communication protocol that allows AI agents built by different frameworks, vendors, or organizations to discover each other, exchange messages, delegate tasks, and collaborate in a standardized way.
Without A2A, each AI agent operates like an isolated application.
Instead of building one giant agent that knows everything, organizations build specialized agents.
Example:
Suppose a user asks:
"Book a flight for my business trip and make sure it fits my team's calendar and budget."
No single agent may have all the required capabilities.
Using A2A:
User
│
▼
Travel Agent
│
├──► Calendar Agent → Check availability
│
├──► Finance Agent → Verify budget
│
└──► Approval Agent → Manager approval
Each agent performs its specialty and returns the result.
In large companies:
Each team may use:
Without a common protocol, every integration requires custom APIs.
A2A provides a common language for communication.
Without A2A:
HR Agent
│
Custom REST API
│
Finance Agent
Travel Agent
│
Different API
│
Calendar Agent
Each pair of agents needs custom integration.
If there are N agents, integrations can grow roughly as N × (N - 1) / 2 in the worst case.
With A2A:
HR Agent
│
Finance Agent
│
Travel Agent
│
Calendar Agent
Everyone speaks the same protocol.
Suppose an agent needs legal advice.
Without A2A:
With A2A:
Planning Agent
↓
Discover
↓
Legal Agent
↓
Capabilities
↓
Can review contracts
Can summarize regulations
The planning agent discovers and uses the legal agent dynamically.
A planning agent doesn't need to solve every problem.
Example:
User
↓
Planning Agent
↓
Generate report
↓
Data Agent
↓
Fetch sales
↓
Analytics Agent
↓
Generate insights
↓
Visualization Agent
↓
Create charts
Each task is delegated to the most suitable agent.
Imagine:
Without A2A, these agents need custom integration.
With A2A, they can collaborate using a shared protocol, regardless of who built them.
Agents may run:
A2A enables communication across these environments without tightly coupling implementations.
Instead of building the same capability repeatedly:
Expense Agent
used by
Finance Team
HR Team
Travel Team
Audit Team
A single specialized agent can serve multiple workflows.
Consider an online shopping scenario.
User
↓
Shopping Agent
↓
Inventory Agent
Check stock
↓
Pricing Agent
Apply discounts
↓
Payment Agent
Process payment
↓
Shipping Agent
Arrange delivery
↓
Notification Agent
Send confirmation
Each agent focuses on its own domain, and A2A coordinates their interaction.
| Benefit | Explanation |
|---|---|
| Interoperability | Agents from different frameworks and vendors can work together. |
| Reusability | Specialized agents can be reused across multiple applications. |
| Scalability | New agents can be added without redesigning existing integrations. |
| Dynamic discovery | Agents can discover available capabilities at runtime. |
| Delegation | Complex tasks are split among specialized agents. |
| Maintainability | Reduces the need for numerous custom point-to-point integrations. |
| Vendor independence | Avoids locking systems into a single AI framework or provider. |
These protocols solve different problems and are often used together.
| A2A (Agent-to-Agent) | MCP (Model Context Protocol) |
|---|---|
| Connects agents | Connects an AI model or agent to tools and data sources |
| Agent ↔ Agent | Agent ↔ Tool |
| Used for collaboration and delegation | Used for accessing capabilities like databases, APIs, files, and SaaS services |
| Enables multi-agent workflows | Enables tool invocation and context retrieval |
| Example: Travel Agent asks Finance Agent to approve a budget | Example: Finance Agent queries a PostgreSQL database or invokes a payment API |
A common architecture is:
User
│
▼
Planning Agent
│
┌─────────┴─────────┐
▼ ▼
Travel Agent Finance Agent
│ │
(MCP) (MCP)
│ │
Flight API ERP Database
Hotel API Budget Service
Here, A2A allows the planning, travel, and finance agents to communicate, while MCP lets each agent interact with the external tools and data sources it needs. Together, they enable modular, interoperable, and scalable AI system
Building an LLM can mean very different things depending on your goal. There are three common paths:
| Goal | Time | GPUs Needed | Cost | Example |
|---|---|---|---|---|
| Train from scratch | Months | Hundreds to thousands | Millions of dollars | GPT, Llama, DeepSeek |
| Continue pre-training an existing model | Days to weeks | 8–128 GPUs | Thousands to tens of thousands | Domain-specific Llama |
| Fine-tune an existing model | Hours to days | 1–8 GPUs | Tens to hundreds of dollars | Chatbot, coding assistant |
If your goal is to understand how companies like OpenAI, Meta, or DeepSeek build an LLM, the lifecycle looks like this.
Data Collection
│
▼
Data Cleaning
│
▼
Tokenizer Training
│
▼
Dataset Tokenization
│
▼
Transformer Design
│
▼
Distributed Training
│
▼
Checkpoint Saving
│
▼
Model Evaluation
│
▼
Instruction Fine-tuning
│
▼
Preference Tuning
(RLHF / DPO / GRPO / RFT)
│
▼
Safety Alignment
│
▼
Quantization (Optional)
│
▼
Inference Optimization
(vLLM / SGLang / TensorRT)
│
▼
Production Deployment
An LLM learns from enormous amounts of text.
Typical sources include:
Example:
Wikipedia
↓
50 TB
Books
↓
20 TB
GitHub
↓
30 TB
Research Papers
↓
10 TB
Total
↓
110 TB Raw Data
The raw data is noisy and cannot be used directly.
Remove:
Example:
110 TB
↓
Cleaning
↓
35 TB High Quality Data
The model cannot understand characters directly.
Instead it converts text into tokens.
Example:
Artificial Intelligence
↓
Artificial
Intelligence
or
playing
↓
play
ing
Modern models typically use:
Vocabulary size:
32,000
or
50,000
or
128,000 tokens
Every document becomes integers.
Example
Hello World
↓
[15496, 2787]
The neural network only sees numbers.
Typical architecture:
Input Tokens
↓
Embedding Layer
↓
Transformer Block
↓
Transformer Block
↓
Transformer Block
↓
...
↓
Final Linear Layer
↓
Vocabulary Probabilities
Each transformer block contains:
LayerNorm
↓
Multi-Head Attention
↓
Residual
↓
LayerNorm
↓
Feed Forward Network
↓
Residual
Typical configuration:
| Parameter | Example |
|---|---|
| Layers | 32 |
| Hidden Size | 4096 |
| Attention Heads | 32 |
| Context Length | 8192 |
| Vocabulary | 128K |
| Parameters | 7B |
Larger models increase these values.
The objective is simple:
Predict the next token.
Example:
The capital of France is
↓
?
↓
Paris
Training samples:
The
↓
cat
↓
sat
↓
on
↓
the
↓
mat
For every token:
Input
↓
Forward Pass
↓
Prediction
↓
Loss
↓
Backpropagation
↓
Update Weights
This process repeats trillions of times.
One GPU is not enough.
Example:
1000 GPUs
↓
Each GPU trains part of model
↓
Synchronize gradients
↓
Repeat
Common techniques:
Frameworks:
Every few thousand training steps:
Model
↓
Checkpoint
↓
Resume Later
A checkpoint contains:
Benchmark the model.
Common evaluations:
Metrics:
Pre-trained models complete text but do not reliably follow instructions.
Train on instruction-response pairs.
Example:
Instruction
Explain recursion
↓
Response
Recursion is...
Datasets:
Methods:
Improve response quality based on human preferences.
Typical pipeline:
Prompt
↓
Two Responses
↓
Human chooses better one
↓
Preference Dataset
↓
Optimization
Methods:
These help produce responses that are more helpful, harmless, and aligned with user intent.
Reduce harmful or unsafe outputs.
Examples:
Reduce model size.
FP32
↓
FP16
↓
INT8
↓
INT4
Benefits:
Production serving includes:
Model
↓
vLLM
↓
Continuous Batching
↓
KV Cache
↓
Streaming
↓
REST API
Optimizations include:
Typical architecture:
Client
│
▼
API Gateway
│
▼
Load Balancer
│
▼
vLLM / SGLang Cluster
│
▼
GPU Servers
│
▼
Monitoring & Logs
| Stage | Common Tools |
|---|---|
| Data Collection | Common Crawl, Wikipedia dumps, GitHub archives |
| Data Processing | Apache Spark, Ray, Python |
| Tokenizer | SentencePiece, Hugging Face Tokenizers |
| Training | PyTorch |
| Distributed Training | DeepSpeed, Megatron-LM, FSDP |
| Experiment Tracking | Weights & Biases, MLflow |
| Storage | S3, HDFS |
| Fine-tuning | PEFT, LoRA, QLoRA |
| Evaluation | lm-evaluation-harness, custom benchmarks |
| Inference | vLLM, SGLang, TensorRT-LLM, TGI |
| Deployment | Docker, Kubernetes, NVIDIA GPUs |
If your goal is to build an LLM yourself rather than just use one, a practical progression is:
This path teaches the same core concepts used to build production LLM systems, while remaining feasible on accessible hardware before scaling up to larger models
| Phase | Responsible Area | Typical Engineer |
|---|---|---|
| Data Collection | Data pipelines | Data Engineer |
| Data Cleaning | Data processing | Data Engineer |
| Tokenization | Tokenizer development | ML Engineer / Research Engineer |
| Model Architecture | Transformer design | AI Research Scientist |
| Pre-training | Training algorithms | Research Scientist |
| Distributed Training | Multi-GPU optimization | Training Systems Engineer |
| Gradient Accumulation | Memory optimization | Training Systems Engineer |
| Mixed Precision | Training optimization | Training Systems Engineer |
| Checkpointing | Fault tolerance | Training Systems Engineer |
| Fine-tuning | Model adaptation | ML Engineer |
| RLHF/DPO | Alignment | Alignment Engineer |
| Quantization | Compression | ML Systems Engineer |
| Inference | Serving optimization | Inference Engineer |
| Deployment | Production infrastructure | MLOps Engineer |
Inference Engineering focuses on making models serve requests efficiently, while Training Engineering focuses on making models train efficiently.
Typical responsibilities include:
Managing:
Weights
↓
Gradients
↓
Optimizer States
↓
Activations
↓
KV Cache (for training)
Data Engineering
│
▼
Training Systems Engineering
(Gradient Accumulation, ZeRO, FSDP)
│
▼
AI Research Engineering
(Transformer, Attention, Losses)
│
▼
Alignment Engineering
(RLHF, DPO, Safety, Preference)
│
▼
Model Compression Engineering
(Quantization, Pruning, Distillation)
│
▼
Inference Engineering
(vLLM, SGLang, FlashAttention, KV Cache)
│
▼
MLOps / Platform Engineering
(Deployment, Autoscaling, Monitoring)
| Training Systems Engineering | Inference Engineering |
|---|---|
| Gradient Accumulation | Continuous Batching |
| Gradient Checkpointing | KV Cache |
| Mixed Precision Training | Quantization |
| ZeRO Optimizer | PagedAttention |
| FSDP | FlashAttention |
| DeepSpeed | vLLM |
| Megatron-LM | SGLang |
| Activation Checkpointing | Speculative Decoding |
| Distributed Optimizers | TensorRT-LLM |
| Optimizer Sharding | Streaming Tokens |
| Category | Common Frameworks |
|---|---|
| Distributed Training | PyTorch Distributed (DDP), FSDP, DeepSpeed, Megatron-LM |
| Memory Optimization | DeepSpeed ZeRO, Activation Checkpointing, Gradient Checkpointing |
| Precision | Automatic Mixed Precision (AMP), BF16, FP16 |
| Communication | NCCL, Gloo |
| Experiment Tracking | Weights & Biases, MLflow |
| Cluster Scheduling | Kubernetes, Slurm, Ray |
Flow Diagram of Data Lakehouse While Data Lake is excels for Machine Learning , Data warehouse is used for Business Intelligence , Data Lak...
