This tutorial examines a real implementation - an LLM ElasticSearch Agent, built using Google Agent Development Kit (yes, I am a hypocrite), that converts natural language queries into database operations - to understand what actually makes agentic systems work in production environments where failures matter.

Understanding What Agentic Systems Actually Solve

Before diving into architecture patterns, let me establish why you'd build an agentic system instead of a traditional application. Consider this user request: "Show me all failed login attempts from last week."

A traditional approach might involve a hardcoded query builder with preset filters. This works until users start asking variations like "Find security incidents from the past 7 days" or "Display authentication failures since Monday." Each variation requires code changes.

An agentic approach treats this as a multi-step reasoning problem. The system needs to understand that "failed login attempts" relates to authentication data, determine which database index contains this information, translate temporal expressions like "last week" into actual date ranges, generate the appropriate query syntax, execute it safely, and present results in human-readable format.

The key insight is that complex user requests often require multiple distinct capabilities working together. Rather than building one monolithic system that handles everything, agentic architectures decompose these requests into specialized agents that can be developed, tested, and debugged independently.

The Multi-Agent Pipeline Pattern

The most important architectural decision in my system was breaking the query process into distinct stages, each handled by a specialized agent. This isn't just good software engineering - it's essential for reliability when dealing with the unpredictable outputs that LLMs can produce.

My pipeline consists of three specialized agents working in sequence. The Index Selection Agent determines which ElasticSearch index contains the relevant data. The Query Generation Agent converts the natural language request into proper ElasticSearch DSL syntax. The Query Execution Agent runs the query safely and interprets results for the user.

class ElasticsearchPipelineAgent:
    def __init__(self):
        # Each agent specializes in one part of the problem
        self.index_selection_agent = create_index_selection_agent()
        self.query_generation_agent = create_query_generation_agent()
        self.query_execution_agent = create_query_execution_agent()

        # SequentialAgent coordinates the pipeline
        self.agent = SequentialAgent(
            name="ElasticsearchPipelineAgent",
            sub_agents=[
                self.index_selection_agent.agent,
                self.query_generation_agent.agent,
                self.query_execution_agent.agent,
            ],
        )

This design provides several practical advantages. First, each agent can be optimized for its specific task with tailored prompts and tools. The Index Selection Agent uses tools for discovering available indices and analyzing their schemas, while the Query Execution Agent focuses on safe query execution and result formatting. Second, failures can be isolated and handled appropriately at each stage. If index selection fails, you know exactly where to look and can potentially recover by prompting the user for clarification. Third, the system becomes much easier to test and debug when you can examine each stage independently.

The sequential approach also handles the state management challenge that trips up many agentic systems. Each agent receives the complete context from previous agents, building up the information needed for the final query execution.

Structured Output: The Foundation of Reliability

The biggest practical challenge in building reliable agentic systems is ensuring that agents produce consistent, parseable outputs that subsequent agents can work with reliably. This is where structured output schemas become absolutely critical.

Each agent in my pipeline defines exactly what it will output using Pydantic models. This isn't just good practice - it's essential for system reliability. Here's what the Index Selection Agent produces:

class IndexSelectionOutput(BaseModel):
    """Structured output for the Index Selection Agent."""

    selected_index: Optional[str] = Field(
        description="Name of the selected index, or null if selection failed"
    )
    index_schema: Optional[IndexSchema] = Field(
        description="Complete schema information for the selected index"
    )
    selection_metadata: SelectionMetadata = Field(
        description="Metadata about the selection process"
    )
    validation: ValidationResult = Field(
        description="Validation results for the selection"
    )

Notice how this structure handles partial failures gracefully. If index selection fails, the selected_index field is null, but the selection_metadata explains why it failed and the validation section provides specific error information. This allows the system to provide meaningful feedback to users rather than cryptic error messages.

The validation component is particularly important. Each agent validates not just its inputs, but also its outputs before passing them to the next stage. The Index Selection Agent doesn't just return an index name - it verifies that the index actually exists, that the schema was retrieved successfully, and that the selection is ready for query generation.

This validation-first approach catches problems early in the pipeline rather than letting them cascade through multiple agents before failing in confusing ways.

Tool Integration: Connecting Agents to Real Systems

Agents become useful when they can interact with external systems through tools. However, tool integration introduces significant complexity around error handling, security, and connection management that doesn't exist in simple LLM applications.

My ElasticSearch tools demonstrate several patterns for reliable tool integration. The most important principle is defensive programming - assume everything will fail and design accordingly.

class QueryExecutionTools:
    def execute_query(self, query_data: Dict[str, Any]) -> Dict[str, Any]:
        try:
            # Extract and validate query components - always check inputs first
            if "generated_query" not in query_data:
                return {"error": "No generated query found in query data"}

            generated_query = query_data["generated_query"]
            query_dsl = generated_query.get("query_dsl")

            # Security validation - ensure read-only operations
            # This prevents agents from accidentally modifying data
            if not self._is_read_only_query(query_dsl):
                return {"error": "Only read-only queries are allowed"}

            # Resource validation - ensure index exists before querying
            if not self.es.indices.exists(index=target_index):
                return {"error": f"Index '{target_index}' does not exist"}

            # Execute query with structured error handling
            response = self.es.search(index=target_index, body=query_dsl)

            # Return clean, structured results for LLM analysis
            return {
                "total_hits": response["hits"]["total"]["value"],
                "documents": [
                    {
                        "id": hit.get("_id"),
                        "score": hit["_score"],
                        "source": hit["_source"],
                    }
                    for hit in response["hits"]["hits"]
                ],
                "took_ms": response["took"]
            }

        except Exception as e:
            logger.error(f"Error executing query: {str(e)}")
            return {"error": f"Failed to execute query: {str(e)}"}

The pattern here is safety first, then functionality. Every tool method validates its inputs, checks that required resources exist, ensures operations are safe to execute, and returns errors in a consistent format that agents can understand and handle appropriately.

Connection management is another critical concern. Tools that interact with external systems need robust connection handling to prevent resource leaks and handle network failures gracefully. My system uses a singleton pattern for ElasticSearch connections, ensuring all tools share a single connection pool while maintaining thread safety across the agent system.

The security validation deserves particular attention. The _is_read_only_query method examines the query DSL to ensure it doesn't contain any write operations. This prevents the system from accidentally modifying data, which is essential when giving AI agents access to production databases.

State Management Between Agent Interactions

Unlike stateless HTTP APIs, agentic systems need to maintain context across multiple agent interactions. The Index Selection Agent needs to communicate its findings to the Query Generation Agent, which in turn needs to pass both the index information and the generated query to the Query Execution Agent.

My system handles this through ADK's session management capabilities, with utility functions that make state sharing explicit and reliable:

def save_index_selection_data(
    context: ToolContext,
    selected_index: str,
    index_schema: Dict[str, Any],
    reasoning: str,
    confidence: str = "high"
) -> str:
    """Save index selection data to session state for the next pipeline agent."""

    selection_data = {
        "selected_index": selected_index,
        "index_schema": index_schema,
        "selection_metadata": {
            # Store the reasoning so later agents understand the decision
            "reasoning": reasoning,
            "confidence": confidence
        },
        "validation": {
            # Include validation status for error handling
            "index_exists": True,
            "schema_retrieved": True,
            "ready_for_query_generation": True
        }
    }

    # Save to session state for next agent to access
    context.state["index_selection_data"] = selection_data
    context.state["selected_index"] = selected_index

    return f"Index selection data saved. Selected: {selected_index}"

This approach makes state management explicit rather than relying on implicit context passing. Each agent can access previous results through well-defined state keys, and the validation information helps agents understand whether they have reliable input data to work with.

State management becomes especially important for error recovery. If the Query Generation Agent fails to produce a valid query, the system still has the index selection results and can potentially retry with a different approach or ask the user for clarification, rather than starting the entire pipeline from scratch.

Observability: Making Agent Decisions Transparent

Production agentic systems require comprehensive observability. When an agent makes a poor decision or produces unexpected results, you need to understand its reasoning process to improve the system. This goes beyond traditional application logging because agent decisions involve complex reasoning that isn't visible from external behavior alone.

My system integrates with Phoenix for distributed tracing, treating each agent interaction as a span in a larger trace that shows how user queries flow through the agent pipeline:

async def process_user_query(runner, session_service, query: str, user_id: str, 
                           app_name: str, logger: logging.Logger, tracer=None):
    """Process user query with comprehensive tracing."""

    if tracer:
        with tracer.start_as_current_span("process_user_query") as span:
            # Record key attributes for debugging later
            span.set_attribute("user.id", user_id)
            span.set_attribute("user.query", query)
            span.set_attribute("app.name", app_name)

            # Process through agent pipeline with full trace visibility
            await _process_query_internal(
                runner, session_service, query, user_id, app_name, logger, span
            )

This provides me with a complete view of how user queries flow through the agent pipeline, how long each stage takes, and where failures occur. More importantly, it captures the reasoning and intermediate results from each agent, making it possible to understand why the system made particular decisions.

I encountered a practical challenge with OpenTelemetry tracing in async environments. The async generators and streaming patterns common in agent frameworks can cause context detachment errors that crash the application. I solved this with safe tracing utilities that handle these issues gracefully:

@contextmanager
def safe_tracing_context():
    """Context manager that safely handles OpenTelemetry tracing errors."""
    try:
        yield
    except Exception as e:
        error_msg = str(e).lower()
        # Check if this is a context-related tracing error
        if any(keyword in error_msg for keyword in ["context", "token", "detach"]):
            # Silently ignore context-related errors to prevent crashes
            logger.debug(f"Suppressed OpenTelemetry context error: {e}")
        else:
            # Re-raise non-context related errors since they're real problems
            raise

This ensures that tracing problems don't crash the agent system while still providing observability when possible. The lesson here is that production systems need to be resilient to their monitoring infrastructure failing.

Error Handling: When Agents Make Poor Decisions

Reliable agentic systems need sophisticated error handling strategies that account for the probabilistic nature of LLM outputs. Not all errors are equal - some should cause the system to stop completely, while others should trigger graceful degradation or alternative approaches.

My system implements error handling at multiple layers, each with different recovery strategies. At the tool level, individual tools return structured error information rather than throwing exceptions. This allows agents to understand what went wrong and potentially try alternative approaches.

At the agent level, agents interpret tool errors and decide whether to retry, use alternative approaches, or escalate the error. The structured output format includes success indicators and error messages that make this decision-making explicit:

class QueryExecutionOutput(BaseModel):
    """Structured output for the Query Execution Agent."""

    execution_results: Optional[ExecutionResults] = Field(
        description="Raw query execution results, or null if execution failed"
    )
    success: bool = Field(
        description="Whether the query execution was successful"
    )
    error_message: Optional[str] = Field(
        description="Error message if execution failed"
    )
    natural_language_response: str = Field(
        description="Natural language response based on analyzing the results"
    )

At the pipeline level, the orchestrator can route queries to alternative handlers if the primary ElasticSearch pipeline fails. For example, if a query can't be mapped to any available index, the system can fall back to answering general questions about ElasticSearch concepts.

The key insight is that different types of errors require different recovery strategies. A syntax error in query generation might be recoverable by retrying with additional context, while a security violation should immediately terminate the request. The structured error handling approach makes these distinctions explicit and actionable.

Configuration and Deployment Patterns

Building reliable agentic systems isn't just about the code - it's also about how they're configured, deployed, and maintained in production environments. My system demonstrates several patterns that have proven essential for operational reliability.

Different agents benefit from different language models based on their task complexity and cost requirements. My configuration approach allows easy experimentation without code changes:

# config.yaml - Flexible model configuration
agents:
  orchestrator: "openai/gpt-4o-mini"
  elasticsearch: "openai/gpt-4o-mini"  
  index_selection: "openai/gpt-3.5-turbo"  # Simpler task, cheaper model
  query_generation: "openai/gpt-4o-mini"   # Complex task, better model
  query_execution: "openai/gpt-4o-mini"

This configuration-driven approach allows me to optimize for cost and performance independently for each agent, and makes it easy to experiment with new models as they become available without touching the core system logic.

The deployment strategy acknowledges that agentic systems typically depend on multiple services. My Docker Compose setup orchestrates ElasticSearch for data storage, Kibana for data visualization, Phoenix for observability, and the agent system itself:

# docker-compose.yml - Complete service orchestration
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    # Configuration for data storage

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0  
    # Configuration for data visualization

  phoenix:
    image: arizephoenix/phoenix:latest
    # Configuration for observability

  llm-es-agent:
    build: .
    # Configuration for the agent system
    environment:
      - ES_HOST=http://elasticsearch:9200
      - PHOENIX_ENDPOINT=http://phoenix:6006

This approach ensures that the entire system can be deployed consistently across development, staging, and production environments. The environment variables handle deployment-specific configuration while keeping the core system logic environment-agnostic.

User Experience: Multiple Interfaces for Different Needs

Production agentic systems often need to serve different types of users through different interfaces. My system demonstrates this with both terminal and web interfaces sharing the same underlying agent logic. This architectural pattern has proven valuable in real deployments.

The core agent processing is abstracted into a reusable class that both interfaces can use:

class UnifiedAgentApp:
    """Unified application class supporting multiple interfaces."""

    def __init__(self):
        self.orchestrator = None
        self.runner = None
        self.session_service = None

    async def process_query(self, query: str, user_id: str) -> Dict[str, Any]:
        """Process user query through the orchestrator agent."""
        # Shared logic for both interfaces - the heavy lifting happens here

    def run_terminal_interface(self):
        """Run the terminal interface for developers."""
        # Terminal-specific UI logic - just handles input/output formatting

    def run_streamlit_interface(self): 
        """Run the web interface for business users."""
        # Web-specific UI logic - handles web-specific presentation

This separation allows the system to serve different user personas - developers who prefer command-line interfaces for debugging and scripting, and business users who prefer web interfaces for ad-hoc queries - without duplicating the complex agent orchestration logic.

The lesson here is that the interface is often less important than the underlying system reliability. Users will adapt to different interfaces as long as the core functionality works consistently and provides useful feedback when things go wrong.

What Actually Works in Production

After building and operating this system, several patterns have proven essential for production reliability that aren't obvious from reading about agentic systems in theory.

Structured everything. Use structured outputs for all agent communications. The time I spent defining Pydantic models upfront saved enormous debugging time later when agents produce unexpected outputs. Structured inputs and outputs make the system testable and debuggable in ways that free-form text communication simply doesn't allow.

Validate at every boundary. Each agent should validate its inputs, perform its work safely, and validate its outputs before passing them to the next agent. This catches problems early rather than letting them cascade through multiple agents before failing in confusing ways.

Design for partial failures. In traditional applications, you often design for success and handle failures as exceptions. In agentic systems, partial failures are common and often recoverable. Design your error handling to distinguish between recoverable issues and hard failures, and make recovery paths explicit.

Make decisions observable. The complexity of agentic systems means you need comprehensive observability to understand system behavior. But don't let observability infrastructure crash your system - implement defensive observability that degrades gracefully when monitoring systems fail.

Security by default. When giving agents access to external systems, implement security controls at the tool level rather than relying on prompt engineering or agent instructions. Agents will eventually try to do things they shouldn't, either through user requests or unexpected reasoning patterns.

Configuration over code. Agent behavior often needs to be tuned based on operational experience. Make key parameters configurable so you can adjust system behavior without code deployments. This is especially important for model selection and cost optimization.

My One Advice

If you're building agentic systems, focus on making each component reliable independently before optimizing agent interactions. The systems that work in production are built on solid engineering fundamentals, not prompt engineering magic.

WTF even is MCP?

I'll admit it - until I started building one myself, I didn't fully understand what MCP was trying to solve. Talk about living under the rock. Primarily because it seemed like such a large mountain to climb, and honestly, the documentation made my brain hurt.

Here's what finally made it click for me: MCP (Model Context Protocol) is basically Anthropic's way of saying "hey, let's standardize how AI assistants talk to your stuff." Your databases, your APIs, your tools - everything.

Picture this: You're one of those people who has very specific requirements for everything. Your morning routine involves exactly 2.5 tablespoons of this specific coffee grind, steamed oat milk at precisely the right temperature, stirred counterclockwise three times (don't judge). You decide to hire an assistant to handle this.

Your assistant is brilliant but has no clue how to use any of your gadgets. The fancy espresso machine, the smart home system, the automated sock-sorting device (yes, that's a thing in my hypothetical world). Without MCP, you'd have to teach each AI assistant how to use each tool individually - different interfaces, different authentication, different everything.

With MCP, you create one "universal manual" that any AI assistant can read. Suddenly, Claude, ChatGPT, or whatever AI assistant comes next can all use your tools through the same standardized interface.

That's when the lightbulb went off: this wasn't just another tech buzzword - it was solving a real integration nightmare that's only getting worse as more AI assistants show up to the party.

Why I decided to reinvent the wheel (spoiler: Google's version has issues)

Now, before you start typing "just use Google's MCP Toolbox, you fool" in the comments, let me explain why I went down this rabbit hole despite a perfectly functional solution already existing.

Google's BigQuery MCP connector works... but it's like that friend who's helpful but comes with a lot of baggage.

First, Google themselves treat it like a science experiment. They literally plaster "BETA" warnings everywhere with the classic "expect breaking changes in future versions" disclaimer. Nothing says "production-ready" like a vendor telling you they might completely change everything tomorrow. I've been burned by Google deprecating products before (cough Google Podcasts cough).

Second, it's about as granular as a sledgehammer. Want to give your AI access to sales data but not employee records? Tough luck. The AI gets access to ALL tables the service account can see. There's no way to say "hey, only show the marketing tables to this user, and only the finance tables to that user."

Third, the data leakage potential gave me anxiety. The only workaround is spinning up multiple MCP instances with different service accounts. So instead of one clean connector, you're managing what I like to call "a zoo of MCP servers." Elegant? About as elegant as duct tape.

But here's what really pushed me over the edge: When you have lots of tables (and BigQuery projects are basically table hoarders), the AI consistently picks the wrong ones. And this isn't Google's fault - it's a fundamental quirk of how LLMs work.

The more options you give them, the worse they get at picking the right one. It's like having a really smart friend who always grabs the first tool they see in your toolbox, even when they need a screwdriver and they picked up a hammer. Your "customer_analytics_2024" table gets ignored while "test_backup_data_2019" gets selected just because it appeared first in the list.

No amount of clever sorting fixes this - alphabetical, by importance, by creation date, even random order. The model will still lean toward the first few options like a kid reaching for candy.

That's when I thought, "You know what? I bet I can try to solve this differently”.

What I built instead

A quick note: I won't be sharing code for this work as it belongs to my employer. I can share the high-level concepts and architecture, but the specifics are proprietary.

Instead of throwing every table at the AI and hoping it picks the right one, I broke down the BigQuery interaction into five distinct tools with a focus on intelligent filtering and user-specific access control.

Tool 1: IAM-Based Content Visibility Controller This tool integrates with our organization's Identity and Access Management system to dynamically determine what datasets and tables a user can access. Rather than relying on service account permissions (which are static), it pulls the authenticated user's identity and cross-references it with our data governance policies in real-time. The tool queries our IAM provider's API to get group memberships and role assignments, then filters the available BigQuery resources accordingly. A user in the marketing group sees marketing datasets, finance users see finance data, and cross-functional analysts get broader access based on their role matrix.

Tool 2: Semantic Dataset Discovery This tool addresses the core table selection problem through semantic search over dataset metadata. It maintains an indexed representation of all dataset names, descriptions, and table schemas, then uses vector similarity search to match user queries with relevant datasets. Instead of presenting 200+ datasets alphabetically, it ranks them by semantic relevance to the user's intent. The key insight: this only works because our organization has invested in proper data cataloging with meaningful dataset names and comprehensive metadata.

Tool 3: Table Metadata Fetcher Once relevant datasets are identified, this tool retrieves table-level metadata including creation dates, last modified timestamps, row counts, and table descriptions. It essentially wraps BigQuery's INFORMATION_SCHEMA queries but filters results based on the datasets identified in Tool 2, preventing information overload.

Tool 4: Schema and Documentation Inspector This tool fetches detailed schema information including column names, data types, constraints, and most importantly, column descriptions and business definitions when available. It pulls from both BigQuery's native schema metadata and our organization's data documentation system to provide context about what each field actually represents.

Tool 5: Query Executor The final tool executes the SQL query with the full context from previous tools. It includes safety mechanisms like query cost estimation and result size limits based on the user's permissions.

What Makes This Different From Google's MCP Toolbox

Yes, Google's BigQuery MCP connector also uses multiple tools for dataset discovery, schema inspection, and query execution. But there are crucial architectural differences:

1. User-Centric vs. Service Account-Centric Access Control Google's connector operates under a single service account's permissions - what the service account can see, everyone can see. My implementation integrates with organizational IAM at request time, providing user-specific data access without spinning up multiple MCP instances. This isn't just a feature difference; it's a fundamental security model change.

2. Semantic vs. Alphabetical Discovery Google's tool discovery presents datasets in essentially alphabetical or chronological order. The AI has to pick from a massive list, leading to the position bias problem I mentioned earlier. My semantic search pre-filters and ranks datasets by relevance, dramatically reducing the cognitive load on the AI model.

3. Context-Aware Tool Chaining While both approaches use multiple tools, Google's connector treats each tool call independently. My implementation maintains context across the tool chain - Tool 3 only fetches metadata for datasets identified by Tool 2, Tool 4 only inspects schemas for tables the user can actually access, etc. This reduces the total information space the AI has to process.

4. Custom Data Governance Integration Google's connector only knows about BigQuery's native metadata. My implementation pulls from our organization's broader data governance infrastructure - business glossaries, data lineage systems, and custom documentation platforms - providing richer context for query generation.

The Reality Check

Here's the honest truth that took me way too long to realize - building this taught me that MCP servers really are just fancy API orchestration wearing a protocol costume. Remember my title about expecting magic and getting glorified API endpoints? Yeah, that's exactly what happened.

The "magic" I was looking for doesn't exist in the MCP protocol itself. It's just a standardized way to wrap your APIs so AI assistants can call them. The real magic (if you can call it that) is in how thoughtfully you design those API interactions and what external systems you decide to integrate with.

All that complexity I was trying to avoid with Google's solution? Well, I didn't eliminate it, I just moved it around and painted it a different color. Instead of dealing with their tool selection issues, I built my own tool selection logic. Instead of managing multiple service accounts, I built complex IAM integration. The work didn't disappear; it just got redistributed to different parts of the system.

But here's the thing - that redistribution actually mattered for our specific use case. We needed user-specific access control and intelligent dataset discovery more than we needed simplicity. Your organization might have completely different priorities, and Google's sledgehammer approach might be exactly what you need.

So did I reinvent the wheel? Absolutely. Was it worth it? That depends on whether you think a wheel with better tires and custom rims is worth the engineering effort, or if you just need something that rolls.

Google Trends suggests the same and most of the references in academic journals coming from the last 1 year. And no, this is not perplexity.ai. It is a metric being used in Generative AI.

How Generative Models Work

Generative models, such as those used in language modeling, operate by predicting the next token in a sequence based on the context provided by preceding tokens. This prediction is grounded in probability distributions learned during training on vast corpora of text. The process is iterative, with each token generated influencing the probabilities for the next.

At a high level, the model calculates the likelihood of every possible token at a given step. For example, if the context is “The cat is on the,” the model might assign the following probabilities to the next word:

• “mat”: 0.7

• “chair”: 0.2

• “sofa”: 0.1

The token with the highest probability (“mat” in this case) is typically selected, although alternative strategies like sampling or beam search can be used to introduce diversity or explore multiple sequences.

The foundation of this token-generation mechanism is the softmax function, which ensures that the probabilities of all tokens sum to 1. This normalization allows the model to make probabilistic predictions and enables perplexity calculations by assessing how closely the predicted distribution aligns with the true sequence.

Probability and Context

The power of generative models lies in their ability to use context effectively. Context is established by analyzing the preceding tokens and constructing a vector representation of their meanings. This representation feeds into the model’s architecture—typically a transformer—to predict the next token.

For instance, in the sentence “The weather today is sunny and,” the model might prioritize weather-related tokens like “warm” or “hot” over unrelated ones. This contextual sensitivity enables generative models to produce coherent and contextually appropriate outputs, making them integral to applications such as chatbots, translation systems, and creative content generation.

Perplexity

Generative models are not a typical model on which you can calculate accuracy and call it a day. The only thing to work with is the probability of each token. Perplexity leverages this set of information to determine a numerical value of each generated text from a model.

At its core, perplexity measures how well a generative model predicts a given sequence. It is calculated using the probabilities assigned by the model to the tokens in the sequence. Lower perplexity indicates that the model has assigned higher probabilities to the correct tokens, which generally correlates with better performance.

The formula for perplexity can be expressed as:

$$\text{Perplexity} = 2^{-\frac{1}{N} \sum_{i=1}^{N} \log_2 P(w_i)}$$

where P(wi) is the probability of the i-th word under the model.

Why Perplexity Matters

1. Model Comparison: Perplexity serves as a benchmark for evaluating and comparing different generative models. Lower perplexity indicates a model’s stronger ability to predict sequences accurately, offering insights into its overall performance.

2. Training Feedback: During the training phase, perplexity acts as a critical feedback mechanism. A steadily decreasing perplexity score suggests that the model is learning effectively. Conversely, stagnation or an increase in perplexity might signal issues such as overfitting or insufficient learning.

3. Data Quality Assessment: High perplexity on specific datasets may indicate that the data contains ambiguities or inconsistencies, prompting a closer examination of the dataset quality.

4. Application Suitability: By measuring perplexity on task-specific data, developers can determine whether a generative model is well-suited for a particular application, such as summarization or dialogue generation.

Limitations of Perplexity

While perplexity is a valuable metric, it is not without limitations:

• Tokenization Dependence: Perplexity scores can vary based on the tokenization scheme used. Different tokenization methods (e.g., word-level vs. subword-level) produce varying perplexity values, complicating direct comparisons.

• Human Readability Disconnect: Low perplexity does not always guarantee outputs that are coherent or contextually appropriate to humans. Complementary evaluations, such as human judgment or task-specific metrics, are often necessary.

• Cross-Lingual Challenges: Perplexity may behave differently across languages with varying syntax and morphology, requiring tailored interpretations for multilingual models.

Is Perplexity Just Confidence?

At a conceptual level, perplexity can be thought of as reflecting the model's confidence in its predictions. However, this confidence is not necessarily tied to correctness. A model can be "confidently wrong," assigning high probabilities to tokens that do not align with human expectations or task requirements. This duality—confidence versus correctness—is one of the reasons perplexity alone is an imperfect metric.

Consider an example where a model predicts the next word in the sentence: “The cat is on the.” If it assigns a high probability to “ceiling” instead of the more contextually appropriate “mat,” the perplexity might appear low, but the prediction is clearly unsuitable.

This disconnect highlights that perplexity measures the internal consistency of the model's probabilistic predictions rather than their human-like coherence or utility.

Living with Perplexity

Despite its limitations, perplexity remains the go-to metric for generative models. Its ease of calculation and alignment with the probabilistic nature of these models make it a convenient choice. However, developers and researchers are increasingly aware of its shortcomings. To address these, perplexity is often supplemented with human evaluations, task-specific metrics like BLEU and ROUGE, and even adversarial testing.

The future of generative AI evaluation likely involves a combination of metrics, balancing quantitative measures like perplexity with qualitative assessments that better capture human expectations. While perplexity may not tell the full story, it provides a crucial foundation for understanding and improving generative models.

Perplexity and Data Quality

A generative model’s performance is inherently tied to the quality of the data it is trained on. High-quality input data often leads to better predictions and, consequently, lower perplexity scores. This connection suggests that when the data is robust—well-curated, comprehensive, and representative of the task at hand—perplexity can serve as a reliable indicator of model performance.

However, it is essential to consider a few nuances:

1. Reflection of Learned Patterns: Perplexity is most meaningful when the model has been trained on data that aligns well with the evaluation dataset. If the training data accurately represents the patterns in the test sequences, low perplexity indicates effective learning.

2. Sensitivity to Outliers: Even with high-quality input, generative models can struggle with edge cases or outliers, leading to higher perplexity scores on those specific examples. While this reflects the model's difficulty, it does not necessarily undermine perplexity as a metric for the majority of cases.

3. Scope of Utility: When data quality is high, perplexity becomes a more direct measure of how well the model captures the structure and probabilities of the language. In this scenario, perplexity may indeed be viewed as a "good metric," particularly for tasks focused on language modeling.

TLDR

Perplexity, a key metric in Generative AI, measures how well a model predicts a sequence (e.g., text) by assessing the alignment between its predicted probabilities and actual outcomes. While useful for model comparison, training feedback, and data quality assessment, perplexity has limitations: it doesn't guarantee human-like coherence, varies with tokenization schemes, and behaves differently across languages. To provide a comprehensive evaluation, perplexity is best used in conjunction with other metrics (e.g., BLEU, ROUGE, human evaluations), especially when working with high-quality, representative training data.

The issue that caused me to dig a rabbit hole

Few weeks ago, I got curious on how the package is doing. Did anyone other than me even downloaded the package or it is just a piece of junk in the discarded pile of hundreds of thousands of PyPi packages. And I copied a simple python script from StackOverflow to get total download stats and got this

Well, to be honest, this is a few too many. When I ran this script, I was expecting to see a number like 10 or 15. Never in my life had I thought it would be more than 3,400!

As you’d think, I was elated seeing the number like this. I didn’t give it much thought until after a few days. TezzCrawler works fine and gets the work done, but it is neither the only package that solves this problem, is not the fastest crawler, and most importantly, I had not even mentioned it to anyone that I have published it, nobody other than me knew it even existed in the past month. So how did a tool with 0 marketing get these many downloads? I’d sure want to know in order to replicate the results in other projects.

And then started a deep dive which eventually left me about $10 poorer.

PyPi Stats

There are a number of analytic tools where you can go and check your package’s statistics. Officially, there are 4 that are promoted but only 2 of them provided download numbers as part of their statistics (and both gave different numbers and none matched with the API provided numbers 😭)

ClickHouse, an open source data warehousing tool, maintains a dashboard called ClickPy for analysing any Python package’s statistics. Searching on the platform gave me a cumulative download count of 2,400. A staggering 1000 difference than what the API counts are!

Oh cool, the package is highly popular in the US, Canada, China, and Russia.

Anyways, this sparked another debate in my head. Why are the numbers so much different?

To get to the bottom of it once and for all, I went to PypiStats in the hopes that I’ll get all my answers there (since they are the ones that provided the Python API I used at the first place). But I left with more questions I started with.

PyPi stats doesn’t provide a cumulative download but only a daily, weekly, and monthly count. But look closely, The last month count as per PyPi Stats is 861 but ClickPy dashboard screenshot from before shows a total download count of 2,100. (WTF is that difference?)

Also, this brought in a new terminology as well - Mirrors. I knew what mirrors are, just didn’t expect PyPi to be using it (either I was stupid or I am stupid to not realize this earlier 🥲)

Here’s a key detail - anybody can boot up their own PyPi server. This server is essentially what is called a Mirror. Depending on how many of the packages from the main PyPi server are reflected onto the Mirror, there are essentially 3 segregations of mirrors - Private, Partial, or Public. Public Mirrors are exact 1-1 replica of PyPi while Private Mirrors are hosted by companies internally for the packages they allow their teams to use for development.

There’s a tool called Bandersnatch that allows you to replicate any Python package to a Mirror. (This information will come in handy in a minute)

So, the download count mentioned on PyPiStats is without Mirrors. 861 downloads from without Mirrors and 2100 downloads with Mirrors. So that means, ~1300 mirrors duplicated the package to the Mirror during the last month and 861 actual downloads happened. I got my actual download count what I was after and I should call it a day now.

861 downloads is still a lot more than 10-15 downloads I was expecting! Did I miss something during the analysis?

The analysis where I lost my money

Okay, till now, I’ve got 3 things - anybody can create a Mirror and a sync between Mirror and main PyPi server can trigger a “download” count in PyPi stats; if someone downloads from a Mirror, it doesn’t reflect in PyPi stats; and ClickPy is a waste of a dashboard.

But the main quest “How many downloads did actually happen for my package?” was still unfulfilled. And at this stage, I found the holy grail - PyPi migrates all of its logs to a BigQuery dataset.

The BigQuery dataset of PyPi consists of 3 tables - package metadata, download events, and download request metadata. Running a really simple query on BigQuery like the below one was a little out of my budget (It was going to process 15TB of data to execute) and I didn’t want to burn my entire budget on 1 query.

SELECT count(*)
FROM bigquery-public-data.pypi.file_downloads
WHERE file.project = 'tezzcrawler'

So I had to get a little creative.

SELECT COUNT(*) as download_frequency,
  DATE_TRUNC(DATE(timestamp), MONTH) AS month
FROM bigquery-public-data.pypi.file_downloads
WHERE project = 'tezzcrawler'
GROUP BY month
ORDER BY month DESC

Running the query, I get… 2,408 downloads. Exactly what ClickPy tells. But wait, I can segregate on how the package was downloaded as well.

SELECT 
  DISTINCT(details.installer.name) as installer,
  COUNT(*) as download_frequency
FROM bigquery-public-data.pypi.file_downloads
WHERE project = 'tezzcrawler'
GROUP BY installer
ORDER BY download_frequency DESC

To save on costs while writing the article, I added a time period in all the queries to get the screenshot of the results to explain the findings. This above query, was executed on a 30 day period.

Here’s what I found on executing the above query.

This is more on the lines of what I was actually expecting. The actual downloads are 39, that happened via pip install command. The rest are some other activities with Bandersnatch being a Mirror. My guess is some bots monitoring PyPi packages directly on the main server instead of first creating a Mirror. Unfortunately, the user agent is not tracked in the download request, only the TLS protocol is and that doesn’t really tell much about the remaining downloads.

So, there we have it, 39 downloads in the last month for TezzCrawler. The actual count, the honest number, and not what these dashboards were feeding my ego earlier. Now I can have some peace, but more importantly, I've learned a valuable lesson in the importance of data accuracy and the nuances of open-source package distribution.

For fellow developers and open-source enthusiasts, take away this: don't be misled by vanity metrics. Dig deeper, question the numbers, and understand the ecosystem your project operates within. It might not always be as glamorous as thousands of downloads, but the true measure of your project's impact lies in its genuine usage and the value it brings to its users. With this newfound understanding, I'm excited to focus on what really matters – improving TezzCrawler for those 39 users, and hopefully, many more to come.

Every Thursday, 2:30PM IST, I’ll share 1 article directly to your mailbox. The next few articles are things I learnt while developing my own RAG Framework, which I would have definitely missed had I stuck to using pre-developed frameworks and probably spent days debugging. If this resonates with you, signup for the free newsletter.

Train-Test Split: The Basics

The train-test split is a fundamental step in machine learning workflows. It divides your data into two parts:

• Training Set: This portion is used to train the model. The model learns patterns and relationships from this data.

• Test Set: This part is reserved for evaluating the model’s performance. It simulates how the model will behave on unseen data.

Python’s scikit-learn library provides a simple way to perform this split:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

In this example, 30% of the data is set aside for testing (test_size=0.3). Setting a random_state ensures reproducibility. The train-test split is critical for assessing whether your model is overfitting or generalizing well.

Time Series Data: Handle with Care

Time series data requires a different approach because the order of observations carries meaningful information. Randomly shuffling the data can break the temporal patterns, leading to unreliable evaluations. For instance, testing a model on past data after training it on future data doesn’t reflect real-world scenarios.

When working with time series data, it’s essential to maintain the chronological order. Train the model on historical data and evaluate it on future data. This ensures that the model’s predictions are based on past trends.

In scikit-learn, the TimeSeriesSplit class facilitates this type of split:

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)
for train_index, test_index in tscv.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

This method allows for multiple train-test splits while preserving the temporal order, providing a robust way to evaluate time series models.

K-Fold Cross-Validation: A Comprehensive Evaluation

K-Fold Cross-Validation is an effective technique for evaluating model performance. It works by dividing the dataset into ‘k’ subsets (folds). The model is trained on ‘k-1’ folds and tested on the remaining fold. This process repeats ‘k’ times, with each fold serving as the test set once. The results are then averaged to provide an overall performance metric.

Here’s how to implement K-Fold Cross-Validation in Python:

from sklearn.model_selection import KFold

kf = KFold(n_splits=5, shuffle=True, random_state=42)
for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

K-Fold Cross-Validation reduces the risk of overfitting and ensures that the model is evaluated across multiple data partitions. This method is particularly useful for smaller datasets where reserving a large test set isn’t feasible.

Stratified K-Fold: Fair Evaluation for Imbalanced Data

When dealing with imbalanced datasets—where some classes are underrepresented—it’s important to ensure that the train and test sets have the same class distribution as the entire dataset. Stratified K-Fold Cross-Validation addresses this issue by maintaining class proportions in each fold.

Here’s how to implement it:

from sklearn.model_selection import StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

Stratified K-Fold ensures fair representation of all classes in both training and testing phases. This leads to more reliable and unbiased performance metrics, especially in classification tasks.

Domain-Specific Splits: Tailored Testing

For certain types of data, domain-specific considerations are crucial when preparing the test set. For instance:

• Spatial Data: For geospatial datasets, it’s often better to split data based on regions. For example, you might train a model on data from one geographic area and test it on another to evaluate how well the model generalizes across locations.

• Demographic Data: For human-centered applications, splitting data by demographic groups can highlight biases or ensure fairness. For example, testing a healthcare model separately on age groups can reveal disparities in predictions.

• Event-Specific Data: In event-driven datasets (e.g., sports or financial markets), splitting by events can ensure that the model is evaluated on entirely different scenarios.

These approaches ensure the test set reflects real-world variability and allows for a more robust evaluation.

Data Augmentation for Robust Evaluation

Data augmentation is a technique commonly used in training, but it’s also valuable for testing. By applying transformations to the test data, you can assess the robustness of your model under different scenarios. For instance:

• Computer Vision: Apply augmentations such as rotations, translations, or noise to test images and evaluate if the model’s predictions remain consistent.

• Natural Language Processing: Add variations like synonym replacement, misspellings, or paraphrasing to test the resilience of language models.

• Audio Processing: Add background noise or change the pitch in audio test data to see how well the model adapts to real-world distortions.

Data augmentation during testing is particularly useful when you want to simulate challenging conditions or validate robustness beyond ideal scenarios.

Why Proper Dataset Preparation Matters

Proper dataset preparation directly impacts the reliability and generalizability of your machine learning models. Here are a few reasons why it’s critical:

1. Avoid Overfitting: Using separate datasets for training and testing prevents the model from memorizing the data rather than learning patterns.

2. Realistic Evaluation: Testing on unseen data mimics real-world scenarios, providing a realistic measure of the model’s performance.

3. Fair Metrics: Techniques like Stratified K-Fold ensure that performance metrics are not biased due to class imbalances.

4. Respect Temporal Patterns: For time series data, maintaining the order of observations leads to more meaningful evaluations.

5. Domain Relevance: Tailored splits and augmented testing ensure the model’s performance is validated in realistic and varied conditions.

Conclusion

Dataset preparation is not just a preliminary step—it’s a cornerstone of machine learning. By applying techniques like train-test splits, time series-specific methods, K-Fold Cross-Validation, Stratified K-Fold, domain-specific splits, and data augmentation, you can build models that are both accurate and generalizable. Investing time in preparing your data ensures that your models are reliable and ready to tackle real-world challenges with confidence.

Metrics for Regression Models: The Basics

Unlike classification models, regression models deal with predicting continuous values. This means we’re less concerned with thresholds and more focused on how far off the predictions are from the actual values.

Ideally, if we just want to see how far off is the prediction, we can just make a difference of actual and predicted values and call it a day. But with Machine Learning, there are no ideal scenarios. Extrapolating this idea to thousands and millions of data points, and you will sometimes get an average error rate of 0. But does that mean your model is performing way too well? Nope. It could be that the model is wrong on both the directions of the actual value equally in terms of cumulative magnitude in each direction.

For instance, imagine a model predicting daily temperatures where the actual temperature is 30°C. If the model predicts 40°C on one day and 20°C on another, the average error might come out to 0, but it’s clearly far from accurate! This highlights the importance of using more robust metrics to assess the performance of regression models.

Keeping this in mind, let's focus how we can carefully craft the metrics that work well for regression models.

Mean Absolute Error

The Mean Absolute Error (MAE) is one of the simplest yet highly interpretable metrics for evaluating regression models. MAE calculates the average of the absolute differences between the actual and predicted values, making it easy to understand and less sensitive to outliers compared to some other metrics.

Formula

The formula for MAE is as follows:

$$\text{MAE} = \frac{1}{n} \sum_{i=1}^n \lvert y_i - \hat{y}_i \rvert$$

where:

• n is the total number of observations.

• y​ represents the true value.

• y^​ represents the predicted value.

In simple terms, it is basically finding the difference, removing the sign (discarding the direction of error), sums all the errors, and averages the final value to get the mean error of the model.

Mean Squared Error

MAE brings an important approach towards calculation of the error - discarding the direction of the error. MSE does exactly the same thing, with a slight change. Instead of calculating the mean, it calculates the square of the difference. This way, we again remove the direction and only emphasis is on the magnitude of the error.

Formula

The formula of MSE is

$$\text{MSE} = \frac{1}{n} \sum_{i=1}^n \left( y_i - \hat{y}_i \right)^2$$

Why MSE when MAE exists?

1. The squaring of the error makes the algorithm sensitive to the magnitude of the error. It amplifies large errors such that even a small portion of large errors will have profound impact on the MSE value.

2. Due to sensitive nature of MSE, it is sensitive to outliers which isn’t the case with MAE.

3. Mean Absolute Error minimizes when the predictions favor the median value of the testing set while MSE minimizes when the predictions are closer to the average of the testing set.

R-Squared (R²) Metric

Now, let’s talk about a metric that is widely recognized and frequently used in the evaluation of regression models: R-squared (R²). While Mean Absolute Error (MAE) and Mean Squared Error (MSE) are good at measuring the magnitude of errors, R-squared takes a different approach. It gives us a measure of how well the regression model fits the data, helping us understand how much of the variance in the target variable is explained by the model.

What Does R-Squared Measure?

R², also called the coefficient of determination, is essentially a percentage that tells us how well the regression model explains the variability of the target variable. A high R² indicates that the model captures most of the variance, whereas a low R² suggests that the model is missing the mark and not explaining much of the variability in the data.

To understand this better, let’s break it down. The idea behind R² is based on comparing two things:

1. The total sum of squares (TSS): This measures how much the actual values vary from the mean of the target variable.

2. The residual sum of squares (RSS): This measures how much the predicted values deviate from the actual values.

R² is then calculated using the formula:

$$R^2 = 1 - \frac{RSS}{TSS}$$

Where:

• RSS (Residual Sum of Squares) is the sum of the squared differences between the observed values and the predicted values. It is mathematically written as

$$RSS = \sum_{i=1}^n (y_i - \hat{y_i})^2$$

• TSS (Total Sum of Squares) is the sum of the squared differences between the observed values and the mean of the observed values. It is mathematically written as

$$TSS = \sum_{i=1}^n (y_i - \bar{y_i})^2$$

Interpretation of R-Squared

• R² = 1: The model perfectly fits the data. All data points fall exactly on the regression line.

• R² = 0: The model doesn’t explain any of the variance in the data. Essentially, it’s no better than predicting the mean of the target variable for all instances.

• R² < 0: This happens when the model performs worse than a simple horizontal line at the mean of the target variable. This indicates that the model is a poor fit, and in some cases, even overfitting or underfitting the data.

The Pros and Cons of R-Squared

Pros:

1. Easy to Interpret: R² is easy to explain to stakeholders because it represents a percentage of the variance explained by the model.

2. Model Comparison: It helps in comparing different models for the same dataset. A higher R² value typically indicates a better model fit.

Cons:

1. Not Robust to Overfitting: A higher R² doesn’t always mean a better model. It can sometimes mislead when dealing with overfitting, as a model might perfectly fit the training data but fail to generalize well to unseen data.

2. Doesn't Handle Non-linearity Well: R² assumes a linear relationship between the independent variables and the dependent variable. It may not be appropriate for complex models that don't exhibit a linear pattern.

3. Insensitive to Changes in Data: Since R² is based on the sum of squares, small changes in the data can sometimes cause large variations in the R² value, especially when the data is noisy.

Adjusted R-Squared: A Better Alternative?

While R² is useful, it has a significant drawback: it always increases when you add more variables to the model, even if those variables aren’t actually contributing useful information. This means that R² can give you an inflated sense of the model's quality when you're working with multiple features.

To address this issue, Adjusted R-squared comes into play. Adjusted R² adjusts the statistic based on the number of predictors in the model. It is particularly useful when comparing models with different numbers of features.

The formula for Adjusted R² is:

$$Adj R^2 = 1 - (\frac{(1-R^2)(n-1)}{n-p-1})$$

Where:

• n is the number of data points.

• p is the number of independent variables (predictors).

When to Use R-Squared?

R² is most effective when you're dealing with linear regression models and are trying to evaluate the model's ability to capture the relationship between input features and the target variable. It can also be useful when comparing models on the same dataset. However, it's important to remember that a high R² doesn’t always indicate a good model, especially if the model is overfitting the data or fails to generalize well to new data.

In summary, R² is a valuable metric in regression analysis, but it should always be considered alongside other metrics like MAE and MSE, as well as techniques like cross-validation, to get a fuller picture of model performance.

Most Machine Learning models, even the classification ones, calculate probability of each class and have a predefined decision boundary of 0.5. It means, any probability greater than 0.5 would mean a positive class and less than 0.5 is a negative class. This may look good in theory but it is not practical. There are scenarios where you need models to be extremely sure when making a positive classification (again, the rare disease classification example from the previous article) and AUC ROC Curve makes a whole lot of sense to find the decision boundary.

AUC-ROC Curve: A Quick Overview

Let’s break it down:

• ROC stands for Receiver Operating Characteristic. Fancy name, but all it means is a graph that shows how well your model separates the positive and negative classes as you tweak the decision threshold.

• AUC stands for Area Under the Curve. This is the number that summarizes the ROC curve into one handy score. (Remember area under the curve concept while learning integration which many question why do we learn this and your teacher couldn’t probably explain you that? Yeah, that area under the curve concept)

In plain English, the ROC curve tells you how good your model is at making the right calls, while the AUC is the gold star rating—higher is better.

Why Should You Care About the AUC-ROC Curve?

Let’s face it: accuracy can be a real jerk sometimes. It looks good on paper but doesn’t always tell you the full story. Imagine a dataset where 99% of the cases are “No” and 1% are “Yes.” A model that just says “No” all the time scores a 99% accuracy. Impressive? Not really.

The AUC-ROC Curve doesn’t fall for such tricks. It doesn’t just focus on getting answers right—it checks if your model can tell the difference between the two classes. It’s like asking, “Does this model have good instincts?”

Breaking Down the ROC Curve

Here’s the gist:

1. True Positive Rate (TPR), also known as Recall.This measures how good the model is at catching actual positives.

$$TPR = \frac{\text{TP}}{\text{TP} + \text{FN}}$$

False Positive Rate (FPR): This checks how often the model cries wolf when there isn’t one.

$$FPR = \frac{\text{FP}}{\text{FP} + \text{TN}}$$

At every threshold, you plot these values to create the ROC curve. A perfect model would hit the top-left corner of the graph (TPR = 1, FPR = 0), which means it’s nailing all positives and ignoring all negatives. The AUC quantifies this—higher is better!

So, How Do You Use It?

You didn’t come here just for theory, right? Let’s get our hands dirty with some code. We’ll use Python and the trusty Scikit-learn library to show how it’s done.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_curve, roc_auc_score
import matplotlib.pyplot as plt

# Generate some synthetic data
X, y = make_classification(n_samples=1000, n_classes=2, weights=[0.9, 0.1], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train a model
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Get predicted probabilities
y_scores = model.predict_proba(X_test)[:, 1]

# Calculate the ROC curve
fpr, tpr, thresholds = roc_curve(y_test, y_scores)

# Calculate AUC
auc = roc_auc_score(y_test, y_scores)
print(f"AUC Score: {auc:.2f}")

# Plot the ROC curve
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, label=f"ROC Curve (AUC = {auc:.2f})")
plt.plot([0, 1], [0, 1], 'k--', label="Random Guess")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve")
plt.legend()
plt.show()

What Do the Results Mean?

1. AUC Score:

   • 1.0: Your model’s a rockstar!

   • 0.5: Your model’s as good as flipping a coin.

   • < 0.5: Uh-oh, something’s seriously wrong.

2. ROC Curve:

   • The closer it hugs the top-left corner, the better.

   • A straight diagonal line? That’s random guessing.

When to Use AUC-ROC vs. Precision-Recall

Here’s a quick tip: AUC-ROC is great for balanced datasets. But when your data is heavily imbalanced (like fraud detection or rare disease diagnosis), precision-recall often give more meaningful insights. Why choose one when you can compare both?

The AUC-ROC Curve can be used for imbalanced datasets, but it’s not always the best tool. Here’s why:

• Balanced Datasets: The ROC curve works well because both the True Positive Rate (TPR) and False Positive Rate (FPR) are meaningful metrics. The model's ability to distinguish between classes is clear and reliable.

• Imbalanced Datasets: When there’s a severe imbalance, the False Positive Rate (FPR) can become misleading. This is because the negative class dominates, making the FPR very small even for a model that’s just guessing. In these cases, the Precision-Recall (PR) curve becomes more informative since it focuses on the positive class (which is usually the minority class in imbalanced datasets).

Using the ROC Curve to Set Decision Boundaries

The AUC-ROC Curve isn’t just about evaluating performance; it’s also a handy tool for finding the optimal decision boundary for your model.

By default, many binary classifiers use 0.5 as the threshold for assigning a positive or negative class. However, this one-size-fits-all approach might not work in all cases. For instance:

• In medical diagnostics, false negatives (missing an actual disease) can be catastrophic. You might want to set a threshold closer to 0.3 or 0.4 to ensure fewer false negatives, even if it means slightly more false positives.

• In spam detection, false positives (marking a legitimate email as spam) are annoying. You might prefer a threshold closer to 0.7 or 0.8 to minimize those errors.

How the ROC Curve Helps

The ROC curve gives you a visual way to assess these trade-offs:

1. At different thresholds, calculate the True Positive Rate (TPR) and False Positive Rate (FPR).

2. Choose the threshold where the balance between TPR and FPR aligns with your business goals.

For example:

• A steeper curve at the top-left corner means the model achieves high TPR with minimal FPR. This is a good spot to consider your threshold.

• A threshold closer to 0.8 may prioritize precision over recall, useful for high-stakes scenarios where false positives are costly.

How to find the optimal decision boundary from TPR & FPR?

You can calculate Youden’s Index for the same. It is just another fancy term but it basically is a difference between TPR & FPR. Wherever this index is highest, that’s where if you use a decision boundary, you’ll get the best results.

There are a certain terminologies that each cell of this confusion matrix gets. To understand the terminology, we need to redefine Class A and Class B from the previous example to Positive and Negative. This would mean our matrix would now look something like this

When the actual label is positive and the predicted one is positive as well, that scenario is called a True Positive (TP). Similarly, when the actual label is negative and the predicted one is negative as well, that scenario is called a True Negative (TN).

The matrix will now look like this

When the prediction is negative but the actual output is positive, the scenario will be called a False Negative (FN) and similarly, when the actual is negative but prediction is positive, the scenario becomes a False Positive (FP).

Now, the False Positive is sometimes referred to as Type I Error and False Negative is referred to as Type II Error. Why Type I and Type II Errors? It will be discussed separately as it is a large topic of its own.

With terminologies completed, let’s derive a few metrics that we can use.

Precision

Precision as a metric explains how precise the model is when predicting a positive output. Meaning, whenever the machine learning model predicted a positive output, how many times was it indeed positive.

Mathematically, it can be written as

$$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$$

Simple, right?

Precision comes in handy during development of the machine learning models for which being correct when making a positive prediction is extremely important. One particular example comes in when building a machine learning model for rare disease identification.

Let’s say you build a model for which you get

• True Positives = 50

• False Positives = 150

• True Negative = 9750

• False Negative = 50

Going by these numbers, we get an accuracy of 98%. A really great number, isn’t it? But looking at the precision, we are only correct 25% times we say the person is positive for a certain disease.

Recall

Recall basically tells the ratio of how many positive examples the model can detect.

Mathematically, you can write it as

$$\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$$

Looking at the same example of rare disease identification, we see that the model identifies only 50% of the positive samples. Meaning, 50% of the people that are actually ill will have a negative test report which means they will not be able to get treatment on time for the disease. We don’t want that to happen at all.

A little detour as this concept is going to come in great detail later in the series but a basic idea is required to understand the next metric.

When we develop a machine learning model, we usually perform hyperparameter tuning to identify the right set of hyperparameters which gives the best results. To do so in an automated fashion, we can make a simpler algorithm if we try to optimize for a single metric.

F1 Score

From the detour, you have gotten the gist of what this metric is. It combines both Precision & Recall to a single metric. It is highly useful for scenarios where both Precision & Recall need to be optimized.

F1 Score is a harmonic mean of both Precision & Recall. If you don’t know what harmonic mean is, it’s this formula:

$$\frac{2}{\text{F1 Score}} = \frac{1}{\text{Precision}} + \frac{1}{\text{Recall}}$$

For the same use case as above, if we put in precision & recall, we get an F1 score of 33.3%.

Before jumping right into the algorithms, we first need to discuss one more idea - preparing the data for testing. This is a topic that should be and would be discussed in a separate post in an excruciating detail. For now, what we’'ll do is take the example dataset from the previous post of this series and segregate it into 2 datasets - one will be used for training and another for testing. This way, we have a dataset for which we know the actual answers and can easily compare the output of the machine learning models we develop.

To do so, we’ll again take use of the scikit-learn library and use a function called train-test-split that does exactly what we described before.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

model = LogisticRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)

The test size of 0.3 means that 30% of the data from the input training data should be reserved for testing purposes. This way, we have the actual output that model should give stored in the variable y_test and the output that the model have in y_pred.

Another important thing to understand is to answer 2 core questions -

1. Why do we evaluate the model?

2. What insights do we need to collect from a model in order to make better decisions?

3. How do we use the metric to optimize the model? (This will be covered in a separate post later in the series)

So all the evaluation algorithms that we are going to look at, we will try to answer these 3 questions. Let’s begin with the evaluation of classification models.

Accuracy

Perhaps one of the most straight forward metric. Let’s say you took an MCQ test that had 100 questions. You answered, 74 correct. Your accuracy is 74%. This is a calculation that all of us have intuitively been doing whenever we see ratios.

In really simple terms, Accuracy is defined as how many times you were correct divided by how many attempts you made. In terms of Machine Learning, you calculate accuracy by how many times the output of the model was correct divided by how many times the model was used.

We can calculate accuracy of our machine learning models using accuracy_score function from the metrics module of the library.

from sklearn.metrics import accuracy_score

accuracy = accuracy_score(y_test, y_pred)

Now, if Accuracy would be the best metric, it would have been the only metric to exist and this blog post would be over right here and I’d probably go to sleep instead of writing this at 11:00PM on a Saturday. But alas, it isn’t.

Let’s take 2 examples here -

Example A:

In this case, there are 9 correct predictions and 1 wrong output. This means, the model is 90% accurate.

Example B:

In this case as well, the accuracy of the model is 90%. But if you look closely, it has wrongly predicted every time the model should have predicted 0. Now this is a scenario that does occur in a lot of problems (credit card fraud identification, for instance) where there are very few examples of a certain class during training and testing the model, due to which the model gets biased to a certain class if not trained properly and carefully. This means, for any problem with an imbalance between classes, accuracy is a wrong metric to use.

There is one more challenge with using accuracy as the source of truth when working with a multi-class classification problem. It will not help you identify, at the minute level, if the model is confusing two of the classes, or is biased towards a single class or is straight up guessing and got lucky.

All these challenges mean that we need to look into a bit more sophisticated evaluation algorithms that do a bit more than just provide a number as a output.

These challenges, lead us to a stepping stone towards the solution - Confusion Matrix.

Confusion Matrix

It is not an evaluation metric, something that should be cleared at the start itself. It is something that you can use alongside the primary metric to answer the question “is my model getting confused between two classes?” while using the primary metric to optimize the model.

A confusion matrix can look something like this:

Rows of the confusion matrix are the actual classes while the columns are the predicted classes. To interpret the above matrix, there were 45 instances where the model predicted output as ‘A’ and it indeed was ‘A’ while 5 times it predicted ‘B’ while it was actually ‘A’. Similarly, 12 times model predicted ‘A’ while it was ‘B’ and 38 times the model correctly predicted ‘B’.

Similar to how you get accuracy in scikit-learn, there is a function for confusion matrix that you can use to get the matrix.

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_true, y_pred)

print(cm)

This matrix addresses one of the two core challenges of accuracy and allows you to understand if the model is confusing two classes. However, it doesn’t yet solve for the imbalance dataset issue for which accuracy is a big challenge.

That’s where we get 3 metrics - Precision, Recall, and F1 Score - all of which will be discussed in the next article of the series.

In Machine Learning, things are a little different. You don’t make the rules. You know what the input is, you know what the output is but don’t really know what operations helped in reaching the specific output.

Let’s take a very simple example to start with.

In this example, you have an input and an output. If I ask you what would be the output when the input is 7, it is pretty straight forward for you to determine the operation here. It is 2^n where n is the input number. We want to emulate the same pattern finding abilities to a computer with Machine Learning so that it can find patterns that even we cannot.

This is an a subset of data from a real world problem being solved with Machine Learning. Try to find a pattern, you have 10 seconds.

Couldn’t find one, right? Well it is just 5 examples and only 4 input values. Most datasets are well over 100,000 examples and can span anywhere from 5 to 500 columns (no hard limit on the columns, just stating a number to make a point).

To solve for such problems, we take use of Machine Learning. The concepts involving Machine Learning go both wide and deep. At every turn of applying Machine Learning, you need to make a decision - which algorithm to use?, how do I check if the algorithm gave a good output?, what operations should I do on the data to make it easier for the algorithm to learn? - just the 3 of the questions you’d need answer to before you design a solution. There are tens of such questions if not a hundred. And the number of questions keep on increasing as you gain experience.

If you have read my previous posts, you’d know how much I emphasize a top-down approach of learning. However, it can’t be used here today. We need to touch base on 2 concepts, before we come back to the top-down approach for the remainder of this series. Please bear with me on this.

Concept 1 - Types of Machine Learning Algorithms

Primarily, there are 2 types of Machine Learning - Supervised and Unsupervised. There are more, but let’s not go into that rabbit hole at the start itself.

In Supervised Machine Learning, you know both the input and the output but don’t know how to reach the output from input and the algorithms that you use to solve for this task fall in the category of Supervised Machine Learning.

In Unsupervised Machine Learning however, you don’t have the output either! Let’s take a real world example, something I’ve worked on before to understand what the heck goes into unsupervised machine learning. For any particular application, sending personalized notification is extremely important. However, it is not really feasible to write personalized notifications for each user (at least wasn’t before the ChatGPT era). So you use the information you have for the users and club them into “N” groups and write a notification for each group. Here, you don’t necessarily have predefined groups. you want the algorithm to self determine the groups in which each user should be.

Concept 2 - Problem Types solved with Machine Learning

Under Supervised Machine Learning, we have 2 problem types - Classification and Regression. Under Unsupervised, we have Clustering (the same example we looked at before).

These are self explanatory problems. For Classification problems, we develop an algorithm that is able to classify a certain data point to one of the many classes the algorithm is designed for. A very famous example you’ll come across when starting with Machine Learning is a Dog vs Cat classifier where in you try to build an algorithm that can classify images of a pet into 2 categories - Dog or a cat. Regression problems, on the other hand, are the ones where the algorithm tries to predict a continuous value instead of a discrete class. An algorithm that estimates the temperature of tomorrow would fall in this category of the problem statement.

Clustering problems, a part of the unsupervised machine learning, requires you to make groups of the data that you have. A prominent example of this has already been discussed before.

Time to get hands dirty

We’ll start with a toy dataset.

This dataset is relatively easy. By just looking at the pattern, it is easy to identify that any input below 10 is getting mapped to 0 while anything larger than 10 is getting mapped to 1.

Let’s implement a really simple script in python.

To do so, we’ll leverage a library called scikit-learn which you can install by running the following command:

pip install scikit-learn

Scikit Learn is the most popular library for Machine Learning and implements a vast majority of algorithms with optimizations and various other concepts that we’ll cover in the later topics.

First, let’s get the data ready:

import numpy as np

X = np.array([9, 12, 13, 4, 6, 11, 7]).reshape(-1, 1)
y = np.array([0, 1, 1, 0, 0, 1, 0])

This creates NumPy array of both the input and the output.

Next, we’ll implement a Logistic Regression algorithm (don’t worry, we’ll cover it) from scikit-learn.

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()

model.fit(X, y)

That’s basically it. You have implemented the first algorithm. [From now on, we’ll call Machine Learning algorithms as 'models’ to be more in line with Machine Learning terminologies]

In Machine Learning terms, when we feed existing data to the model, we are essentially training it. To train a model, all the models have a .fit() function where we pass our existing data.

So what we have done here is essentially trained the Logistic Regression algorithm on the data that we previously generated.

Now, to get predictions on the new set of data, you can just do:

y_test = np.array([8, 18])

y_predictions = model.predict(y_test)

print(y_predictions)
# [0, 1]

And that’s it. We have our first model, trained and doing predictions!

If this was helpful, you can follow the blog or bookmark the series and I’ll be sharing 1-2 articles every week on this series.

In this article, we’ll cover how to use Ollama. In the next, we will discuss the intricacies of Ollama to the depth that no other blog has gone before.

Building your own Chatbot with Ollama

Step 1 - Get the Server running

You will first need to install Ollama server. The process varies a bit for the Operating System you are using so I’ll list all 3 here.

MacOS

There are essentially 2 different ways for you to get started with Ollama on Mac. We’ll of course cover the easiest one. Installation with brew. Running the following command will install the Ollama server on your system.

brew install ollama

I told you it would be really easy to do so.

Linux

To install Ollama on linux, just copy the following command.

curl -fsSL https://ollama.com/install.sh | sh

Windows

On windows, Ollama is still in preview but you can download .exe of Ollama from https://ollama.com/download/windows and that will install Ollama on your system.

Step 2 - Testing if it works

On Windows, you have to just open the .exe you downloaded to start the server. For the rest, we can start the server with the command

ollama serve

This will start the server on the machine itself.

Note that at this stage, the server doesn’t have access to any LLM model. To download the model onto the server, you’ll need to use a different command.

Step 3 - Downloading the Model

Downloading the model is pretty straight forward. Knowing which one to download, is not. There are 116 different LLMs that Ollama supports at the time of writing this and this is excluding different variants of each of the model.

You can explore the entire repository of models that Ollama supports at https://ollama.com/library

For the purpose of this blog, I’ll stick to Qwen2 model. Not because it is better or anything, it has one variant which is just 350MBs and one of the few models that I can run on my laptop itself without renting out a VM.

To download, open a separate terminal while keeping the server running (pro-tip, you can also keep the server running in the background by adding & to the end of the command and keep using the same terminal for next steps) and execute the following step.

ollama pull qwen2:0.5b-instruct

It will download the Qwen2 model with a variant that has 0.5 billion parameters. You can in same way download llama3.1, phi3, or any other model that you find that suits your needs.

Step 4 - Interact with the model

We can interact with the model directly from the command line itself. To prompt a model with the prompt “Who is the Prime Minister of India?”, you will need to run,

ollama run qwen2:0.5b-instruct "Who is the Prime Minister of India?"

This command will first load the model onto the GPU or the CPU, whichever is available and prompt the model with your query. Internally, it is making an API call to “/api/generate” endpoint which you will be able to see on the logs it will print on the terminal alongside the response.

Step 5 - Building your Chat app with this

To do so, we’ll use Ollama’s Python client (there is also a JavaScript one) and use Streamlit to build the interface for the chat application.

Installing the dependencies.

pip install ollama streamlit

Next, just copy paste this code.

import streamlit as st
from ollama import Client

client = Client(host="http://localhost:11434")

st.title("Hey, it's just like ChatGPT, but free!")

# Initialize chat history
if "messages" not in st.session_state:
    st.session_state.messages = []

# Display chat messages from history on app rerun
for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])

# Accept user input
if prompt := st.chat_input("What is up?"):
    # Add user message to chat history
    st.session_state.messages.append({"role": "user", "content": prompt})

    # Display user message in chat message container
    with st.chat_message("user"):
        st.markdown(prompt)

    # Generate and display assistant response (example)
    response = client.chat(model="qwen2:0.5b-instruct", messages=st.session_state.messages)["message"]["content"]
    with st.chat_message("assistant"):
        st.markdown(response)

    # Add assistant response to chat history
    st.session_state.messages.append({"role": "assistant", "content": response})

I too generated this code with Perplexity and just added the model name 😂 If you are new to Streamlit, there is this wonderful playlist you can check.

Now, just execute the script with

streamlit run main.py

And you’ll instantly have a chat interface like below

And there we have it. Our own AI chatbot!

In the next article next week, we will be discussing Ollama in much more detail. I’ll share everything I learnt during my last two weeks reading Ollama codebase (which, trust me I have spent hours scratching the internet for, you’ll not find anywhere else). So for that, please follow and subscribe to the newsletter.

Until then, have a healthy and happy week.

While we are discussing the use cases, you might want to hold and think of one case where you would never use Bidirectional Neural Networks and comment the same.

How BiRNN work (or essentially any Bidirectional network works) is simple. There are 2 different RNN layers processing data from either direction and the output is then merged together to form the final output.

BiRNNs and other bidirectional networks are extremely complex, computationally speaking. This makes them harder to train, require more memory, and takes considerably more time to train. However, they do usually perform better in terms of unidirectional networks in terms of accuracy and also can handle variable length sequences better.

Attention Mechanism is a way to provide the model with the information of which aspects of the training data are more useful for making prediction. If you could, try to compare the Attention Mechanism with that of your brain. When reading something, do you put equal emphasis on each word or more emphasis on the keywords? Or when you've just watched a movie and tell the story to someone else, do you narrate a whole 3 hour script or a summary of the most-important bits? It's the same case with the Attention Mechanism. It focuses on the most important bits of information.

Where does Attention finds it's use?

The primary use-case for Attention Mechanism is in the Natural Language Processing tasks. And it also happens to be the reason this was invented. Translation of sentences from one language to another often poses a challenge of remembering the context for a longer periods of time and Attention enables it.

However, lately, it has found its use in the Computer Vision tasks as well. Enabling the model to focus on the pixels that add the more value to the final prediction. We'll discuss more on it when we cover the Computer Vision topics.

How does Attention even work?

The core idea behind an attention mechanism is relatively straightforward: it allows a model to automatically focus on the most relevant parts of the input for making a decision or prediction.

Each part of the input has its individual weight. These weights are called attention weights and basically tell the model how much focus each part of the input deserves. Now, it depends on the kind of Attention Mechanism used (more on it in just a sec) but the gist is the same.

These weights make up a vector representation of the input called the context vector. This vector is used by the model to identify the part of the inputs to focus more on.

There are 2 primary forms of Attention - Global and Local. It is self-explanatory what it means. Global assigns attention weights to the entirety of the input while the Local assigns attention weights to the specific parts of the input. There are many more, and each deserves its own article. Perhaps once a few more concepts are covered first.

(I am skipping over the architecture for now. Will bring it up once the Transformers are covered in this series.)

An astonishing difference, right? In this post let's learn what GRUs are, how they learn, and what is the reason behind GRUs not being the goto algorithm for NLP.

What are GRUs and how does it learn?

Gated Recurrent Units (GRUs) were designed with one purpose. To give comparable performance of LSTM while reducing the number of trainable parameters. And they do it very effectively. And this is how they achieve it.

Contrary to LSTMs (it's also counterintuitive), output of a GRU is only 1 value. If you have checked my previous post on LSTMs, LSTMs tend to have 2 outputs, one aligned with the short term memory and another the long term memory. So the question remains, how does it maintain the context for long?

GRUs simplify the LSTM design with two gates: a reset gate and an update gate. These gates decide what information should be passed to the output. The update gate helps the model determine the amount of past information (from previous time steps) that needs to be passed along to the future. This is akin to how much of the long-term memory should be used in the current state. The reset gate, on the other hand, decides how much past information to forget. This functionality is somewhat similar to the forget gate in LSTMs, but GRUs combine this with the input gate into a single update gate to make the model more efficient.

Despite having only one output, GRUs are able to maintain the context for long sequences through a clever balancing act performed by these gates. They modulate the flow of information inside the unit without separate memory cells, which are present in LSTMs. This allows GRUs to still capture dependencies from large spans of time, deciding at each step what to keep from the past and what new information to add. By adapting these gates' settings at each step in the sequence, GRUs can keep track of long-term dependencies, thus enabling them to maintain context and perform various sequence modeling tasks effectively.

Why aren't they used as much as LSTMs?

GRUs for the most part, give a close fight to LSTM in terms of performance. The final decision of which one to use lies on the dataset and the problem at hand. It's not written on stone but from experience, when dealing with datasets with smaller sequence lengths, GRUs outperform LSTMs and the inverse happens when the sequence length starts to increase. This means, LSTMs are better when it comes to remembering context for a longer time compared to GRUs.

Secondly, GRUs have been around for far less time than LSTMs. GRUs were introduced only in 2014. Coincidentally, Attention Mechanism (we'll discuss this in next week's post) was also introduced around the same time. Attention brought most of engineer's and researchers attention towards it, overshadowing GRUs.

Just to make it clear, it is not like GRUs are not at all used, it's just the use-cases that the industry is focussed on right now, fit LSTMs and Attention Mechanism more. For instance, chatbots. The need to remember the context of conversation for a long time is extremely important. And LSTMs do it better than GRUs.

To fix the problem of Vanishing Gradient, Long-Short Term Memory (LSTM) Networks were introduced. LSTM maintains the context in 2 different ways - Long Term Memory and Short Term Memory. Both have a different purpose.

Before diving into the architecture and representation of LSTMs, it is important to understand the concept of Short Term and Long Term Memory and what exactly does it do.

Short Term Memory to put it simply, keeps the latest information in the memory that is important for making predictions.

Long Term Memory is used to keep important information in context while making prediction for a long time. Basically, it is a persistent storage of all the important keywords in the data.

Here’s an extremely rough approximation of how it looks.

How LSTMs do the magic?

This overwhelming diagram below is a representation of a cell in LSTM.

The line on top, the green one, that is what is called Cell State which essentially is the Long Term Memory.

The line on bottom, the pink one, that is the Hidden State which, if you are following along could guess, is Short Term Memory.

Slowing down for a second. We were discussing the RNN problems. How does the Cell State and Hidden State solve it? Well, if you look closely, there is no weight to Cell State. Meaning, there is absolutely no chance of gradient to explode or vanish, something that is possible in the case of RNNs.

Moving on, let’s label the diagram for better understanding.

The box 1 is a way to determine the impact of input and the short term memory on the long term memory. It controls how much of the long term memory needs to be remembered. Hence, it is also known as Forget Gate.

The box 2 and 3 determines how much of the input to this unit should be remembered for the long time. Box 3 is calculating the potential long term memory for this input while box 2 is responsible for determining the “how much” part. Important to note, the only difference in both the boxes is the activation function. These 2 boxes make up what is called Input Gate.

Box 4 and 5 are responsible for calculating the Short Term Memory for the next LSTM unit. Box 5 is where the calculation of potential short term memory takes place which is done using the Long Term Memory and not the input and the “how much” aspect is calculated by box 4. These boxes make up what is called Output Gate.

So with the use of Input Gate, Forget Gate, Output Gate, Cell State, and Hidden State (that's a lot!), LSTMs solve the problem of vanishing gradient with Recurrent Neural Networks.

Let’s get started.

For the purpose of this tutorial, I am using Netflix Stock Price Data available on Kaggle and only choose to predict the Daily High of the stock price.

Step 1 - Preprocess the Data

from sklearn.preprocessing import MinMaxScaler

high_data = data["High"].values.reshape(-1,1)

scaler = MinMaxScaler()
scaled_high_data = scaler.fit_transform(high_data)

The code snippet above does 2 things.

First, it selects only the “High” column data and converts it into a NumPy Array. Then it changes its shape. Understand this from the example.

data["High"].values

This part of the code is responsible for converting the “High” column data to a NumPy Array. It will give the output like

[10,20,30,40,50]

Then, .reshape(-1,1) converts this array into a two-dimensional array where in each dimension, you have only 1 element. The output of the above array now looks like this

[[10],
 [20],
 [30],
 [40], 
 [50]]

Step 2 - Preparing the Dataset

def create_dataset(dataset, time_step=1):
    X, Y = [], []
    for i in range(len(dataset)-time_step-1):
        a = dataset[i:(i+time_step), 0]
        X.append(a)
        Y.append(dataset[i + time_step, 0])
    return np.array(X), np.array(Y)


X, y = create_dataset(scaled_high_data)

X = X.reshape(X.shape[0], X.shape[1], 1)

In Step 2 of this tutorial, we're preparing our dataset for the RNN model.

The `create_dataset` function is key here. It takes our scaled stock price data and creates sequences for training. Each sequence consists of consecutive days' 'High' prices, determined by time_step. If `time_step` is 1, we use today's price to predict tomorrow's. The function goes through the entire dataset, creating these sequences (X) and their next day's price (Y). Finally, we reshape X to fit the RNN's expected input shape, making it ready for the neural network to process.

This step is crucial as it aligns our data with the way RNNs learn temporal patterns.

Step 3 - Keras Model

model = Sequential()

model.add(SimpleRNN(units=50, activation="relu", input_shape=(1,1)))
model.add(Dense(units=1))

model.compile(optimizer="adam", loss="mean_squared_error")

model.fit(X, y, epochs=2, batch_size=16, verbose=1)

The process of developing a model with Keras begins by initializing a Sequential model, which allows us to stack layers linearly. Within this model, we add a SimpleRNN layer with 50 units. The activation function 'relu' helps the model learn non-linear patterns. The input_shape is set according to our preprocessed data. Next, we add a Dense layer with a single unit. This is our output layer that will predict the next day's 'High' value.

The model is compiled with the 'adam' optimizer and 'mean_squared_error' loss function, both standard choices for regression problems. Finally, we fit the model to our prepared data (X and y). The epochs parameter controls how many times the model will see the entire dataset, and batch_size determines how many data points the model sees before updating its internal parameters. With these steps, our model is ready to learn from the data and make predictions.

And that’s it, you are ready to run the code and build your own Stock Price Prediction Model.

Towards Intuitive Understanding

The above diagram is a high-level representation of 1 neuron of an RNN. Unlike a simple neural network, it has a feedback loop. Output from the Activation Function is fed back to the input and then summed together before sending it again to the activation function.

I’ll be honest, this representation always confused me. Coming from a programming point of view, I was confused, wouldn’t this mean we are always stuck in the loop? The following representation actually cleared the confusion.

Notice anything different in this representation?

The feedback loop basically affects the next data point in the sequence, not the same one. Input X1 output from the activation function is multiplied with Weight W2 and summed with Input X2 before sending it to the activation function. This process continues till forever.

The Curse in Disguise

Take a good look at the unrolled representation of the RNN once more. Especially the feedback loop. The same thing that makes RNNs remember historical context are also its curse (and the basis of further advancements in NLP and Sequence Data Processing).

Let’s consider 2 examples - one where W2 is too small and another where W2 is too large. I’ll keep it intuitive. Mathematically, we’ll discuss in a separate post.

Too Small W2

If the weight of the feedback loop is too small (for the argument sake, let’s say 0.1) the impact of X1 on the prediction of ŷ5 (for instance) would be negligible. Why? Because X1 got multiplied with a tiny value of 0.1 so many times that it has now almost approached 0 (X1x0.1x0.1x0.1x0.1 = X1x0.0001). So no matter what X1 is, it will be only able to provide 0.01% of it to ŷ5.

This problem means that the historical information is not well preserved when making the predictions for a sequence that occurs after a few data points have been processed.

This problem has a name - Vanishing Gradient.

Too Large W2

If the weight of the feedback loop is too large (anything above 1, so for the sake of argument, let’s say 2) the impact of X1 on the prediction of ŷ5 would overshadow the impact of X2 which in turn would overshadow X3 and so on. Meaning, older the history, more the impact it would have on the predictions. Recent events would have next to no impact on the output.

This problem too has a name - Exploding Gradient.

TLDR;

1. Recurrent Neural Networks are a type of Neural Networks that uses historical information to predict next data point in the sequence.

2. RNNs are cursed with the exact thing that makes them unique. If the weight of feedback loop is too small, the gradient vanishes (meaning, RNN would hardly remember the past) and if the weight of feedback loop is too large, the gradient explodes (meaning, RNN would live in the ancient history, present and recent history would have less impact in predictions).