The post From Amazon Reviews to Numbers: A Hands-On Tour of One-Hot, Bag of Words, and TF-IDF appeared first on Vijay Gokarn.

The post The GenAI Landscape: From Zero to Transformer Series name: GenAI Mastery Series — Chapter 02 appeared first on Vijay Gokarn.

The post Creating AI Storytelling Agents Using Flowise: A Step-by-Step Guide appeared first on Vijay Gokarn.

The post Long Context LLM Comparison appeared first on Vijay Gokarn.

Retrieval-Augmented Generation. It is a method that combines the strengths of retrieval-based models and generative models to improve the performance and accuracy of AI systems, particularly in natural language processing tasks.

How RAG Works:

1. Retrieval Step: The system first retrieves relevant documents or pieces of information from a large corpus based on the input query. This retrieval process helps to bring in contextually relevant information that the generative model might need to generate a more accurate response.
2. Generation Step: After retrieving the relevant information, the generative model (often a large language model like GPT) uses this information as a basis to generate a coherent and contextually appropriate response.

Applications:

• Question Answering: RAG models can be used to answer questions by retrieving relevant text from a knowledge base and then generating an answer based on that information.
• Chatbots: In conversational AI, RAG models help to provide more accurate and context-aware responses by pulling in relevant information before generating a reply.
• Content Creation: For generating content such as articles, reports, or summaries, RAG models can retrieve relevant data and then generate content that integrates this information effectively.

RAG models help in grounding the limitations of generative AI and removing the hallucinations from the responses.

Why We Needed RAG

Before the advent of long context LLMs, traditional language models had severe limitations in processing and understanding large amounts of text. This constraint hindered their ability to perform tasks like:

• Summarizing lengthy documents
• Answering complex questions requiring extensive knowledge
• Generating text based on large datasets

RAG emerged as a solution to this problem. By retrieving relevant information from external knowledge bases, RAG could effectively expand the model’s access to information, improving its performance on these tasks.

Long Context LLMs New Kid on the block

With the development of long context LLMs, the landscape has changed significantly. These models can now process and understand much larger amounts of text directly, reducing the reliance on external knowledge sources.

Long Context LLMs

• Core concept: Directly process and understand a larger amount of text within a single input.
• Strengths:
  • Can capture complex relationships within the text.  
  • Potentially better at understanding nuances and context.
• Weaknesses:
  • Limited by the maximum context window size.
  • Can be computationally expensive for very long inputs

This has led to a debate about whether long context LLMs will render RAG obsolete. Here is comparison of long context LLM by major players

The Reality: A Complex Interplay

While long context LLMs are impressive, they are not a panacea. Here’s why:

• Computational Costs: Processing extremely long contexts is computationally expensive and time-consuming.
• Attention Limitations: Attention mechanisms, essential for long context models, can still struggle with capturing complex relationships within massive amounts of text.
• Information Overload: Feeding an LLM with an overwhelming amount of information can lead to dilution of focus and potential hallucinations.

Therefore, RAG is not entirely obsolete. It still offers several advantages:

• Efficiency: RAG can be more efficient in retrieving and processing specific information.
• Scalability: RAG can handle virtually unlimited amounts of data.
• Focus: By providing the LLM with targeted information, RAG can improve accuracy and reduce hallucinations.

In conclusion, the relationship between long context LLMs and RAG is complex and evolving. The optimal approach often involves a hybrid strategy, combining the strengths of both technologies. The specific choice depends on the task, the available resources, and the desired level of performance.

The post Long Context LLM’s vs RAG appeared first on Vijay Gokarn.

The post AWS Sagemaker Jumpstart and AWS Bedrock Choosing the Right AI Tool for Your Needs appeared first on Vijay Gokarn.

1. Speed

FastAPI is one of the fastest web frameworks available, thanks to its use of Starlette for the web parts and Pydantic for the data parts. It’s designed to be as fast as possible, ensuring that your API has minimal latency and high throughput.

2. Ease of Use

FastAPI leverages Python’s type hints, making the code easier to write and understand. It automatically generates interactive API documentation (using Swagger UI and ReDoc), which simplifies testing and understanding of the API endpoints.

3. Data Validation

FastAPI uses Pydantic for data validation and parsing. This ensures that the data sent to the API is correctly formatted and validated before any further processing.

4. Automatic Documentation

FastAPI generates interactive API documentation from your code using OpenAPI. This is extremely helpful for developers, as they can see and interact with the API directly from the browser.

5. Asynchronous Support

FastAPI has first-class support for asynchronous programming, making it easy to write asynchronous endpoints that can handle large numbers of concurrent requests efficiently.

6. Dependency Injection

FastAPI provides a powerful dependency injection system that makes it easy to manage and inject dependencies into your endpoints, which can simplify the design of your application and improve its testability.

7. Security

FastAPI includes tools to handle security and authentication, like OAuth2 and JWT tokens, right out of the box.

8. Community and Ecosystem

FastAPI has a growing community and a rich ecosystem of plugins and extensions. It integrates well with other popular Python libraries and tools, such as SQLAlchemy for ORM, Celery for background tasks, and others.

Example: Building a tasklist FastAPI

1. Imports

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import List, Optional
from uuid import UUID, uuid4

• FastAPI: The web framework for building APIs.
• HTTPException: Used to raise HTTP errors.
• BaseModel: A base class from Pydantic for creating data models.
• List, Optional: Type hints for better code readability and type checking.
• UUID, uuid4: For generating unique identifiers for each task or blog post.

2. Initialize FastAPI

app = FastAPI()

• Creates an instance of the FastAPI application.

3. Define the Task Model

class Task(BaseModel):
    id: Optional[UUID] = None
    title: str
    description: Optional[str] = None
    completed: bool = False

• Task: This is a data model for tasks with optional id, title, optional description, and completed status.

4. In-Memory Storage

tasks = []

• tasks: A list to store the tasks. In a real application, this would typically be a database.

5. Create Task Endpoint

@app.post("/tasks/", response_model=Task)
def create_task(task: Task):
    task.id = uuid4()
    tasks.append(task)
    return task

• @app.post(“/tasks/”): This is a POST endpoint to create a new task.
• create_task: A function that accepts a Task, assigns it a unique id, and adds it to the tasks list.

6. Read All Tasks Endpoint

@app.get("/tasks/", response_model=List[Task])
def read_tasks():
    return tasks

• @app.get(“/tasks/”): This is a GET endpoint to read all tasks.
• read_tasks: A function that returns the list of all tasks.

7. Read Task by ID Endpoint

@app.get("/tasks/{task_id}", response_model=Task)
def read_task(task_id: UUID):
    for task in tasks:
        if task.id == task_id:
            return task
        
    raise HTTPException(status_code=404, detail="Task not found")

• @app.get(“/tasks/{task_id}”): This is a GET endpoint to read a specific task by its ID.
• read_task: A function that searches for a task by its ID. If found, it returns the task; otherwise, it raises a 404 error.

8. Update Task Endpoint

@app.put("/tasks/{task_id}", response_model=Task)
def update_task(task_id: UUID, task_update: Task):
    for idx, task in enumerate(tasks):
        if task.id == task_id:
            updated_task = task.copy(update=task_update.dict(exclude_unset=True))
            tasks[idx] = updated_task
            return updated_task
        
    raise HTTPException(status_code=404, detail="Task not found")

• @app.put(“/tasks/{task_id}”): This is a PUT endpoint to update an existing task.
• update_task: A function that updates a task’s information by copying the updated fields and replacing the old task. If the task is not found, it raises a 404 error.

9. Delete Task Endpoint

@app.delete("/tasks/{task_id}", response_model=Task)
def delete_task(task_id: UUID):
    for idx, task in enumerate(tasks):
        if task.id == task_id:
            return tasks.pop(idx)
    
    raise HTTPException(status_code=404, detail="Task not found")

• @app.delete(“/tasks/{task_id}”): This is a DELETE endpoint to delete a task by its ID.
• delete_task: A function that removes a task from the list if found; otherwise, it raises a 404 error.

10. Run the Application

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

• if name == “main”: This ensures that the app runs only if the script is executed directly.
• uvicorn.run: Runs the FastAPI app using Uvicorn, a lightning-fast ASGI server.

Git

The post FAST API appeared first on Vijay Gokarn.

The post Analyzing Wikipedia Articles with Langchain and OpenAI in Databricks appeared first on Vijay Gokarn.

from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
from transformers import BartTokenizer, BartForConditionalGeneration

# ── RETRIEVER: encodes the user's question into a dense vector ──
question_encoder = DPRQuestionEncoder.from_pretrained(
    'facebook/dpr-question_encoder-single-nq-base'
)
question_tokenizer = DPRQuestionEncoderTokenizer.from_pretrained(
    'facebook/dpr-question_encoder-single-nq-base'
)

# ── RETRIEVER: encodes each document into a dense vector ──
context_encoder = DPRContextEncoder.from_pretrained(
    'facebook/dpr-ctx_encoder-single-nq-base'
)
context_tokenizer = DPRContextEncoderTokenizer.from_pretrained(
    'facebook/dpr-ctx_encoder-single-nq-base'
)

# ── GENERATOR: seq2seq model that produces the final answer ──
generator_tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')
generator = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')

# Each document is a dict with title + text
# In production: load from PDF, DB, S3, or a vector store
documents = [
    {"title": "Document 1", "text": "This is the text of document 1."},
    {"title": "Document 2", "text": "This is the text of document 2."},
    # Add your real documents here...
]

import faiss
import numpy as np

context_embeddings = []

for doc in documents:
    inputs     = context_tokenizer(doc['text'], return_tensors='pt')
    embeddings = context_encoder(**inputs).pooler_output.detach().numpy()
    context_embeddings.append(embeddings[0])

# Stack into (num_docs, 768) numpy array
context_embeddings = np.array(context_embeddings)

# Build FAISS flat L2 index — exact search, best for small-medium corpora
index = faiss.IndexFlatL2(context_embeddings.shape[1])
index.add(context_embeddings)

print(f"Index built: {index.ntotal} vectors of dim {context_embeddings.shape[1]}")

def retrieve_documents(query, top_k=5):
    # Encode the query into a dense vector
    inputs            = question_tokenizer(query, return_tensors='pt')
    question_embedding = question_encoder(**inputs).pooler_output.detach().numpy()

    # Search FAISS for the top_k nearest document vectors
    _, indices = index.search(question_embedding, top_k)

    # Return the actual document dicts
    return [documents[idx] for idx in indices[0]]


# ── Test it ──
query         = "What is the text of document 1?"
retrieved_docs = retrieve_documents(query)
print(retrieved_docs)

def generate_response(query, retrieved_docs):
    # Concat all retrieved doc texts into one context string
    context = " ".join([doc['text'] for doc in retrieved_docs])

    # Tokenize query + context together (truncate at 1024 tokens)
    inputs = generator_tokenizer(
        query + " " + context,
        return_tensors = 'pt',
        max_length     = 1024,
        truncation     = True
    )

    # Generate with beam search for higher quality output
    summary_ids = generator.generate(
        inputs['input_ids'],
        num_beams     = 4,
        max_length    = 512,
        early_stopping = True
    )

    return generator_tokenizer.decode(summary_ids[0], skip_special_tokens=True)


# ── End-to-end test ──
query         = "What is the text of document 1?"
retrieved_docs = retrieve_documents(query)
response      = generate_response(query, retrieved_docs)
print(response)

The post RAG: Retrieval-Augmented Generation. appeared first on Vijay Gokarn.

import pandas as pd

# Read the raw dataset into a DataFrame
df = pd.read_csv('your_data_file.csv')

# Quick shape check before cleaning
print(f"Rows: {df.shape[0]:,}  |  Columns: {df.shape[1]}")

# Boolean mask: True for every row that is a duplicate
duplicate_mask = df.duplicated()

# How many duplicates exist?
print(f"Duplicate rows found: {duplicate_mask.sum():,}")

# Inspect the duplicate rows themselves
duplicates = df[df.duplicated()]
print(duplicates)

# See ALL occurrences of duplicated rows (including originals)
all_dupes = df[df.duplicated(keep=False)]
print(all_dupes.sort_values(by=df.columns.tolist()))

# ── Strategy 1: keep the FIRST occurrence (default) ──
df_keep_first = df.drop_duplicates(keep='first')

# ── Strategy 2: keep the LAST occurrence ──
#    useful when later rows represent updated/corrected records
df_keep_last = df.drop_duplicates(keep='last')

# ── Strategy 3: drop ALL occurrences of any duplicated row ──
#    use when any repeated row is invalid data
df_drop_all = df.drop_duplicates(keep=False)

# ── Subset: deduplicate only on specific columns ──
df_subset = df.drop_duplicates(subset=['customer_id', 'order_date'], keep='first')

# ── Reset the index after removal (clean 0-based index) ──
df_cleaned = df_keep_first.reset_index(drop=True, inplace=False)

# Confirm rows removed
print(f"Before: {len(df):,}  |  After: {len(df_cleaned):,}  |  Removed: {len(df) - len(df_cleaned):,}")

# Export to CSV — index=False keeps the file clean
df_cleaned.to_csv('cleaned_data.csv', index=False)

print("Cleaned data exported to cleaned_data.csv")

# Optional: also export to Parquet for better performance at scale
df_cleaned.to_parquet('cleaned_data.parquet', index=False)

import pandas as pd

# ── 1. Load ─────────────────────────────────────────────
df = pd.read_csv('your_data_file.csv')
print(f"Loaded {len(df):,} rows")

# ── 2. Detect ────────────────────────────────────────────
duplicates = df[df.duplicated()]
print(f"Duplicate rows found: {len(duplicates):,}")
print(duplicates)

# ── 3a. Keep last occurrence of each duplicate row ───────
df_cleaned = df.drop_duplicates(keep='last')

# ── 3b. Keep first occurrence (swap in if preferred) ─────
# df_cleaned = df.drop_duplicates(keep='first')

# ── 3c. Reset the index to a clean 0-based sequence ──────
df_cleaned.reset_index(drop=True, inplace=True)

print(f"Rows after cleaning: {len(df_cleaned):,}")

# ── 4. Export ─────────────────────────────────────────────
df_cleaned.to_csv('cleaned_data.csv', index=False)
print("Exported to cleaned_data.csv")

The post Pandas Remove Duplicates appeared first on Vijay Gokarn.