Overview

Pipeline architecture

The full pipeline runs end-to-end inside a single Databricks notebook. Wikipedia articles are loaded from HuggingFace, cleaned to first-line summaries, batched, and sent to GPT-4 via LangChain’s chain interface. Responses are parsed from JSON into a DataFrame.

Step 1

Install required packages

In a Databricks notebook, use %pip magic commands to install packages into the cluster. The %restart_python command refreshes the interpreter to pick up the new packages without restarting the whole cluster.

Step 2

Import libraries

Standard Python utilities (json, time, os) combined with LangChain for the LLM interface, HuggingFace Datasets for Wikipedia data loading, and tqdm for progress visibility during batch processing.

Step 3

Load & clean the dataset

The HuggingFace wikimedia/wikipedia dataset is massive — we take a 10,000 article slice from the English November 2023 snapshot. The cleaning step extracts only the first line of each article (the summary sentence), which is sufficient for category classification and drastically reduces token usage.

Step 4

Configure OpenAI + LangChain

Use getpass to securely prompt for the API key without echoing it to the notebook output. Then initialize ChatOpenAI — LangChain’s wrapper around the OpenAI Chat Completions API.

Step 5 — Core Logic

Define the prompt template

The ChatPromptTemplate structures the conversation: a system message sets the classification task with all 50 categories, and the human message carries the article payload. The double curly braces {{ }} in the JSON schema escape the literal braces so LangChain doesn’t treat them as template variables.

Step 6

Build the chain & test it

LangChain’s pipe operator | composes the prompt template and the LLM into a reusable chain. One call to .invoke() with a single article validates the whole setup before committing to batch processing.

Step 7 — Core Loop

Batch processing with rate-limit handling

Processing 1,000 articles one-by-one would quickly hit OpenAI’s requests-per-minute limit. The solution: accumulate inputs into batches of 8 and call .batch() with a 1.5-second sleep between each batch. tqdm wraps the loop to give live progress in the notebook.

Step 8

Parse results into a DataFrame

Not every LLM response will be valid JSON — network hiccups, model refusals, and malformed outputs all happen at scale. The pattern below separates successful parses from failures so you can inspect and retry the failures without losing the successful results.

Context

Why duplicates matter

Duplicate rows skew aggregations, inflate record counts, bias ML model training, and produce misleading visualizations. A sales total that counts the same transaction twice, a classifier trained on repeated samples — both produce results that look correct but aren’t. Clean data is the foundation everything else is built on.

Pipeline

The four-step deduplication pipeline

Step 1

Load your dataset

Start by reading your data into a pandas DataFrame. pd.read_csv() is the standard entry point for flat files. From here, all deduplication operations work on the in-memory DataFrame — your source file is never modified.

import pandas as pd

# Read the raw dataset into a DataFrame
df = pd.read_csv('your_data_file.csv')

# Quick shape check before cleaning
print(f"Rows: {df.shape[0]:,}  |  Columns: {df.shape[1]}")

Step 2

Detect & inspect duplicates

df.duplicated() returns a boolean Series — True for every row that is a duplicate of an earlier row. Always inspect before you remove — understanding what the duplicates look like helps you choose the right strategy.

# Boolean mask: True for every row that is a duplicate
duplicate_mask = df.duplicated()

# How many duplicates exist?
print(f"Duplicate rows found: {duplicate_mask.sum():,}")

# Inspect the duplicate rows themselves
duplicates = df[df.duplicated()]
print(duplicates)

# See ALL occurrences of duplicated rows (including originals)
all_dupes = df[df.duplicated(keep=False)]
print(all_dupes.sort_values(by=df.columns.tolist()))

Step 3

Remove duplicates — three strategies

drop_duplicates() returns a new DataFrame by default — the original is untouched. The keep parameter controls which occurrence survives. After removing, reset_index(drop=True) gives you a clean sequential index starting from 0.

# ── Strategy 1: keep the FIRST occurrence (default) ──
df_keep_first = df.drop_duplicates(keep='first')

# ── Strategy 2: keep the LAST occurrence ──
#    useful when later rows represent updated/corrected records
df_keep_last = df.drop_duplicates(keep='last')

# ── Strategy 3: drop ALL occurrences of any duplicated row ──
#    use when any repeated row is invalid data
df_drop_all = df.drop_duplicates(keep=False)

# ── Subset: deduplicate only on specific columns ──
df_subset = df.drop_duplicates(subset=['customer_id', 'order_date'], keep='first')

# ── Reset the index after removal (clean 0-based index) ──
df_cleaned = df_keep_first.reset_index(drop=True, inplace=False)

# Confirm rows removed
print(f"Before: {len(df):,}  |  After: {len(df_cleaned):,}  |  Removed: {len(df) - len(df_cleaned):,}")

Step 4

Export the clean data

Write the deduplicated DataFrame back to a CSV. Setting index=False prevents pandas from writing the row index as an extra column — your downstream consumers will thank you.

# Export to CSV — index=False keeps the file clean
df_cleaned.to_csv('cleaned_data.csv', index=False)

print("Cleaned data exported to cleaned_data.csv")

# Optional: also export to Parquet for better performance at scale
df_cleaned.to_parquet('cleaned_data.parquet', index=False)