Technical Tutorials

As organizations transition to AI-driven knowledge systems, the challenge isn't just building the RAG pipeline; it's handling the data you already have. Millions of institutional records reside in obsolete formats: Outlook MSG files, standalone EML emails, and Word 97-2003 documents. These files contain unstructured, noisy data that breaks standard ingestion pipelines. This guide explores strategies to sanitize, parse, and structure this legacy content for modern vector stores.

Why Legacy Formats Break Pipelines

Modern vector databases expect clean, semantic text. Legacy formats introduce three types of noise:

Metadata Bloat: MSG files contain binary headers and MAPI properties that obscure the actual content.
Structural Complexity: EML files mix headers, base64 encoded bodies, and attachments in a single stream.
Formatting Artifacts: Old Word docs (.doc) rely on OLE compound files, often resulting in garbled text when naively extracted.

Processing EML and MSG Emails

Email is high-value data but low-quality format. The goal is to extract the body and key metadata (sender, date) while stripping HTML tags and base64 noise.

For EML files, Python's email library is standard. For MSG, you need specialized tools like extract-msg.


import email
from email import policy
import extract_msg

def process_eml(filepath):
    with open(filepath, 'rb') as f:
        msg = email.message_from_binary_file(f, policy=policy.default)
    
    # Extract plain text body
    body_part = msg.get_body(preferencelist=('plain', 'html'))
    text = body_part.get_content() if body_part else ""
    
    # Clean HTML if body is rich text
    import re
    text = re.sub(r'<[^>]+>', '', text)
    
    return {
        'subject': msg['subject'],
        'sender': msg['from'],
        'date': msg['date'],
        'content': text.strip()
    }

def process_msg(filepath):
    with extract_msg.Message(filepath) as msg:
        return {
            'subject': msg.subject,
            'sender': msg.sender,
            'date': str(msg.date),
            'content': msg.body
        }

Handling Old Word Documents (.doc)

Unlike .docx (which is a zip archive of XML), .doc is an OLE2 binary format. Converting these requires a headless conversion engine or a specialized parser.

The most reliable strategy is converting .doc to .docx or .txt first using LibreOffice in headless mode, then parsing the result. This normalizes the data source.


# Shell script to convert .doc to .txt via LibreOffice
# Input: /data/legacy/docs
# Output: /data/processed/txt

libreoffice --headless --convert-to txt --outdir /data/processed/txt /data/legacy/docs/*.doc

Once converted to plain text, you can apply standard NLP chunking. Avoid using raw PDF-to-text heuristics on old Word docs, as they often miss layout context and produce fragmented sentences that degrade embedding quality.

Chunking Strategy for Noisy Data

Legacy documents often have inconsistent heading structures. Fixed-size chunking (e.g., 500 tokens) will frequently split sentences or separate headers from their content.

Use structure-aware chunking:

Identify headers (all caps, bold, or specific numbering patterns).
Group content under the most recent header.
If a group exceeds the max token limit, split it only at paragraph boundaries.

Conclusion

Ingesting legacy formats is a data hygiene task, not just a parsing task. By normalizing EML/MSG metadata and converting OLE-based Word docs to clean text, you significantly improve the semantic quality of your embeddings. Always prototype the parsing pipeline on a small, diverse sample of your actual legacy archive before scaling to the full dataset.