Model Context Protocol (MCP)

Smart Tool Filtering: Pruning Unused MCP Tools

Oct 30, 2026 · 3 views
MCPLLM Engineeringcontext-optimizationtoken-efficiencytool-pruning

The Model Context Protocol (MCP) has revolutionized how Large Language Models (LLMs) interact with external tools and data sources. However, as the number of available tools grows, so does the context window required to describe them. This "context bloat" can degrade model performance, increase latency, and drive up API costs. Implementing smart tool filtering is no longer optional; it is a critical engineering practice for scalable MCP implementations.

The Problem: Context Bloat in MCP

When an MCP server registers a large number of tools, each tool definition—including its name, description, input schema, and parameter constraints—consumes tokens in the system prompt. For instance, a server with 50 tools might add 10,000 to 15,000 tokens to every request. This overhead crowds out the actual user query and conversation history, forcing the LLM to operate with a smaller effective context window. Furthermore, excessive tool definitions can confuse the model, leading to tool selection errors or hallucinations.

Strategy: Dynamic Pruning Before Inference

The solution lies in dynamically filtering tools before they are sent to the LLM. Instead of sending the full list of available tools, the MCP client should only include those relevant to the current context. This can be achieved through several strategies:

  • Intent-Based Filtering: Use a lightweight classifier or keyword matching to predict which tool categories are relevant to the user's query.
  • Role-Based Access: Limit tools based on the user's role or permissions, ensuring only authorized tools are exposed.
  • Recency and Frequency: Prioritize tools that have been used recently or frequently, assuming they are more likely to be relevant.
  • Embedding Similarity: Embed the user's query and tool descriptions, then select tools with the highest semantic similarity.

Implementation Example

Here is a simplified example in Python demonstrating how to filter tools based on semantic similarity using sentence embeddings. This approach is robust and scalable for large tool sets.


import numpy as np
from sentence_transformers import SentenceTransformer

# Load a pre-trained model for embedding
model = SentenceTransformer('all-MiniLM-L6-v2')

def filter_tools_by_relevance(user_query, tools, top_k=10):
    """
    Filter tools based on semantic similarity to the user query.
    
    Args:
        user_query (str): The user's input.
        tools (list): List of tool objects with 'name', 'description', and 'schema'.
        top_k (int): Number of top tools to return.
    
    Returns:
        list: Top k most relevant tools.
    """
    # Prepare texts for embedding
    tool_texts = [f"{tool['name']}: {tool['description']}" for tool in tools]
    
    # Generate embeddings
    query_embedding = model.encode(user_query)
    tool_embeddings = model.encode(tool_texts)
    
    # Calculate cosine similarity
    query_embedding = query_embedding / np.linalg.norm(query_embedding)
    tool_embeddings = tool_embeddings / np.linalg.norm(tool_embeddings, axis=1, keepdims=True)
    
    similarities = np.dot(tool_embeddings, query_embedding.T)
    
    # Get indices of top k similarities
    top_indices = np.argsort(similarities)[::-1][:top_k]
    
    return [tools[i] for i in top_indices]

# Example usage
all_tools = [
    {"name": "search_web", "description": "Search the web for current information.", "schema": {...}},
    {"name": "calculate", "description": "Perform mathematical calculations.", "schema": {...}},
    {"name": "send_email", "description": "Send an email to a recipient.", "schema": {...}},
    # ... hundreds of other tools
]

user_query = "What is the stock price of Apple today?"
relevant_tools = filter_tools_by_relevance(user_query, all_tools, top_k=5)

print(f"Selected {len(relevant_tools)} tools for inference:")
for tool in relevant_tools:
    print(f"- {tool['name']}")

In this example, the `search_web` tool is likely to be selected due to its high semantic similarity to the query about stock prices, while `calculate` or `send_email` are pruned. This reduces the context size significantly while maintaining high relevance.

Best Practices and Considerations

  • Caching: Cache embeddings for tool descriptions to avoid recomputing them on every request.
  • Fallback Mechanism: If no tools meet a similarity threshold, consider returning a default set of general-purpose tools or a "no tool available" message.
  • Monitoring: Track which tools are selected and which are used by the LLM to refine your filtering strategy over time.
  • Latency Trade-off: Ensure that the filtering process itself does not add significant latency. Use efficient embedding models and vector databases if the tool count is very large.

Conclusion

Smart tool filtering is a crucial optimization for any production-grade MCP system. By dynamically pruning unused tool definitions before LLM inference, you can reduce context bloat, improve model accuracy, and lower operational costs. As the MCP ecosystem matures and tool registries grow, investing in efficient filtering strategies will be key to delivering fast, reliable, and cost-effective AI applications.

Share:
Related Posts