AI Toolkit for Visual Studio Code

What is AI Toolkit

AI Toolkit is a powerful extension for Visual Studio Code that streamlines agent development. With AI Toolkit, you can:

• 🔍 Explore and evaluate models from a wide range of providers—including Anthropic, OpenAI, GitHub—or run models locally using ONNX and Ollama.
• ? Build and test agents in minutes with prompt generation, quick starters, and seamless MCP tool integrations.

Complete features include:

Feature	Description	Screenshot
Model Catalog	Discover and access AI models from multiple sources including GitHub, ONNX, Ollama, OpenAI, Anthropic, and Google. Compare models side-by-side and find the perfect fit for your use case.
Playground	Interactive chat environment for real-time model testing. Experiment with different prompts, parameters, and multi-modal inputs including images and attachments.
Agent Builder	Streamlined prompt engineering and agent development workflow. Create sophisticated prompts, integrate MCP tools, and generate production-ready code with structured outputs.
Bulk Run	Execute batch prompt testing across multiple models simultaneously. Ideal for comparing model performance and testing at scale with various input scenarios.
Model Evaluation	Comprehensive model assessment using datasets and standard metrics. Measure performance with built-in evaluators (F1 score, relevance, similarity, coherence) or create custom evaluation criteria.
Fine-tuning	Customize and adapt models for specific domains and requirements. Train models locally with GPU support or leverage Azure Container Apps for cloud-based fine-tuning.
Model Conversion	Convert, quantize, and optimize machine learning models for local deployment. Transform models from Hugging Face and other sources to run efficiently on Windows with CPU, GPU, or NPU acceleration.
Tracing	Monitor and analyze the performance of your AI applications. Collect and visualize trace data to gain insights into model behavior and performance.

Getting started

We recommend starting with models hosted by GitHub.

• Follow the installation guide to set up AI Toolkit for your device.
• From the extension tree view, select CATALOG > Models to explore models available. We recommend to getting started with models hosted by GitHub.
• From the model card, select Try in Playground to start experimenting the capability of an AI Model.

Build AI agents

The key feature of AI Toolkit is to build AI agents. The agent builder provides a set of tools to help you create and optimize your AI agents. You can use the agent builder to:

• ? Generate and improve prompts with natural language
• 🔁 Iterate and refine prompts based on model responses
• 🧩 Break down tasks with prompt chaining and structured outputs
• ? Test integrations with real-time runs and tool use such as MCP servers
• 💡 Generate production-ready code for rapid app development
• 🧷 Use variables in prompts
• 🧪 Run agents with test cases to validate your agent easily
• 📊 Evaluate the accuracy and performance of your agent with built-in or custom metrics
• 🔗 Function calling support: Enable agents to invoke external functions dynamically
• 🗂? Agent versioning and version comparison for evaluation results
• 🐞 Local tracing and debugging of agents
• 🚀 Deploy your models and agents to Azure AI Foundry

And a lot of features are coming soon, stay tuned for:

• ?? Deploy your agent to the cloud

Agents can now connect to external tools through MCP (Model Control Protocol) servers, enabling them to perform real-world actions like querying a database, accessing APIs, or executing custom logic.

Feature	Description	Screenshot
Connect to an Existing MCP Server	Use tools from command(stdio) or HTTP (server-sent event)
Build and Scaffold a New MCP Server	Start creating your own MCP server from a simple scaffold and test in Agent Builder

Feedback and resources

We value your feedback to help shape our roadmap. Explore our developer documentation for more features, open issues or share suggestions on GitHub, or join our Discord community to connect with other developers.

AI Toolkit ?? Developer Community.

Data and telemetry

The AI Toolkit for Visual Studio Code collects usage data and sends it to Microsoft to help improve our products and services. Read our privacy statement to learn more. This extension respects the telemetry.enableTelemetry setting which you can learn more about at disable telemetry reporting.