Compare practical tools for local inference, model serving, retrieval, application frameworks, and developer workflows.
Beginner to intermediate · Local CLI, desktop app, and API server
Run open models locally with a simple CLI, model library, desktop app, and local API.
Best for: Developers who want repeatable local model workflows and simple app integrations.
Beginner · Desktop app with local server mode
Desktop app for discovering, downloading, chatting with, and serving local LLMs.
Best for: Builders who want a visual workflow for testing local models before wiring them into apps.
Beginner · Desktop app
Open-source desktop app for running local AI models with a friendly ChatGPT-like interface.
Best for: Users who want a local-first desktop AI workspace with open-source software.
Beginner · Desktop app and SDK
Local AI desktop and SDK project for running open models on consumer machines.
Best for: Beginners and teams testing local private chat on desktops.
Intermediate · Portable local executable
Mozilla-backed project for packaging LLMs into portable executable files.
Best for: Portable demos, simple local distribution, and experiments where one-file packaging matters.
Intermediate to advanced · Local runtime/library
Core C/C++ inference project behind many local GGUF model workflows.
Best for: Low-level local inference, quantized models, CPU/GPU experimentation, and embedded deployments.
Intermediate · Self-hosted web app
Self-hosted AI platform and chat interface for Ollama and other model backends.
Best for: Private ChatGPT-style workspaces over local or self-hosted models.
Beginner to intermediate · Desktop app or self-hosted workspace
Private AI workspace for documents, agents, and local or hosted model providers.
Best for: Private document chat, internal knowledge bases, and workspace-based assistants.
Intermediate · Self-hosted web app
Open-source multi-provider chat platform with a familiar assistant interface.
Best for: Teams wanting a self-hosted chat UI across multiple model providers.
Beginner · Desktop app
Desktop AI workspace for local and cloud models with a user-friendly interface.
Best for: Individuals who want a polished desktop workspace for local and hosted models.
Intermediate · Web app
Open-source modern chat UI for multiple model providers and assistant workflows.
Best for: Builders who want a polished self-hostable chat interface with provider flexibility.
Intermediate · IDE extension
Open-source AI coding assistant for VS Code and JetBrains with configurable models and context.
Best for: Developers who want a local or provider-flexible coding assistant inside their existing IDE.
Intermediate · CLI coding agent
Terminal-based AI pair programmer that edits files in your local git repo.
Best for: Developers who like command-line workflows and want AI edits tied to git diffs.
Intermediate · VS Code extension
Open-source VS Code coding agent for planning, editing files, and using tools.
Best for: Developers who want an agentic coding workflow inside VS Code.
Intermediate · VS Code extension
VS Code AI coding agent forked from the Cline ecosystem with multi-mode workflows.
Best for: Developers testing agentic coding modes, custom roles, and local/provider model setups.
Intermediate · VS Code extension
Open-source VS Code coding agent focused on agentic development workflows.
Best for: Developers comparing modern Cline-style open coding agents.
Intermediate · CLI coding agent
Terminal-based coding agent for working with models and repositories from the command line.
Best for: Developers who want a CLI-native coding agent experience.
Intermediate · Self-hosted coding assistant
Self-hosted AI coding assistant for code completion and team-controlled coding workflows.
Best for: Teams that want more control over code completion infrastructure.
Intermediate · IDE assistant and platform
AI coding assistant from Sourcegraph focused on codebase context and developer workflows.
Best for: Teams already using Sourcegraph or needing codebase-aware assistance.
Intermediate · AI app platform
Open-source platform for building LLM apps, agents, workflows, and RAG applications.
Best for: Teams building internal AI apps without starting from scratch.
Beginner to intermediate · Low-code AI app builder
Low-code builder for LLM apps, agents, retrieval flows, and AI workflow prototypes.
Best for: Fast prototypes for RAG apps, chatbots, agents, and chained AI workflows.
Intermediate · Python/JS framework
Framework ecosystem for LLM apps, agents, tools, retrieval, and observability workflows.
Best for: Developers building custom LLM apps that need integrations and agent patterns.
Intermediate · Python/TS framework
Data framework for connecting LLMs to documents, databases, retrieval, and agents.
Best for: RAG applications where data ingestion, indexing, and retrieval are central.
Intermediate · Python framework
Open-source framework for production-style search, RAG, and NLP pipelines.
Best for: Teams building retrieval pipelines with a more engineering-oriented architecture.
Intermediate · Python framework
Framework for orchestrating role-based AI agents and multi-agent workflows.
Best for: Experiments where multiple specialized agents coordinate around tasks.
Intermediate to advanced · Python framework
Microsoft-backed framework for building multi-agent AI applications.
Best for: Developers exploring research and production patterns for agent collaboration.
Intermediate · Automation platform
Workflow automation platform for connecting AI tools, APIs, databases, and business systems.
Best for: Builders who want AI workflows connected to real business processes and integrations.
Intermediate · Vector database
Open-source vector database with strong filtering and production-oriented retrieval features.
Best for: RAG apps that need vector search, metadata filtering, and self-hosted or managed options.
Beginner to intermediate · Vector database
Open-source embedding database commonly used for quick RAG prototypes.
Best for: Small teams and prototypes that need fast local retrieval setup.
Intermediate · Vector database
Open-source vector database and AI-native search platform with hybrid search features.
Best for: Teams that want vector search, hybrid search, and managed/self-hosted deployment choices.
Intermediate to advanced · Vector database
Open-source vector database designed for large-scale similarity search.
Best for: Teams planning larger vector search systems and scalable retrieval infrastructure.
Intermediate · Vector database
Developer-friendly vector database built around Lance columnar data format workflows.
Best for: AI apps that need local-first or data-lake-friendly vector storage.
Beginner to intermediate · Database extension
PostgreSQL extension for storing embeddings and running vector similarity search.
Best for: Teams that already use Postgres and want a simple RAG database path.
Advanced · Inference server
High-throughput open-source LLM serving engine for production and research workloads.
Best for: Serving open models at higher throughput with batching and OpenAI-compatible APIs.
Advanced · Inference server/framework
Fast serving framework and programming interface for language model applications.
Best for: Serving modern open models with efficient inference and structured generation workflows.
Advanced · Inference server
Hugging Face server for deploying and serving text generation models.
Best for: Teams already using Hugging Face model workflows and deployment patterns.
Intermediate · LLM gateway/proxy
Proxy and SDK for routing requests across many LLM providers with OpenAI-compatible interfaces.
Best for: Teams managing multiple model providers, budgets, keys, and routing rules.
Intermediate · Local inference server
Open-source OpenAI-compatible local inference server for multiple model types.
Best for: Builders who want a self-hosted local API that mimics OpenAI-style endpoints.
Advanced · AI service platform
Model serving platform for packaging, deploying, and operating AI services.
Best for: Teams turning model code into deployable services with repeatable infrastructure.
Intermediate · Observability platform
Open-source observability, tracing, prompt management, and evaluation platform for LLM apps.
Best for: Teams shipping LLM apps that need traces, evaluations, prompts, and production visibility.
Intermediate · Observability and eval platform
Arize Phoenix is an open-source observability and evaluation tool for LLM and ML systems.
Best for: Teams debugging RAG, tracing LLM calls, and evaluating application behavior.
Advanced · Observability standard/tooling
Open standard and ecosystem for traces, metrics, and logs increasingly used in LLM app observability.
Best for: Engineering teams that want AI telemetry to fit existing observability systems.
Intermediate · Evaluation framework
Open-source framework for evaluating RAG pipelines and LLM application quality.
Best for: RAG builders who need repeatable retrieval and answer-quality checks.
Intermediate · Evaluation framework
Open-source LLM evaluation framework for unit-testing model outputs and app behavior.
Best for: Developers who want tests around LLM responses, agents, and RAG systems.
Intermediate to advanced · Local node-based workflow app
Node-based interface for building advanced local image generation workflows.
Best for: Creators and builders who need precise, reusable image generation pipelines.
For builders
Submit it for review or sponsor a featured placement on OpenSourcesAI. You can also email sponsors@opensourcesai.com.