An Overview of Kythe

The Kythe project was founded to provide and support tools and standards that encourage interoperability among programs that manipulate source code. At a high level, the main goal of Kythe is to provide a standard, language-agnostic interchange mechanism, allowing tools that operate on source code — including build systems, compilers, interpreters, static analyses, editors, code-review applications, and more — to share information with each other smoothly.

The remainder of this document gives a basic introduction to the ideas and rationale behind Kythe, and provides links to other more specific documentation about the project.

As the size and scope of a software project grows, developers rely more and more on tools for routine tasks such as building, testing, deploying, debugging, refactoring, and analyzing their source code. Even for a small development team such as a startup, the toolchain used to write, test, and deploy code can be fairly complex — subsuming multiple languages, a variety of source control systems, a variety of build & deployment tools, test frameworks, and of course a great patchwork of scripts to tie it all together.

When a codebase is small and dependencies are few, it’s fairly easy to do these tasks manually: A small codebase of a few thousand files can be imported into an IDE and manipulated directly. As the codebase grows, however, it assumes more and more dependencies on external (“third party”) libraries and tools. Moreover, as a developer team grows, the complexity of performing those tasks increases to the point where it is difficult — or even infeasible — to build, debug, and test the project efficiently on a single workstation.

Kythe grew out of our experience creating a large-scale semantic index of cross-references for the enormous, multi-lingual internal codebase at Google. We found that engineers often lose a lot of time adapting a new tool to their project, and in the process wind up re-inventing a solution to a problem that had already been solved by some other program (e.g., the compiler or an analysis tool embedded in the IDE). The main reason this happens is that the existing solution usually doesn’t play well with the other tools the developers are using. Some teams ‘work around’ this problem by forcing everyone to use the same tools; but in our experience, that approach scales poorly: Integrated environments work well for the languages and tools that are integrated, but the cost of adding new pieces is high. Developer tool-preferences are highly diverse and idiosyncratic, and developers' productivity declines sharply when they are forced to use tools they dislike.

The main premise of Kythe, therefore, is that programming language tools ought to be able to talk to each other easily: Not just the tools for a given language, but across all the languages used in your project — and not just for a single chosen development environment, but (potentially) for any workable combination of tools your developers may use. Considering editors, compilers, build systems, analysis tools, deployment, testing, continuous integration — there are a lot of options for each of these and more; but at present relatively few combinations actually work well together in practice.

The best way to view Kythe is as a “hub” for connecting tools for various languages, clients and build systems. By defining language-agnostic protocols and data formats for representing, accessing and querying source code information as data, Kythe allows language analysis and indexing to be run as services. This, in turn, enables lightweight (“thin”) composition of analysis tools with client tools such as editors, IDEs, and code browsers.

A hub-and-spoke model reduces the overall work to integrate L languages, C clients, and B build systems from a worst-case of O(L×C×B) — combinatorial in the size of the ecosystem — to O(L+C+B): Implementing Kythe compatibility for a given compiler, editor, or build system is, roughly, a constant up-front cost for each component, after which that component can interoperate with all the existing pieces directly.

To make this model work, Kythe provides a language-agnostic graph structure to capture build-system and compiler metadata, as well as semantic information about source code such as cross-references (e.g., definitions and their usages, type information, and cross-language associations). By design, the Kythe graph schema is liberal and extensible — we’ve defined a number of useful subgraphs, but new node and edge kinds are structured so that the graph can easily be extended without recourse to a central authority.

One of the basic design principles of Kythe is that interoperability should not be ‘all-or-nothing’: Tools should adjust gracefully to missing or incomplete data. For many purposes, we’ve found that some information is almost always better than none. At the same time, it is better to emit incomplete data than to emit incorrect data. In practice, the important point is that tools should not “give up” in the presence of incomplete data, as partial results are often still useful.

Although Kythe provides interoperability for many purposes, it does not cover every possible situation. By design, there are some specific problems we are explicitly not attempting to solve with this project:

In short: We are not interested in an UNCOL.

Our goal is to provide a language for sharing data between tools, and while we find that this works well for a large class of interesting problems, there will always be situations that are highly specific to a particular language or data model. For such cases, it’s entirely appropriate to use a representation that is tuned to that purpose.

The core of the Kythe project centers around three themes, which are embodied in our open-source tools and supported by the Kythe team at Google together with any interested contributors:

Essentially all that is needed to participate in the Kythe ecosystem is for a tool to consume and/or emit data in the Kythe format, and — where appropriate — to follow the Kythe schema. You’ll need:

Other documents and examples cover the details of how these pieces are implemented in practice, but the unifying principles are the common data representation and schema.