[clangd] Enable auto-index behind a flag.

Summary:
Ownership and configuration:
The auto-index (background index) is maintained by ClangdServer, like Dynamic.
(This means ClangdServer will be able to enqueue preamble indexing in future).
For now it's enabled by a simple boolean flag in ClangdServer::Options, but
we probably want to eventually allow injecting the storage strategy.

New 'sync' command:
In order to meaningfully test the integration (not just unit-test components)
we need a way for tests to ensure the asynchronous index reads/writes occur
before a certain point.
Because these tests and assertions are few, I think exposing an explicit "sync"
command for use in tests is simpler than allowing threading to be completely
disabled in the background index (as we do for TUScheduler).

Bugs:
I fixed a couple of trivial bugs I found while testing, but there's one I can't.
JSONCompilationDatabase::getAllFiles() may return relative paths, and currently
we trigger an assertion that assumes they are absolute.
There's no efficient way to resolve them (you have to retrieve the corresponding
command and then resolve against its directory property). In general I think
this behavior is broken and we should fix it in JSONCompilationDatabase and
require CompilationDatabase::getAllFiles() to be absolute.

Reviewers: kadircet

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D54894

5 files changed, 83 insertions, 0 deletions

diff --git a/clang-tools-extra/test/clangd/Inputs/background-index/compile_commands.json b/clang-tools-extra/test/clangd/Inputs/background-index/compile_commands.json
new file mode 100644
index 00000000000..1bb835f642d
--- /dev/null
+++ b/clang-tools-extra/test/clangd/Inputs/background-index/compile_commands.json
@@ -0,0 +1,5 @@
+[{
+  "directory": "DIRECTORY",
+  "command": "clang foo.cpp",
+  "file": "DIRECTORY/foo.cpp"
+}]
diff --git a/clang-tools-extra/test/clangd/Inputs/background-index/definition.jsonrpc b/clang-tools-extra/test/clangd/Inputs/background-index/definition.jsonrpc
new file mode 100644
index 00000000000..89d50482300
--- /dev/null
+++ b/clang-tools-extra/test/clangd/Inputs/background-index/definition.jsonrpc
@@ -0,0 +1,51 @@
+{
+  "jsonrpc": "2.0",
+  "id": 0,
+  "method": "initialize",
+  "params": {
+    "processId": 123,
+    "rootPath": "clangd",
+    "capabilities": {},
+    "trace": "off"
+  }
+}
+---
+{
+  "jsonrpc": "2.0",
+  "method": "textDocument/didOpen",
+  "params": {
+    "textDocument": {
+      "uri": "file://DIRECTORY/bar.cpp",
+      "languageId": "cpp",
+      "version": 1,
+      "text": "#include \"foo.h\"\nint main(){\nreturn foo();\n}"
+    }
+  }
+}
+---
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "method": "sync",
+  "params": null
+}
+---
+{
+  "jsonrpc": "2.0",
+  "id": 2,
+  "method": "textDocument/definition",
+  "params": {
+    "textDocument": {
+      "uri": "file://DIRECTORY/bar.cpp"
+    },
+    "position": {
+      "line": 2,
+      "character": 8
+    }
+  }
+}
+# CHECK: "uri": "file://DIRECTORY/foo.cpp"
+---
+{"jsonrpc":"2.0","id":3,"method":"shutdown"}
+---
+{"jsonrpc":"2.0","method":"exit"}
diff --git a/clang-tools-extra/test/clangd/Inputs/background-index/foo.cpp b/clang-tools-extra/test/clangd/Inputs/background-index/foo.cpp
new file mode 100644
index 00000000000..c42ca4d0737
--- /dev/null
+++ b/clang-tools-extra/test/clangd/Inputs/background-index/foo.cpp
@@ -0,0 +1,2 @@
+#include "foo.h"
+int foo() { return 42; }
diff --git a/clang-tools-extra/test/clangd/Inputs/background-index/foo.h b/clang-tools-extra/test/clangd/Inputs/background-index/foo.h
new file mode 100644
index 00000000000..9539f1d7588
--- /dev/null
+++ b/clang-tools-extra/test/clangd/Inputs/background-index/foo.h
@@ -0,0 +1,4 @@
+#ifndef FOO_H
+#define FOO_H
+int foo();
+#endif
diff --git a/clang-tools-extra/test/clangd/background-index.test b/clang-tools-extra/test/clangd/background-index.test
new file mode 100644
index 00000000000..fbd3da884ea
--- /dev/null
+++ b/clang-tools-extra/test/clangd/background-index.test
@@ -0,0 +1,21 @@
+# We need to splice paths into file:// URIs for this test.
+# UNSUPPORTED: win32
+
+# Use a copy of inputs, as we'll mutate it (as will the background index).
+# RUN: rm -rf %t
+# RUN: cp -r %S/Inputs/background-index %t
+# Need to embed the correct temp path in the actual JSON-RPC requests.
+# RUN: sed -i -e "s|DIRECTORY|%t|" %t/*
+
+# We're editing bar.cpp, which includes foo.h.
+# foo() is declared in foo.h and defined in foo.cpp.
+# The background index should allow us to go-to-definition on foo().
+# RUN: clangd -background-index -lit-test < %t/definition.jsonrpc | FileCheck %t/definition.jsonrpc
+
+# Test that the index is writing files in the expected location.
+# RUN: ls %t/.clangd-index/foo.cpp.*.idx
+
+# Test the index is read from disk: delete code and restart clangd.
+# FIXME: This test currently fails as we don't read the index yet.
+# RUN: rm %t/foo.cpp
+# RUN: clangd -background-index -lit-test < %t/definition.jsonrpc | not FileCheck %t/definition.jsonrpc