spotify/NFHTTP

6 years ago

A cross platform C++ HTTP framework.

Raison D’être 💭

At Spotify we have performed studies that show the efficacy of using native backed solutions for interfacing to backends, especially when it came to the battery life of certain devices. In order to carry this forward in the cross-platform C++ world, we created this library that provides a common interface to many of the system level HTTP interfaces, and predictable caching and request hooking. We found that many of the current solutions that claimed to do this lacked key supports for many kinds of platforms, and ended up being libraries that heavily favoured 1 platform and gave the other platforms a generic implementation. We also wanted to provide a caching layer that was consistent across all platforms in our layered architecture.

Architecture 📐

NFHTTP is designed as a common C++ interface to communicate with different systems over HTTP! The API allows you to create objects to make Requests and read Responses. To initiate, send and receive messages you create and use a Client object. This is a layered architecture where requests and responses can pass through multiple places in the stack and get decorated or have actions taken upon them.

The layer design is as follows:

The Modification layer, which takes requests and responses, performs any modifications on them that might be required by the functions provided to the factory, and forwards them on.
The Multi-Request Layer, which takes a request, determines if the same request is currently being executed, then ties the response to that request with the response currently coming in from the previously sent request.
The Caching Layer, which takes a request, determines whether it is cached and if so sends a response immediately, if not forwards the request, and when it receives the response stores the response in its cache.
The Native Layer, which takes a request and converts it to a system level call depending on the system the user is using, then converts the response back to an NFHTTP response and sends the response back up the chain.

Our support table looks like so:

In addition to this, it is also possible to use curl on any of the above platforms or boost ASIO (provided by CPP REST SDK).

Dependencies 🌐

Installation 📥

NFHTTP is a Cmake project, while you are free to download the prebuilt static libraries it is recommended to use Cmake to install this project into your wider project. In order to add this into a wider Cmake project (who needs monorepos anyway?), simply add the following line to your CMakeLists.txt file:

add_subdirectory(NFHTTP)

For iOS/OSX

Generate an Xcode project from the Cmake project like so:

$ git submodule update --init --recursive
$ mkdir build
$ cd build
$ cmake .. -GXcode

For linux

Generate a Ninja project from the Cmake project like so:

$ git submodule update --init --recursive
$ mkdir build
$ cd build
$ cmake .. -GNinja

For Android

Use gradle

android {
    compileSdkVersion 26
    defaultConfig {
        applicationId "com.spotify.nfhttptest_android"
        minSdkVersion 19
        targetSdkVersion 26
        versionCode 1
        versionName "1.0"
        externalNativeBuild {
            cmake {
                cppFlags ""
                arguments "-DANDROID_APP=1 -DANDROID=1"
            }
        }
    }

    sourceSets {
        main {
            jniLibs.srcDirs = ['src/main/cpp']
        }
    }

    externalNativeBuild {
        cmake {
            path "../CMakeLists.txt"
        }
    }
}

For Windows

Generate a Visual Studio project from the Cmake project like so:

$ mkdir build
$ cd build
$ cmake .. -G "Visual Studio 12 2013 Win64"

Usage example 👀

In order to execute HTTP requests, you must first create a client like so:

auto client = nativeformat::http::createClient(nativeformat::http::standardCacheLocation(),
                                               "NFHTTP-" + nativeformat::http::version());

It is wise to only create one client per application instance, in reality you will only need one (unless you need to separate the caching mechanism for your own reasons). After you have done this you can proceed to creating request objects like so:

const std::string url = "http://localhost:6582/world";
auto request = nativeformat::http::createRequest(url, std::unordered_map<std::string, std::string>());

This will create a GET request with no added headers to send to the localhost:682/world location. This does not mean other headers will not be added, we have multiple layers that will add caching requirement headers, language headers, content size headers and the native layer can also add headers as it sees fit. After we have created our request we can then execute it:

auto token = client->performRequest(request, [](const std::shared_ptr<nativeformat::http::Response> &response) {
    printf("Received Response: %sn", response->data());
});

The callback will be called asynchronously in whatever thread the native libraries post the response on, so watch out for thread safety within this callback. In order to execute requests synchronously on whatever thread you happen to be on, you can perform the follow actions:

auto response = client->performSynchronousRequest(request);
printf("Received Response: %sn", response->data());

You might wonder how you can hook requests and responses, this can be done when creating the client, for example:

auto client = nativeformat::http::createClient(nativeformat::http::standardCacheLocation(),
                                               "NFHTTP-" + nativeformat::http::version(),
                                               [](std::function<void(const std::shared_ptr<nativeformat::http::Request> &request)> callback,
                                                  const std::shared_ptr<nativeformat::http::Request> &request) {
                                                 printf("Request URL: %sn", request->url().c_str());
                                                 callback(request);
                                               },
                                               [](std::function<void(const std::shared_ptr<nativeformat::http::Response> &response, bool retry)> callback,
                                                  const std::shared_ptr<nativeformat::http::Response> &response) {
                                                 printf("Response URL: %sn", response->request()->url().c_str());
                                                 callback(response, false);
                                               });

Here we have hooked the client up to receive requests and responses via the hook functions. Because we are now part of the layered architecture, we can perform any changes we want on the requests or responses, such as decorating with OAuth tokens, redirecting to other URLs, retrying responses or even cancelling responses altogether. If you are interested in the concept of cache pinning, it can be done like so:

client->pinResponse(response, "my-offlined-entity-token");

This will then ensure that the response is in the cache until it is explicitly removed, and ignore all backend caching directives.

Contributing 📬

Contributions are welcomed, have a look at the CONTRIBUTING.md document for more information.