GPU-accelerated ML inference in Vespa Cloud

In machine learning, computing model inference is a good candidate for being
accelerated by special-purpose hardware, such as GPUs. Vespa supports
evaluating multiple types of machine-learned models in stateless
containers, for
example TensorFlow,
ONNX,
XGBoost and
LightGBM models. For many use-cases
using a GPU makes it possible to perform model inference with higher
performance, and at a lower price point, compared to using a general purpose
CPU.

Today we’re introducing support for GPU-accelerated ONNX model inference in
Vespa, together with support for GPU instances in Vespa Cloud!

Vespa Cloud

If you’re using Vespa Cloud, you can get started with
GPU instances in AWS zones by updating the <nodes> configuration in your
services.xml file. Our cloud platform will then provision and configure GPU
instances automatically, just like regular instances. See the services.xml
reference documentation for
syntax details and examples.

You can then configure which models to evaluate on the GPU in the
<model-evaluation> element, in services.xml. The GPU device number is
specified as part of the ONNX inference
options
for your model.

See our pricing page for details on GPU
pricing.

Open source Vespa

GPUs are also supported when using open source Vespa. However, when running
Vespa inside a container, special configuration is required to pass GPU devices
to the container engine (e.g. Podman or Docker).

See the Vespa documentation
for a tutorial on how to configure GPUs in a Vespa container.

CORD-19 application benchmark

While implementing support for GPUs in Vespa, we wanted to see if we could find
a real-world use-case demonstrating that a GPU instance can be a better fit than
a CPU instance. We decided to run a benchmark of our CORD-19
application – a
Vespa application serving the COVID-19 Open Research Dataset. Its source code is
available on GitHub.

Our benchmark consisted of a query where the top 30 hits are re-ranked, using a
22M Transformer model using batch inference. The measured latency is end-to-end,
and includes retrieval and inference.

See our recent blog
post for
more information about using a Transformer language model to re-rank results.

We compared the following node configurations:

• GPU: 4 vCPUs, 16GB memory, 125 GB disk, 1 GPU with 16GB memory (Vespa Cloud
  cost: 1.87$/hour)
• CPU: 16 vCPUs, 32GB memory, 125 GB disk (Vespa Cloud cost: $2.16/hour)

Results

Conclusion

The GPU of the GPU instance was saturated at 4 clients, with an average
end-to-end request latency at 212 ms and a throughput of 18.8 QPS. The CPU
instance had a higher average latency, at 1011 ms with 4 clients and a
comparatively low throughput of 3.95 QPS.

So, in this example, the average latency is reduced by 79% when using a GPU,
while costing 13% less.