Partitioned In-Memory Graphs

The Semantics Platform supports partitioned in-memory graphs for storing and querying RDF data. These graphs use an extremely compact indexing format that minimizes the system memory required. As the name implies, the data in the graph can be partitioned to allow parallel read and write access to the data. This parallel access can greatly improve query execution times. Partitioned in-memory graphs manage all their data within the memory space of a single computer process. This is in contrast to a distributed graph which can manage RDF data in multiple processes on multiple machines (I will discuss these is another post).

A writable partitioned in-memory graph can compiled into a read-only form that requires even less memory to hold the same amount of data.

In the case of Semantics.Datacenter, partitioned in-memory graphs are backed by files, with support for journaling, to provide persistent storage of RDF data. Each partition in graph is storage in a separate file. This enables parallel read/write access to the data in the graph and vastly improves data load times.

Creating a Partitioned In-Memory Graph

Using the Model Manager you can create a partitioned in-memory graph in local client model or on a running instance of a Semantics.Datacenter server. The image below shows the screen for creating a graph on Semantics.Datacenter. The graph can be configured to store either triples or quads and also supports full-text indexing.

Loading a Partitioned In-Memory Graph

In a previous posting I discussed how to load data into a partitioned in-memory graph hosted in a local client model using the Model Manger (see Working with large RDF files in the Model Manager). However in the case of a graph hosted by Semantics.Datacenter this is accomplished much more efficiently by using the Semantics.Datacenter Jobs Framework. The Jobs Framework allows you partition the data load process into a set of tasks that can be run in parallel which often requires much less memory. I will discuss this process in another post.