Working with large RDF files in the Model Manager

Often it is necessary to load and query a large RDF file to understand the data you are working with. The Model Manager is a good tool for this but has some limitations that I will describe in this posting.

In-Memory Graphs are intended to be used for short term storage of RDF data for performing some application processing. They are not intended to be used as a persistent store of RDF data. In the case of the Model Manger, any data imported into an In-Memory graph is stored in the project file. A Model Manager project file is primarily for storing the configuration of a model. It is not very good at all for storing large amounts of RDF data. Meaning it is not a database.

We have two database products that are very good at storing large amounts of RDF data. These products are called Semantics.Server and Semantics.Datacenter.

The links below provide information about both of these products.

http://www.intellidimension.com/products/semantics-server/

http://www.intellidimension.com/products/semantics-datacenter/

We have two instructional videos that show how to setup, load and query RDF data using each (see links below).

http://www.intellidimension.com/developers/videos/v2.aspx

http://www.intellidimension.com/developers/videos/v3.aspx

If you do not want to use one of our database server products. There is a more efficient way to work with large RDF data files using In-Memory graphs in the Model Manager. You can "Import" a RDF file into an In-Memory graph but then save it as an image file. This is done by creating an image file or read-only image file from the menu on the In-Memory graph. These files have a very compact storage format and can be loaded much more quickly than parsing an RDF file via the import menu. If you don't not intend on modifying any of the data I suggest you use read-only image file.

Now here is the IMPORTANT step. Drop the graph before saving your project so its contents are not saved to the project file. We are working on a better solution for this that will be available in a future release.

When you open your project file next time in the Model Manager. Just create a new In-Memory graph and load the image file using the menu on that graph. This will load much faster than an RDF file because it does not need to parse the file and it can be read directly into memory. Of coarse you will still need enough memory on you local machine to hold the entire contents of the image file but this will be less than the memory that is needed to parse the RDF file or load the data from the Model Manager project file.