Introduction

Data sets

The table below provides an overview of the data sets that are available for download. Note that most of these data sets are still maintained in their original formats. Therefore a simple procedure for converting the data must be run regularly. Each data set is distributed in a single, compressed file. The migration guide shows how specific elements in the original formats are mapped to RDF statements. All classes and properties are defined in a single OWL file that can be downloaded along with the data files.

Each data set is distributed in a single file. Duplication of data within and between different files is avoided. The file containing the UniProt data, for example, contains statements connecting proteins to taxa. Further data on the referenced taxa is stored only in the file containing the taxonomy data set. This approach was chosen for the following reasons:

• Always including all referenced data everywhere is not practical.
• Including data increases transfer and processing time. Users who download the complete data set may simply want to import data into their own database which is likely to be normalized.
• Different copyrights and licensing terms may apply to different data sets.

This is how the different files are linked:

Read like this: uniprot.rdf.gz (describes proteins) references components.rdf.gz (describes cellular components) through an "encoded in" property.

We could consider providing an option to retrieve all data including any referenced data for a specific protein, for example, from a web server. Note that OWL only allows us to describe what relationships may exist, but does not restrict how data is physically packaged and distributed.

Conventions

Data is managed at the level of data sets and objects corresponding to individual database entries. Statements are therefore always initially generated from such objects and may be used to recreate objects. To ensure that this can be done in a streaming manner without having to first load all data into memory or a database all statements that describe a particular object are grouped together. But how do we recognize which statements have been grouped together when reading data? There is no direct language or syntax level support for grouping statements. Different data sets can be separated by storing them in separate files. Unfortunately the same approach is not practical for individual objects which may number in the millions. We therefore store all data in such a way that if when reading we expect a series of objects of a type T, any statement with a predicate rdf:type and an rdf:object value of T can be considered to be the first statement for a new object.

x rdf:type T # Start of first top-level object
x rdfs:label 'Foo'
x property y
y rdf:type U
...
z rdf:type T # Start of second top-level object

Statements that can be efficiently inferred with help of information found in the OWL file are generally not written to disk. Bidirectional and symmetric properties, for example, are only stated in one direction. This also helps avoid cycles in the graph which might otherwise complicate parsing.

Statements consist of resources and literals. Each resources is assigned a unique identifier. It is possible for us to distinguish several kinds of resources in our data:

• Resources that correspond to database entries in a life science database. These are always identified with a PURL, see below.
• Classes and properties. These are either defined in one of the standard namespaces (RDF, RDFS, OWL) or in the OWL file distributed with the data.
• Resources without stable identifiers. These are directly or indirectly referenced from life science resources. The identifiers are not stable because they do not occur in the original data source. While importing the data hexadecimal numbers starting from #_1 are generated. The hash prefix makes the identifiers local to the document in which the identifier appears. This allows data from several files to be loaded into the same database without having to worry about conflicts.

Up to date only few databases provide stable URLs for their data. We therefore assign PURLs ourselves, in the following form:

http://purl.uniprot.org/{database}/{identifier}

Note that the resolving authority is always set to uniprot.org. This does not in any way imply that we are responsible for the resource, but states that we know how to resolve the identifier. Identifiers are resolved using a simple system based on URL templates. Setting up a standard compliant resolving service seems inefficient and overkill, at least for the time being. We do not currently support version numbers. According to the specification version numbers may by appended to the identifier after another colon.

All classes and properties defined by us are located in the http://purl.uniprot.org/core/ namespace. The full name of a resource such as the class Gene is therefore:

http://purl.uniprot.org/core/Gene

Several alternative naming schemes exist - the only requirement is that a resource must be identified by a valid URI. The W3C traditionally uses URIs such as http://purl.uniprot.org/core#Gene. It is common practice (though not required) to set up a web server that will return some relevant documentation when the corresponding URL is requested. The problem with the approach used by the W3C is that the complete documentation for all of the ontology will have to be returned every time, as the HTTP protocol does not require clients to transmit the document fragment identifier (the part after the '#'). Another option are LSIDs, e.g. urn:lsid:uniprot.org:core:Gene, however the resolution mechanism for LSIDs is a bit heavyweight and not widely supported (and while being able to resolve a URI is not a requirement, a lot of applications can benefit from this, including e.g. Semantic Web crawlers and browsers).

XML syntax

The standard RDF XML syntax allows the same set of statements to be serialized in quite a few different ways. The document structure we chose is intended to be simple to understand, even for users who have no knowledge of RDF.

This is the basic outline of a document (reformatted for readability):

<?xml version="1.0" encoding="UTF-8"?>

<rdf:RDF 
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:owl="http://www.w3.org/2002/07/owl#
  xmlns="http://purl.uniprot.org/core/"
>
  ...
</rdf:RDF>

The set of namespace prefixes is always the same. Prefixes for the RDFS and owl namespaces are defined so that we may use standard properties such as rdfs:label, rdfs:comment and owl:sameAs. The default namespace is the namespace in which all classes and properties are defined.

The body consists of a series of non-nested rdf:Description elements:

<rdf:Description rdf:about="http://purl.uniprot.org/uniprot/P30089">
  <rdf:type rdf:resource="http://purl.uniprot.org/core/Protein"/>
  <created>1978-04-24</created> <!-- Simple property with literal value. -->
  <annotation rdf:resource="#_1"/> <!-- Property pointing to another resource. -->
  <annotation rdf:ID="_2" rdf:resource="#_3"/> <!-- Property directly attached to a statement. -->
  ...
</rdf:Description>

<rdf:Description rdf:about="#_1">
  <rdf:type rdf:resource="http://purl.uniprot.org/core/Annotation"/>
  <rdfs:comment>...</rdfs:comment>
  ...
</rdf:Description>

<rdf:Description rdf:about="#_2">
  <note>...</note>
</rdf:Description>

In place of rdf:Description, owl:Thing could have been used. However this does not add any information, and may lead to confusion when dealing with hierarchical data where resources may be both individuals and classes.

As a note on the side there are currently several proposals [TriX, RXR] for improving the official syntax, which is often considered verbose and ugly. However the solutions suggested by most of these proposals would nearly double files sizes. This may not be an issue for small data sets, but seems unrealistic for the data we have. On the other hand a mechanism for grouping together a set of statements in a file would be welcome.

Finally, a comparison of distribution file sizes for UniProt data in various formats (in GB):

Parsers

RDF data that is stored in RDF/XML documents can be processed with the help of common SAX or DOM parsers. Alternatively, specialized RDF parsers can be used. RDF parsers either support event- or graph-based parsing. The event-based parsing is comparable to SAX, except that instead of low level events complete statements are passed to the callback handler. The graph-based parsing is comparable to DOM, except that data is loaded into a generic graph rather than a tree structure. Open source RDF parsers are available for most programming languages. Unfortunately none of them met our performance requirements. We were able to improve performance by an order of magnitude by using a custom parser that is limited to the syntax subset described in the previous section. Note that our restricted syntax can be processed by any standard compliant parser, and we in turn may use a standard compliant parser as a fallback solution for processing RDF data from external sources.

In the requirements section we stressed the importance of providing low-tech access to our data. For this purpose a simple Perl module was created. The module has no external dependencies. The graph can be navigated through dynamically generated methods that correspond to the predicates attached to a resource. Depending on the method invocation context single values or arrays are returned. Currently this module only allows data to be read.

use Expasy::RDF;

my $parser = Expasy::RDF::Parser->new('uniprot.rdf');

while (my $entry = $parser->next)
{
  print 'Fields: ', join(', ', keys %$entry), "\n";
  my $id = $entry->id;
  my $mass = $entry->sequence->mass;
  
  foreach my $annotation ($entry->annotation)
  {
    print $annotation->type, ': ', $annotation, "\n";
    print 'Authors: ', join(', ', $_->author), "\n"
      foreach ($annotation->citation);
  }
}

$parser->close;

Shorthand syntax

Since the RDF XML syntax is too verbose for viewing and editing data by hand, a shorthand syntax is required. While several proposals exist already, we ended up creating our own flavor as we needed a simple way to express statements about statements, which is something none of the existing solutions provide.

@prefix <http://purl.uniprot.org/core/>
@prefix rdf <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
@prefix rdfs <http://www.w3.org/2000/01/rdf-schema#>
@prefix owl <http://www.w3.org/2002/07/owl#>
@prefix uniprot <http://purl.uniprot.org/uniprot/>
@prefix doi <http://dx.doi.org/>

uniprot:P30089 
  rdf:type Protein
  curated true
  mnemonic 'UPA3_HUMAN'
  name 'Corticotropin-lipotropin precursor'
  citation :1
  rdfs:seeAlso <http://www.expasy.org/cd40lbase/> [:2]

:1
  rdf:type Journal_Citation
  title 'Structure of cDNA clones coding for...'
  author 'Matsubara H.', 'Yamanaka T.'
  name 'Biochem. J.'
  volume '260'
  pages '509-516'
  date 1989
  owl:sameAs doi:10.1002/jcc.10386

:2
    rdf:type Related_Statement
    rdfs:comment 'European CD40L defect database'

Diff and patch

Computing differences between small sets of statements is straightforward, if no blank nodes occur in the data. This, incidently, is a side effect of the syntax restrictions outlined in a previous section. After sorting the statements by subject the standard diff algorithm can be applied. The only difference is that instead of lines we are dealing with statements. If resource identifiers are stable we can go one step further an allow patches to be applied in almost any order. The same approach is supported by some version control systems [arch].

Diff diff = new Diff();
Patch patch = diff.createPatch(statements, modifiedStatements);
...
assertEquals(modifiedStatements, patch.apply(statements));

Visualization

A popular way of visualizing data are graph drawing tools [IsaViz]. Unfortunately the output of such tools is rarely useful when the input is complex. Another approach is to generate simple web pages with lists of properties for each resource [Joseki]. Ideally, schema information such as rdfs:labels and rdfs:comments is be integrated into such views. If the syntax of the input follows a strict set of rules XSL style sheets can be used to generate web pages, see screenshot below.

Ontology

The ontology describing all classes and properties was created and is maintained with Protege. Protege does an excellent job of providing a simple user interface for viewing and editing OWL files. One minor issue with Protege is that it does not allow arbitrary URNs as identifiers for owl:Things but instead required the identifiers to be splittable into a namespace part and an NCName.

Wherever possible we use standard RDFS properties rather than introducing our own terminology. In particular:

• rdfs:label
• rdfs:comment
• rdfs:seeAlso

Unfortunately OWL does not support restrictions on these properties. Another issue is that OWL does not allow us to describe reified statements. In consequence it is not possible to define a complete class Statement_With_Evidence that has as its members all statements to which at least one evidence property is attached.

Triple store

In order to be able to efficiently store and retrieve individual sets of statements and run queries some kind of database is required. In particular such a system must be able to:

• Load more than a hundred million statements within hours, not days, even on cheap hardware.
• Run arbitrary queries and return results within seconds or minutes.
• Store or retrieve small sets of statements within milliseconds.
• Return statements in insertion order. Even where there is no meaning in the ordering of statements this behavior is often preferred by users. In some cases it may be possible to apply special sorting routines, but it shouldn't be necessary to do so every time data is retrieved from the store.

Several open source solution were evaluated but did not meet our performance requirements. To be fair, few projects claim to support such large data sets and most focus on providing advanced features such as inference capabilities instead. Kowari was the most promising solution, but does not at the time fulfill the last two requirements. We therefore implemented a simple solution based on a relation database.

Benefits over a conventional database schema:

• Can add new data sets without having to do anything.
• Users creating queries only need to be familiar with the logical structure of the data, not how the data is mapped to specific tables.

Drawbacks:

• No constraints are enforced by the database. However, most constraints are too complex to be handled by the database anyway. Also, constraints should be kept together and not duplicated unnecessarily. For a possible solution, see next section.
• Performance is worse. However: A schema with a single table can be tuned more carefully than a schema containing dozens of different tables.

RDF seems well suited not only for integrated but also for federated approaches to storing data. However it is difficult to achieve reasonable performance when working with large distributed data sets.

Rule engine

In order to detect conflicts between the OWL schema and instance data or to infer statements that are not stated explicitly a rule engine was set up. This approach also allows rules that can not be expressed in OWL (e.g. the value of a property begin must not be greater than the value of another property end of the same resource). A prototype was implemented with Jess. Data from the schema is translated into facts, as is instance data. The rule engine then uses a few general rules to detect certain conflicts.