Why is NoSQL becoming more popular

NoSQL databasesSimple illustration of "real world contexts"

Thomas Schäfer: "That is a huge amount of data, several million data records that have to be evaluated there, which of course are not pre-structured, but you can imagine it as if several million loose sheets of paper are lying one behind the other, which then first have to be made machine-readable and then to gain the relevant knowledge from it. "

Manfred Kloiber: This is how Hessian Finance Minister Thomas Schäfer sums up the tasks that the investigators at the Federal Criminal Police Office will have to face. Because the BKA bought the Panama Papers. The criminalists suspect that the evaluation of the papers unmasked a lot of tax evaders in Germany. Peter Welchering, you have researched what evaluation work is planned. Will the data analysts at the BKA proceed in a similar way to the journalists of the Guardian and other media companies, who were able to come up with some astonishing research results?

Germany develops powerful software for NoSQL databases

Peter Welchering: In particular, the colleagues from the Guardian had broad support from academia. And ultimately they could only carry out their research with the help of a huge graph database. Not only computer scientists helped, but also many humanities scholars, who have been working with data models for graph databases for many years, especially in the field of digital humanities, and above all with scripts for the query language Cypher.

The BKA will not be able to access such support from the academia. But the BKA can use interfaces that are now on the market for NoSQL databases and especially for graph databases in order to then evaluate the Panama Papers without having to go too deep into the subtleties of graph technology, for example. For several years, especially in Germany, enormously powerful software for NoSQL databases and also for graph databases has been developed, which offers easy-to-use user interfaces and, above all, extremely strong data security. That is why database experts assume that the entire area of ​​NoSQL databases will develop quite rapidly. This has only just become possible thanks to tailor-made application software from third-party providers. And according to the unanimous assessment of the database experts, there is likely to be an incredibly strong trend in the database market.

Manfred Kloiber: What is going on in the database market, and how interfaces and application packages for NoSQL databases have developed and will continue to develop, we discussed with Dr. Thorsten Liebig from Derivo GmbH in Ulm. He has been relying on NoSQL and graph databases very early on, namely since 2010.

Illustration of "real world contexts"

Thorsten Liebig: So, I believe that NoSQL databases in general and graph databases in particular will gradually continue to dominate the market and expand their market shares because they are simply a very natural means of mapping real-world relationships. SQL databases are still justified, but for many problems, be it in medicine or in other areas, such as road networks, power grids, the term network is already in the data, and that's why Graph databases very suitable.

Speaker: Graph databases are particularly suitable for precisely determining and documenting complex structures, such as in the Panama Papers. Because they represent a real network of people, places or events and depict it as a graphic. A few clicks as a query to the graph database are enough to find out, for example, who transferred what black money and where. But graph databases are not only particularly suitable for such research.

Data can be read in directly

Thorsten Liebig: There are a lot of medical graph data stocks in genomics, in anatomy, in pharmacy, where it is shown which active ingredients are related to which side effects, for example, etc. There is a lot of data that can be loaded directly. This way is very short.

Speaker: Powerful user interfaces have been developed so that the path is very short and the user does not first have to familiarize himself with the inaccessible depths of a database query language like Cypher.

Thorsten Liebig: Users want to see directly from the data which links exist to other objects, for example, and then select them specifically.

"Users should be able to produce their own custom categories and filters"

Speaker: This only works if the data model is powerful enough to be able to search for and show unforeseeable links and relationships. Such a data model has to develop. The database developers and their users are important for this process. Because here a very special kind of cooperation is required.

Thorsten Liebig: "This is the goal of our approach so that the database developers carry out a pre-classification of the objects and the user can produce and reuse his own user-defined categories and filters during his research."

Speaker: For NoSQL databases and, in particular, for graph databases, powerful user packages have been created that work with extremely flexible data models. In particular, the developer community around the graph database Neo4j has recognized that these user packages must be based on common standards. And so evaluations are now also possible with the Panama Papers, which just a few years ago would have required an enormous development effort and would have completely overwhelmed any investigative authority. At the same time, many developers, especially in the area of ​​graph databases, remain extremely skeptical. Because after the completion of a user package, they no longer have any influence on the specific areas of application. That is why there is currently a very exciting discussion going on in the developer community about the limits to the use of such powerful graph databases, their interfaces and their user interfaces.