Problems with importing big osm file to neo4j

Multi tool use
Multi tool useThe name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP


Problems with importing big osm file to neo4j



I am trying to create a consistent Neo4j spatial database from osm file. I used methods from spatial neo4j manual, but I get errors.


https://github.com/neo4j-contrib/spatial



As a result of these errors the resulting database does not allow to display a layer or layers and I cannot make queries. I do not know what to do - shall I fix the integrity of the osm file or resulting neo4j database.



This is the error message Neo4j displays:



query:


MATCH p=()-[r:LAYER]->() RETURN p LIMIT 25



answer:


Neo.DatabaseError.General.UnknownError: While loading relationships for Node[6276] a Relationship[87889784] was encountered that had startNode: 0 and endNode: 0, i.e. which had neither start nor end node as the node we're loading relationships for



I am unable to delete this faulty node, nor the relationships. I tried:


MATCH ()-[r]->() WHERE ID(r) = 87889784 DELETE r;
match (n) where id(n)=6276 detach delete (n);



The osm file I am trying to download is malopolskie-latest.osm.
Here is the java code I am using:


import org.neo4j.gis.spatial.osm.OSMImporter;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.factory.GraphDatabaseFactory;
import org.neo4j.unsafe.batchinsert.BatchInserter;
import org.neo4j.unsafe.batchinsert.BatchInserters;
import javax.xml.stream.XMLStreamException;
import java.io.File;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
public class SthClass {
public File dir = new File("/home/joanna/Pulpit/ADS/db");
public void downloadData() {
OSMImporter importer = new OSMImporter("malopolskie");
Map<String, String> config = new HashMap<String, String>();
config.put("neostore.nodestore.db.mapped_memory", "90M" );
config.put("dump_configuration", "true");
config.put("use_memory_mapped_buffers", "true");
BatchInserter batchInserter = null;
try {
batchInserter = BatchInserters.inserter(dir, config);
} catch (IOException e) {
e.printStackTrace();
}
try {
importer.importFile(batchInserter, "<path>/malopolskie-latest.osm", false);
} catch (IOException e) {
e.printStackTrace();
} catch (XMLStreamException e) {
e.printStackTrace();
}
batchInserter.shutdown();
GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase(dir);
importer.reIndex(db, 10000);
db.shutdown();
}
}



And pom.xml file:


<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>groupId</groupId>
<artifactId>Neo4j_project</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>6</source>
<target>6</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j</artifactId>
<version>3.5.0-alpha04</version>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-spatial</artifactId>
<version>0.25.5-neo4j-3.4.1</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>neo4j-contrib-releases</id>
<url>https://raw.github.com/neo4j-contrib/m2/master/releases</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>neo4j-contrib-snapshots</id>
<url>https://raw.github.com/neo4j-contrib/m2/master/snapshots</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
</project>



Finally, this is the output when I execute the code above (only error messages):


Missing user or uid: {node_osm_id=13822575, lat=50.0820701, lon=20.0325677, version=10, timestamp=1425301494000, changeset=0}
Missing user or uid: {node_osm_id=13822576, lat=50.0806794, lon=20.0343585, version=5, timestamp=1314342563000, changeset=0}
Missing user or uid: {node_osm_id=13822577, lat=50.0793819, lon=20.0360597, version=5, timestamp=1314072478000, changeset=0}
Missing user or uid: {node_osm_id=13822581, lat=50.0817227, lon=20.0365203, version=5, timestamp=1314342563000, changeset=0}
Missing user or uid: {node_osm_id=13822694, lat=50.081361, lon=20.0355958, version=7, timestamp=1314474145000, changeset=0 }
Missing user or uid: {node_osm_id=13822695, lat=50.0809538, lon=20.0361383, version=6, timestamp=1314474146000, changeset=0}
Missing user or uid: {node_osm_id=13822696, lat=50.0807261, lon=20.035708, version=3, timestamp=1314342563000, changeset=0}
Missing user or uid: {node_osm_id=13822697, lat=50.0806165, lon=20.0358464, version=3, timestamp=1314342563000, changeset=0}
Missing user or uid: {node_osm_id=13822698, lat=50.0807141, lon=20.036034, version=4, timestamp=1314342564000, changeset=0}
Missing user or uid: {node_osm_id=13823454, lat=50.0812194, lon=20.0402708, version=5, timestamp=1257548572000, changeset=0}

/home/joanna/Pulpit/ADS/malopolskie-latest.osm[97579248]: Cannot find member: {type=node, ref=530544342, role=admin_centre}
/home/joanna/Pulpit/ADS/malopolskie-latest.osm[97579248]: Cannot find member: {type=way, ref=218247010, role=outer}
/home/joanna/Pulpit/ADS/malopolskie-latest.osm[97579248]: Cannot find member: {type=way, ref=171269885, role=outer}
/home/joanna/Pulpit/ADS/malopolskie-latest.osm[97579248]: Cannot find member: {type=way, ref=323994747, role=outer}
/home/joanna/Pulpit/ADS/malopolskie-latest.osm[97579248]: Cannot find member: {type=way, ref=323994745, role=outer }
/home/joanna/Pulpit/ADS/malopolskie-latest.osm[97579248]: Cannot find member: {type=way, ref=171269831, role=outer}
/home/joanna/Pulpit/ADS/malopolskie-latest.osm[97579248]: Cannot find member: {type=way, ref=171269874, role=outer}
/home/joanna/Pulpit/ADS/malopolskie-latest.osm[97579248]: Cannot find member: {type=way, ref=324016155, role=outer}
/home/joanna/Pulpit/ADS/malopolskie-latest.osm[97579248]: Cannot find member: {type=way, ref=171269884, role=outer}
/home/joanna/Pulpit/ADS/malopolskie-latest.osm[97579248]: Cannot find member: {type=way, ref=171269880, role=outer}

/home/joanna/Pulpit/ADS/malopolskie-latest.osm[98394658]: When processing the relations, there were 189063 missing members

Exception in thread "main" org.neo4j.gis.spatial.SpatialDatabaseException: java.lang.IllegalArgumentException: org.opengis.referencing.datum.DatumFactory is not an ImageIO SPI class
at org.neo4j.gis.spatial.utilities.LayerUtilities.makeLayerAndNode(LayerUtilities.java:80)
at org.neo4j.gis.spatial.SpatialDatabaseService.createLayer(SpatialDatabaseService.java:332)
at org.neo4j.gis.spatial.SpatialDatabaseService.createLayer(SpatialDatabaseService.java:322)
at org.neo4j.gis.spatial.SpatialDatabaseService.getOrCreateLayer(SpatialDatabaseService.java:228)
at org.neo4j.gis.spatial.SpatialDatabaseService.getOrCreateLayer(SpatialDatabaseService.java:238)
at org.neo4j.gis.spatial.osm.OSMImporter.reIndex(OSMImporter.java:239)
at org.neo4j.gis.spatial.osm.OSMImporter.reIndex(OSMImporter.java:226)
at SthClass.downloadData(SthClass.java:40)
at Main.main(Main.java:8)
Caused by: java.lang.IllegalArgumentException: org.opengis.referencing.datum.DatumFactory is not an ImageIO SPI class
at java.desktop/javax.imageio.spi.ServiceRegistry.checkClassAllowed(ServiceRegistry.java:722)
at java.desktop/javax.imageio.spi.ServiceRegistry.<init>(ServiceRegistry.java:117)
at org.geotools.factory.FactoryRegistry.<init>(FactoryRegistry.java:155)
at org.geotools.factory.FactoryRegistry.<init>(FactoryRegistry.java:146)
at org.geotools.factory.FactoryCreator.<init>(FactoryCreator.java:82)
at org.geotools.referencing.ReferencingFactoryFinder.getServiceRegistry(ReferencingFactoryFinder.java:115)
at org.geotools.referencing.ReferencingFactoryFinder.getFactories(ReferencingFactoryFinder.java:180)
at org.geotools.referencing.ReferencingFactoryFinder.getCRSAuthorityFactories(ReferencingFactoryFinder.java:455)
at org.geotools.referencing.CRS.lookupIdentifier(CRS.java:1086)
at org.geotools.referencing.CRS.lookupEpsgCode(CRS.java:1121)
at org.neo4j.gis.spatial.DefaultLayer.initialize(DefaultLayer.java:222)
at org.neo4j.gis.spatial.utilities.LayerUtilities.makeLayerInstance(LayerUtilities.java:87)
at org.neo4j.gis.spatial.utilities.LayerUtilities.makeLayerAndNode(LayerUtilities.java:78)
... 8 more









By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

L,uAJan3QXg5dA3l A mLkefDsSkEh iMoED3,YwG1h MNnSj,gH,p9yuRr,UCnCqgjPfK,4JYUYGCkiZU
-
xCJs3Q5 x weI,YmYPzWQmGUT7gke,IP,zJPEBUjKIKawVJCfvpvtfohEXUgtoB0k96KwIKuiPoY,42KlfR0Rp9kNhh wpC2 2PkT

Popular posts from this blog

Keycloak server returning user_not_found error when user is already imported with LDAP

Keycloak server returning user_not_found error when user is already imported with LDAP I am currently running a keycloak server as a service in a kubernetes cluster. I can access it perfectly with the initial keycloak admin account and used it to setup a new realm using LDAP for User Federation. The connection and queries seem to work fine as a I can search for Users and they pop up with their correct username, first name, last name and e-mail so all seemed fine. When I then log out and try to log in with a normal user account that exists in the AD and has been imported I just find get the following error: WARN [org.keycloak.events] (default task-8) type=LOGIN_ERROR, realmId=master, clientId=security-admin-console, userId=null, ipAddress=<my ip>, error=user_not_found, auth_method=openid-connect, auth_type=code, redirect_uri=http://myserver.com/auth/admin/master/console/#/realms/MyRealm, code_id=a8ca7f06-dfec-4e6e-8f6b-74eb871f71da, username=myUser So it seems that in some way wh...
Read more

PHP parse/syntax errors; and how to solve them?

Image
PHP parse/syntax errors; and how to solve them? Everyone runs into syntax errors. Even experienced programmers make typos. For newcomers it's just part of the learning process. However, it's often easy to interpret error messages such as: PHP Parse error: syntax error, unexpected '{' in index.php on line 20 The unexpected symbol isn't always the real culprit. But the line number gives a rough idea where to start looking. Always look at the code context . The syntax mistake often hides in the mentioned or in previous code lines . Compare your code against syntax examples from the manual. While not every case matches the other. Yet there are some general steps to solve syntax mistakes . This references summarized the common pitfalls: Unexpected T_STRING Unexpected T_VARIABLE Unexpected '$varname' (T_VARIABLE) Unexpected T_CONSTANT_ENCAPSED_STRING Unexpected T_ENCAPSED_AND_WHITESPACE Unexpected $end Unexpected T_FUNCTION… Unexpected { { Unexpected } } Unexpe...
Read more

415 Unsupported Media Type while sending json file over REST Template

Image
Clash Royale CLAN TAG#URR8PPP 415 Unsupported Media Type while sending json file over REST Template I am trying to send a json file over REST Template. When I send it via POST man as MULTIPART_FORM_DATA, it works fine. The name I am supposed to give is specific (lets say aaa ). Attached screenshot of POSTMAN. But when I try same in code as specified in another stackoverflow post, I get 415 Unsupported Media Type error as org.springframework.web.client.HttpClientErrorException: 415 Unsupported Media Type at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) ~[spring-web-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:616) ~[spring-web-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:572) ~[spring-web-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.web.client.RestTemplate.execute(RestT...
Read more