Spark Stream Error: Could not compute split, block input-123456789 not found

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP


Spark Stream Error: Could not compute split, block input-123456789 not found



I have searched around for similar threads, but this question has never been solved so I am asking again with more information.
When running a Spark Streaming application (Spark 1.6) using a custom Receiver connecting to a Redis database, it seems to all go well for around one hour, before I get the error:


Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 410.0 failed 4 times, most recent failure: Lost task 6.3 in stage 410.0 (TID 33586, sd-9155-ad87.nam.nsroot.net): java.lang.Exception: Could not compute split, block input-1-1532536651000 not found
at org.apache.spark.rdd.BlockRDD.compute(BlockRDD.scala:51)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)



If I go to the job where this Exception occured, I see a different error message:


java.io.FileNotFoundException: *Filename* (No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:88)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:181)
at org.apache.spark.util.collection.WritablePartitionedPairCollection$$anon$1.writeNext(WritablePartitionedPairCollection.scala:56)
at org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:659)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:72)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)



If I restart the application without deleting the checkpointing directory, all input batches will stay in the "queued" state and cannot be processed.



This seems to be solved by deleting the checkpointing directory and rerunning the application. However, this is less than ideal as I need it to run for a longer period of time without Exception and while recovering from the checkpointing directory. Why is this error occurring and how can I correct it?









By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

-

Popular posts from this blog

Keycloak server returning user_not_found error when user is already imported with LDAP

Keycloak server returning user_not_found error when user is already imported with LDAP I am currently running a keycloak server as a service in a kubernetes cluster. I can access it perfectly with the initial keycloak admin account and used it to setup a new realm using LDAP for User Federation. The connection and queries seem to work fine as a I can search for Users and they pop up with their correct username, first name, last name and e-mail so all seemed fine. When I then log out and try to log in with a normal user account that exists in the AD and has been imported I just find get the following error: WARN [org.keycloak.events] (default task-8) type=LOGIN_ERROR, realmId=master, clientId=security-admin-console, userId=null, ipAddress=<my ip>, error=user_not_found, auth_method=openid-connect, auth_type=code, redirect_uri=http://myserver.com/auth/admin/master/console/#/realms/MyRealm, code_id=a8ca7f06-dfec-4e6e-8f6b-74eb871f71da, username=myUser So it seems that in some way wh
Read more

PHP parse/syntax errors; and how to solve them?

Image
PHP parse/syntax errors; and how to solve them? Everyone runs into syntax errors. Even experienced programmers make typos. For newcomers it's just part of the learning process. However, it's often easy to interpret error messages such as: PHP Parse error: syntax error, unexpected '{' in index.php on line 20 The unexpected symbol isn't always the real culprit. But the line number gives a rough idea where to start looking. Always look at the code context . The syntax mistake often hides in the mentioned or in previous code lines . Compare your code against syntax examples from the manual. While not every case matches the other. Yet there are some general steps to solve syntax mistakes . This references summarized the common pitfalls: Unexpected T_STRING Unexpected T_VARIABLE Unexpected '$varname' (T_VARIABLE) Unexpected T_CONSTANT_ENCAPSED_STRING Unexpected T_ENCAPSED_AND_WHITESPACE Unexpected $end Unexpected T_FUNCTION… Unexpected { { Unexpected } } Unexpe
Read more

How to scale/resize CVPixelBufferRef in objective C, iOS

Image
Clash Royale CLAN TAG #URR8PPP How to scale/resize CVPixelBufferRef in objective C, iOS I am trying to resize an image from a CVPixelBufferRef to 299x299. Ideally is would also crop the image. The original pixelbuffer is 640x320, the goal is to scale/crop to 299x299 without loosing aspect ratio (crop to center). I found code to resize a UIImage in objective c, but none to resize a CVPixelBufferRef. I have found various very complicated examples of object C many different image types, but none specifically for resizing a CVPixelBufferRef. What is the easiest/best way to do this, please include the exact code. ... I tried the answer from selton, but this did not work as the resulting type in the scaled buffer is not correct (goes into assert code), OSType sourcePixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer); int doReverseChannels; if (kCVPixelFormatType_32ARGB == sourcePixelFormat) { doReverseChannels = 1; } else if (kCVPixelFormatType_32BGRA == sourcePixelFormat) {
Read more