[jira] [Updated] (APEXMALHAR-2557) TimeBasedDedupOperator fails with NullPointer

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (APEXMALHAR-2557) TimeBasedDedupOperator fails with NullPointer

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/APEXMALHAR-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vlad Rozov updated APEXMALHAR-2557:
-----------------------------------
    Description:
While using the TimeBasedDedupOperator for deduping, I see that operator 
keeps failing with below NullPointer exception.

I also see that operator is always high on CPU usages. Almost reaching 100%. 
No matter the setting any values for vcores or container memory for operator

{noformat}
2018-03-22 15:10:10,037 INFO  stram.FSRecoveryHandler 
(FSRecoveryHandler.java:rotateLog(103)) - Creating 
hdfs://littleredns/user/SVDATHDP/datatorrent/apps/application_1519410901484_187748/recovery/log 
2018-03-22 15:10:10,056 INFO  stram.StreamingContainerParent 
(StreamingContainerParent.java:log(170)) - child msg: Stopped running due to 
an exception. java.lang.NullPointerException 
        at org.apache.hadoop.io.file.tfile.TFile$Writer.append(TFile.java:387) 
        at com.datatorrent.lib.fileaccess.TFileWriter.append(TFileWriter.java:66) 
        at org.apache.apex.malhar.lib.state.managed.BucketsFileSystem.writeBucketData(BucketsFileSystem.java:179) 
        at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager.transferWindowFiles(IncrementalCheckpointManager.java:139) 
        at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$1.run(IncrementalCheckpointManager.java:110) 
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745) 
 context: 
PTContainer[id=3(container_e3125_1519410901484_187748_01_000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state=ACTIVE]]] 
2018-03-22 15:10:10,915 WARN  stram.StreamingContainerManager 
(StreamingContainerManager.java:processOperatorFailure(1439)) - Operator 
failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1 
{noformat}

  was:
While using the TimeBasedDedupOperator for deduping, I see that operator 
keeps failing with below NullPointer exception.

I also see that operator is always high on CPU usages. Almost reaching 100%. 
No matter the setting any values for vcores or container memory for operator

{noformat}
2018-03-22 15:10:10,037 INFO  stram.FSRecoveryHandler 
(FSRecoveryHandler.java:rotateLog(103)) - Creating 
hdfs://littleredns/user/SVDATHDP/datatorrent/apps/application_1519410901484_187748/recovery/log 
2018-03-22 15:10:10,056 INFO  stram.StreamingContainerParent 
(StreamingContainerParent.java:log(170)) - child msg: Stopped running due to 
an exception. java.lang.NullPointerException 
        at org.apache.hadoop.io.file.tfile.TFile$Writer.append(TFile.java:387) 
        at com.datatorrent.lib.fileaccess.TFileWriter.append(TFileWriter.java:66) 
        at 
org.apache.apex.malhar.lib.state.managed.BucketsFileSystem.writeBucketData(BucketsFileSystem.java:179) 
        at 
org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager.transferWindowFiles(IncrementalCheckpointManager.java:139) 
        at 
org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$1.run(IncrementalCheckpointManager.java:110) 
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745) 
 context: 
PTContainer[id=3(container_e3125_1519410901484_187748_01_000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state=ACTIVE]]] 
2018-03-22 15:10:10,915 WARN  stram.StreamingContainerManager 
(StreamingContainerManager.java:processOperatorFailure(1439)) - Operator 
failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1 
{noformat}


>   TimeBasedDedupOperator fails with NullPointer
> -----------------------------------------------
>
>                 Key: APEXMALHAR-2557
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2557
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>    Affects Versions: 3.8.0
>            Reporter: Vivek Bhide
>            Priority: Major
>
> While using the TimeBasedDedupOperator for deduping, I see that operator 
> keeps failing with below NullPointer exception.
> I also see that operator is always high on CPU usages. Almost reaching 100%. 
> No matter the setting any values for vcores or container memory for operator
> {noformat}
> 2018-03-22 15:10:10,037 INFO  stram.FSRecoveryHandler 
> (FSRecoveryHandler.java:rotateLog(103)) - Creating 
> hdfs://littleredns/user/SVDATHDP/datatorrent/apps/application_1519410901484_187748/recovery/log 
> 2018-03-22 15:10:10,056 INFO  stram.StreamingContainerParent 
> (StreamingContainerParent.java:log(170)) - child msg: Stopped running due to 
> an exception. java.lang.NullPointerException 
>         at org.apache.hadoop.io.file.tfile.TFile$Writer.append(TFile.java:387) 
>         at com.datatorrent.lib.fileaccess.TFileWriter.append(TFileWriter.java:66) 
>         at org.apache.apex.malhar.lib.state.managed.BucketsFileSystem.writeBucketData(BucketsFileSystem.java:179) 
>         at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager.transferWindowFiles(IncrementalCheckpointManager.java:139) 
>         at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$1.run(IncrementalCheckpointManager.java:110) 
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
>         at java.lang.Thread.run(Thread.java:745) 
>  context: 
> PTContainer[id=3(container_e3125_1519410901484_187748_01_000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state=ACTIVE]]] 
> 2018-03-22 15:10:10,915 WARN  stram.StreamingContainerManager 
> (StreamingContainerManager.java:processOperatorFailure(1439)) - Operator 
> failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1 
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)