KafkaSinglePortExactlyOnceOutputOperator throwing exception about violation about Exactly once

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

KafkaSinglePortExactlyOnceOutputOperator throwing exception about violation about Exactly once

Vivek Bhide
Hi,

I was testing the KafkaSinglePortExactlyOnceOutputOperator and I see below
error in case of operator fails. This is not reproducible every time but
occurs more than 50% of the time. I tried testing it with single operator
instance, with multiple partitions and parallel partitions but i couldn't
get the pattern since failure doesn't happen always.

Is there anything I missing out during operator setup or cleanup



Regards
Vivek



--
Sent from: http://apache-apex-users-list.78494.x6.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: KafkaSinglePortExactlyOnceOutputOperator throwing exception about violation about Exactly once

Vivek Bhide
Exception stack trace is below

2018-03-13 17:00:53,219 INFO
com.datatorrent.stram.StreamingContainerManager: Container
container_1520983362676_0002_01_000007 buffer server: 80e65028f6a8:60970
2018-03-13 17:00:54,811 INFO com.datatorrent.stram.StreamingContainerParent:
child msg: Stopped running due to an exception. java.lang.RuntimeException:
Violates Exactly once. Not all the tuples received after operator reset.
        at
org.apache.apex.malhar.kafka.KafkaSinglePortExactlyOnceOutputOperator.endWindow(KafkaSinglePortExactlyOnceOutputOperator.java:190)
        at
com.datatorrent.stram.engine.GenericNode.processEndWindow(GenericNode.java:153)
        at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:397)
        at
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428)
 context:
PTContainer[id=3(container_1520983362676_0002_01_000007),state=ACTIVE,operators=[PTOperator[id=3,name=kafkaOutputOperator,state=ACTIVE]]]





--
Sent from: http://apache-apex-users-list.78494.x6.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: KafkaSinglePortExactlyOnceOutputOperator throwing exception about violation about Exactly once

Sandesh Hegde
1. Upstream needs to be idempotent, if not then during recovery you will see that exception. 
2. If you are using a custom tuple, please make sure that it HashCode & Equals methods are properly implemented.


On Wed, Mar 14, 2018 at 10:09 AM Vivek Bhide <[hidden email]> wrote:
Exception stack trace is below

2018-03-13 17:00:53,219 INFO
com.datatorrent.stram.StreamingContainerManager: Container
container_1520983362676_0002_01_000007 buffer server: 80e65028f6a8:60970
2018-03-13 17:00:54,811 INFO com.datatorrent.stram.StreamingContainerParent:
child msg: Stopped running due to an exception. java.lang.RuntimeException:
Violates Exactly once. Not all the tuples received after operator reset.
        at
org.apache.apex.malhar.kafka.KafkaSinglePortExactlyOnceOutputOperator.endWindow(KafkaSinglePortExactlyOnceOutputOperator.java:190)
        at
com.datatorrent.stram.engine.GenericNode.processEndWindow(GenericNode.java:153)
        at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:397)
        at
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428)
 context:
PTContainer[id=3(container_1520983362676_0002_01_000007),state=ACTIVE,operators=[PTOperator[id=3,name=kafkaOutputOperator,state=ACTIVE]]]





--
Sent from: http://apache-apex-users-list.78494.x6.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: KafkaSinglePortExactlyOnceOutputOperator throwing exception about violation about Exactly once

Vivek Bhide
Thanks Sandesh.. It helped

Regards
Vivek



--
Sent from: http://apache-apex-users-list.78494.x6.nabble.com/