Kryo version and default serializer

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Kryo version and default serializer

Aaron Bossert
I am writing an application that includes a custom object.  One of the fields is of type Instant and seems to not agree with the version of Kryo that is used in Apex (see exception below).  I now that kryo 4.X supports serializing Instant fields, but for the life of me, cannot seem to find a way to either get Apex to use a newer version of Kryo.  Can anyone help?  Perhaps setting a different streamcodec that uses the newer version of kryo would work, but I cannot seem to find an example of how to set a different streamcodec despite references to the two (Kryo and Java) that ship with Apex.

ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - Catastrophic Error: Execution halted due to Kryo exception!
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): java.time.Instant
Serialization trace:
systemTime (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at com.datatorrent.stram.codec.DefaultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)


--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group


Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Thomas Weise-2
Hi,

It is not necessary to use a different stream codec when you have control over the class that is causing the error. You can annotate a different field serializer instead, using the @FieldSerializer.Bind annotation. Here is an example:


It would be good to upgrade Kryo to 4.x in apex-core master. Would you like to create a JIRA for it? (http://apex.apache.org/contributing.html)

Thanks,
Thomas





On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <[hidden email]> wrote:
I am writing an application that includes a custom object.  One of the fields is of type Instant and seems to not agree with the version of Kryo that is used in Apex (see exception below).  I now that kryo 4.X supports serializing Instant fields, but for the life of me, cannot seem to find a way to either get Apex to use a newer version of Kryo.  Can anyone help?  Perhaps setting a different streamcodec that uses the newer version of kryo would work, but I cannot seem to find an example of how to set a different streamcodec despite references to the two (Kryo and Java) that ship with Apex.

ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - Catastrophic Error: Execution halted due to Kryo exception!
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): java.time.Instant
Serialization trace:
systemTime (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at com.datatorrent.stram.codec.DefaultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)


--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group



Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Aaron Bossert
Thanks, Thomas!  I have implemented the fix you suggested (@FieldSerializer.BInd) and also created a JIRA ticket: https://issues.apache.org/jira/browse/APEXCORE-816

On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <[hidden email]> wrote:
Hi,

It is not necessary to use a different stream codec when you have control over the class that is causing the error. You can annotate a different field serializer instead, using the @FieldSerializer.Bind annotation. Here is an example:


It would be good to upgrade Kryo to 4.x in apex-core master. Would you like to create a JIRA for it? (http://apex.apache.org/contributing.html)

Thanks,
Thomas





On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <[hidden email]> wrote:
I am writing an application that includes a custom object.  One of the fields is of type Instant and seems to not agree with the version of Kryo that is used in Apex (see exception below).  I now that kryo 4.X supports serializing Instant fields, but for the life of me, cannot seem to find a way to either get Apex to use a newer version of Kryo.  Can anyone help?  Perhaps setting a different streamcodec that uses the newer version of kryo would work, but I cannot seem to find an example of how to set a different streamcodec despite references to the two (Kryo and Java) that ship with Apex.

ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - Catastrophic Error: Execution halted due to Kryo exception!
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): java.time.Instant
Serialization trace:
systemTime (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at com.datatorrent.stram.codec.DefaultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)


--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group






--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group


Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Aaron Bossert
I know it has been a little while since I brought this up...I pulled down the 3.7.1-RELEASE branch from GitHub in order to test out the change to Kryo v. 4.0.2 and encountered no problems while building, however, when I try to do mvn install, there are several failed tests.  I am hoping, that other than the obvious (to me, anyway) errors such as not finding yarn-site.xml on my laptop which does not have hadoop installed, someone may be able to see what is going on?

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramRecoveryTest
-------------------------------------------------------------------------------
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest)  Time elapsed: 0.115 sec  <<< FAILURE!
java.lang.AssertionError: flush count expected:<1> but was:<2>
at com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)


-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.engine.StatsTest
-------------------------------------------------------------------------------
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)  Time elapsed: 3.266 sec  <<< FAILURE!
java.lang.AssertionError: Validate input port queue size -1
at com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
at com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramMiniClusterTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
com.datatorrent.stram.StramMiniClusterTest  Time elapsed: 0.002 sec  <<< ERROR!
java.io.FileNotFoundException: /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml (No such file or directory)
at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramLocalClusterTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
testAppPath(com.datatorrent.stram.StramLocalClusterTest)  Time elapsed: 0.138 sec  <<< FAILURE!
java.lang.AssertionError: checkpoint path exists
at com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)

testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)  Time elapsed: 5.191 sec  <<< FAILURE!
java.lang.AssertionError: ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out exists
at com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.client.AppPackageTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
testAppPackage(com.datatorrent.stram.client.AppPackageTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.AssertionError: Number of applications expected:<2> but was:<0>
at com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)

testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)  Time elapsed: 0.001 sec  <<< FAILURE!
java.lang.AssertionError: Should consist of an app called MyFirstApplication
at com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)


-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.cli.ApexCliTest
-------------------------------------------------------------------------------
Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.104 sec  <<< ERROR!
com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package
at com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)

testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.045 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default>
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)

testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.045 sec  <<< ERROR!
com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package matching "MyFirstApplication"
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)

testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.058 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default>
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)


On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <[hidden email]> wrote:
Thanks, Thomas!  I have implemented the fix you suggested (@FieldSerializer.BInd) and also created a JIRA ticket: https://issues.apache.org/jira/browse/APEXCORE-816

On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <[hidden email]> wrote:
Hi,

It is not necessary to use a different stream codec when you have control over the class that is causing the error. You can annotate a different field serializer instead, using the @FieldSerializer.Bind annotation. Here is an example:


It would be good to upgrade Kryo to 4.x in apex-core master. Would you like to create a JIRA for it? (http://apex.apache.org/contributing.html)

Thanks,
Thomas





On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <[hidden email]> wrote:
I am writing an application that includes a custom object.  One of the fields is of type Instant and seems to not agree with the version of Kryo that is used in Apex (see exception below).  I now that kryo 4.X supports serializing Instant fields, but for the life of me, cannot seem to find a way to either get Apex to use a newer version of Kryo.  Can anyone help?  Perhaps setting a different streamcodec that uses the newer version of kryo would work, but I cannot seem to find an example of how to set a different streamcodec despite references to the two (Kryo and Java) that ship with Apex.

ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - Catastrophic Error: Execution halted due to Kryo exception!
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): java.time.Instant
Serialization trace:
systemTime (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at com.datatorrent.stram.codec.DefaultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)


--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group






--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group





--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group


Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Pramod Immaneni-2
Aaron,

The question might be best served on the [hidden email] mailing list as you are starting to make changes to the sources. Nothing obvious jumps out to be, you don't need hadoop on your system to build the sources successfully with all the tests passing. The test failures you are seeing may or may not be related to the kryo change. What happens when you try to build the original sources with changing the Kryo version? What is your environment, mac, linux?

On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <[hidden email]> wrote:
I know it has been a little while since I brought this up...I pulled down the 3.7.1-RELEASE branch from GitHub in order to test out the change to Kryo v. 4.0.2 and encountered no problems while building, however, when I try to do mvn install, there are several failed tests.  I am hoping, that other than the obvious (to me, anyway) errors such as not finding yarn-site.xml on my laptop which does not have hadoop installed, someone may be able to see what is going on?

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramRecoveryTest
-------------------------------------------------------------------------------
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest)  Time elapsed: 0.115 sec  <<< FAILURE!
java.lang.AssertionError: flush count expected:<1> but was:<2>
at com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)


-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.engine.StatsTest
-------------------------------------------------------------------------------
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)  Time elapsed: 3.266 sec  <<< FAILURE!
java.lang.AssertionError: Validate input port queue size -1
at com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
at com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramMiniClusterTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
com.datatorrent.stram.StramMiniClusterTest  Time elapsed: 0.002 sec  <<< ERROR!
java.io.FileNotFoundException: /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml (No such file or directory)
at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramLocalClusterTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
testAppPath(com.datatorrent.stram.StramLocalClusterTest)  Time elapsed: 0.138 sec  <<< FAILURE!
java.lang.AssertionError: checkpoint path exists
at com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)

testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)  Time elapsed: 5.191 sec  <<< FAILURE!
java.lang.AssertionError: ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out exists
at com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.client.AppPackageTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
testAppPackage(com.datatorrent.stram.client.AppPackageTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.AssertionError: Number of applications expected:<2> but was:<0>
at com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)

testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)  Time elapsed: 0.001 sec  <<< FAILURE!
java.lang.AssertionError: Should consist of an app called MyFirstApplication
at com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)


-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.cli.ApexCliTest
-------------------------------------------------------------------------------
Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.104 sec  <<< ERROR!
com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package
at com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)

testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.045 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default>
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)

testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.045 sec  <<< ERROR!
com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package matching "MyFirstApplication"
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)

testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.058 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default>
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)


On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <[hidden email]> wrote:
Thanks, Thomas!  I have implemented the fix you suggested (@FieldSerializer.BInd) and also created a JIRA ticket: https://issues.apache.org/jira/browse/APEXCORE-816

On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <[hidden email]> wrote:
Hi,

It is not necessary to use a different stream codec when you have control over the class that is causing the error. You can annotate a different field serializer instead, using the @FieldSerializer.Bind annotation. Here is an example:


It would be good to upgrade Kryo to 4.x in apex-core master. Would you like to create a JIRA for it? (http://apex.apache.org/contributing.html)

Thanks,
Thomas





On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <[hidden email]> wrote:
I am writing an application that includes a custom object.  One of the fields is of type Instant and seems to not agree with the version of Kryo that is used in Apex (see exception below).  I now that kryo 4.X supports serializing Instant fields, but for the life of me, cannot seem to find a way to either get Apex to use a newer version of Kryo.  Can anyone help?  Perhaps setting a different streamcodec that uses the newer version of kryo would work, but I cannot seem to find an example of how to set a different streamcodec despite references to the two (Kryo and Java) that ship with Apex.

ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - Catastrophic Error: Execution halted due to Kryo exception!
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): java.time.Instant
Serialization trace:
systemTime (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at com.datatorrent.stram.codec.DefaultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)


--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group






--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group





--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group


Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Aaron Bossert
Ah, you make a good point about the dev list...I am using IntelliJ on a Mac.  When I tried building without any changes, I also encountered failures.  I can re-run the non-modified sources to do a diff and see if the errors were different...

Sent from my iPhone

On Jun 11, 2018, at 15:11, Pramod Immaneni <[hidden email]> wrote:

Aaron,

The question might be best served on the [hidden email] mailing list as you are starting to make changes to the sources. Nothing obvious jumps out to be, you don't need hadoop on your system to build the sources successfully with all the tests passing. The test failures you are seeing may or may not be related to the kryo change. What happens when you try to build the original sources with changing the Kryo version? What is your environment, mac, linux?

On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <[hidden email]> wrote:
I know it has been a little while since I brought this up...I pulled down the 3.7.1-RELEASE branch from GitHub in order to test out the change to Kryo v. 4.0.2 and encountered no problems while building, however, when I try to do mvn install, there are several failed tests.  I am hoping, that other than the obvious (to me, anyway) errors such as not finding yarn-site.xml on my laptop which does not have hadoop installed, someone may be able to see what is going on?

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramRecoveryTest
-------------------------------------------------------------------------------
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest)  Time elapsed: 0.115 sec  <<< FAILURE!
java.lang.AssertionError: flush count expected:<1> but was:<2>
at com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)


-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.engine.StatsTest
-------------------------------------------------------------------------------
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)  Time elapsed: 3.266 sec  <<< FAILURE!
java.lang.AssertionError: Validate input port queue size -1
at com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
at com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramMiniClusterTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
com.datatorrent.stram.StramMiniClusterTest  Time elapsed: 0.002 sec  <<< ERROR!
java.io.FileNotFoundException: /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml (No such file or directory)
at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramLocalClusterTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
testAppPath(com.datatorrent.stram.StramLocalClusterTest)  Time elapsed: 0.138 sec  <<< FAILURE!
java.lang.AssertionError: checkpoint path exists
at com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)

testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)  Time elapsed: 5.191 sec  <<< FAILURE!
java.lang.AssertionError: ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out exists
at com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.client.AppPackageTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
testAppPackage(com.datatorrent.stram.client.AppPackageTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.AssertionError: Number of applications expected:<2> but was:<0>
at com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)

testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)  Time elapsed: 0.001 sec  <<< FAILURE!
java.lang.AssertionError: Should consist of an app called MyFirstApplication
at com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)


-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.cli.ApexCliTest
-------------------------------------------------------------------------------
Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.104 sec  <<< ERROR!
com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package
at com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)

testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.045 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default>
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)

testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.045 sec  <<< ERROR!
com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package matching "MyFirstApplication"
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)

testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.058 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default>
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)


On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <[hidden email]> wrote:
Thanks, Thomas!  I have implemented the fix you suggested (@FieldSerializer.BInd) and also created a JIRA ticket: https://issues.apache.org/jira/browse/APEXCORE-816

On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <[hidden email]> wrote:
Hi,

It is not necessary to use a different stream codec when you have control over the class that is causing the error. You can annotate a different field serializer instead, using the @FieldSerializer.Bind annotation. Here is an example:


It would be good to upgrade Kryo to 4.x in apex-core master. Would you like to create a JIRA for it? (http://apex.apache.org/contributing.html)

Thanks,
Thomas





On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <[hidden email]> wrote:
I am writing an application that includes a custom object.  One of the fields is of type Instant and seems to not agree with the version of Kryo that is used in Apex (see exception below).  I now that kryo 4.X supports serializing Instant fields, but for the life of me, cannot seem to find a way to either get Apex to use a newer version of Kryo.  Can anyone help?  Perhaps setting a different streamcodec that uses the newer version of kryo would work, but I cannot seem to find an example of how to set a different streamcodec despite references to the two (Kryo and Java) that ship with Apex.

ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - Catastrophic Error: Execution halted due to Kryo exception!
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): java.time.Instant
Serialization trace:
systemTime (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at com.datatorrent.stram.codec.DefaultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)


--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group






--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group





--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group


Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Aaron Bossert
I sent an e-mail to the dev list...but have not heard any responses.  How active is that list?  Do you know?  Basically, I have a workaround that will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2 or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal.  I have two options left: go it alone and try to update Kryo in the apex-core, or simply change the default serializer.  The hiccup I am running into is that I don't see how (from the documentation) to set the default serializer...perhaps I am missing it.  Can someone point me to where that is in the docs?

On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <[hidden email]> wrote:
Ah, you make a good point about the dev list...I am using IntelliJ on a Mac.  When I tried building without any changes, I also encountered failures.  I can re-run the non-modified sources to do a diff and see if the errors were different...

Sent from my iPhone

On Jun 11, 2018, at 15:11, Pramod Immaneni <[hidden email]> wrote:

Aaron,

The question might be best served on the [hidden email] mailing list as you are starting to make changes to the sources. Nothing obvious jumps out to be, you don't need hadoop on your system to build the sources successfully with all the tests passing. The test failures you are seeing may or may not be related to the kryo change. What happens when you try to build the original sources with changing the Kryo version? What is your environment, mac, linux?

On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <[hidden email]> wrote:
I know it has been a little while since I brought this up...I pulled down the 3.7.1-RELEASE branch from GitHub in order to test out the change to Kryo v. 4.0.2 and encountered no problems while building, however, when I try to do mvn install, there are several failed tests.  I am hoping, that other than the obvious (to me, anyway) errors such as not finding yarn-site.xml on my laptop which does not have hadoop installed, someone may be able to see what is going on?

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramRecoveryTest
-------------------------------------------------------------------------------
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest)  Time elapsed: 0.115 sec  <<< FAILURE!
java.lang.AssertionError: flush count expected:<1> but was:<2>
at com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)


-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.engine.StatsTest
-------------------------------------------------------------------------------
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)  Time elapsed: 3.266 sec  <<< FAILURE!
java.lang.AssertionError: Validate input port queue size -1
at com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
at com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramMiniClusterTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
com.datatorrent.stram.StramMiniClusterTest  Time elapsed: 0.002 sec  <<< ERROR!
java.io.FileNotFoundException: /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml (No such file or directory)
at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.StramLocalClusterTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
testAppPath(com.datatorrent.stram.StramLocalClusterTest)  Time elapsed: 0.138 sec  <<< FAILURE!
java.lang.AssertionError: checkpoint path exists
at com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)

testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)  Time elapsed: 5.191 sec  <<< FAILURE!
java.lang.AssertionError: ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out exists
at com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)

-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.client.AppPackageTest
-------------------------------------------------------------------------------
Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
testAppPackage(com.datatorrent.stram.client.AppPackageTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.AssertionError: Number of applications expected:<2> but was:<0>
at com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)

testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)  Time elapsed: 0.001 sec  <<< FAILURE!
java.lang.AssertionError: Should consist of an app called MyFirstApplication
at com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)


-------------------------------------------------------------------------------
Test set: com.datatorrent.stram.cli.ApexCliTest
-------------------------------------------------------------------------------
Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.104 sec  <<< ERROR!
com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package
at com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)

testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.045 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default>
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)

testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.045 sec  <<< ERROR!
com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package matching "MyFirstApplication"
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)

testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)  Time elapsed: 0.058 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default>
at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)


On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <[hidden email]> wrote:
Thanks, Thomas!  I have implemented the fix you suggested (@FieldSerializer.BInd) and also created a JIRA ticket: https://issues.apache.org/jira/browse/APEXCORE-816

On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <[hidden email]> wrote:
Hi,

It is not necessary to use a different stream codec when you have control over the class that is causing the error. You can annotate a different field serializer instead, using the @FieldSerializer.Bind annotation. Here is an example:


It would be good to upgrade Kryo to 4.x in apex-core master. Would you like to create a JIRA for it? (http://apex.apache.org/contributing.html)

Thanks,
Thomas





On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <[hidden email]> wrote:
I am writing an application that includes a custom object.  One of the fields is of type Instant and seems to not agree with the version of Kryo that is used in Apex (see exception below).  I now that kryo 4.X supports serializing Instant fields, but for the life of me, cannot seem to find a way to either get Apex to use a newer version of Kryo.  Can anyone help?  Perhaps setting a different streamcodec that uses the newer version of kryo would work, but I cannot seem to find an example of how to set a different streamcodec despite references to the two (Kryo and Java) that ship with Apex.

ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - Catastrophic Error: Execution halted due to Kryo exception!
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): java.time.Instant
Serialization trace:
systemTime (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at com.datatorrent.stram.codec.DefaultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
at com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)


--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group






--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group





--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group




--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group


Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Pramod Immaneni-2
Hi Aaron,

That would be great if you can pitch in wherever you can. Let us know how we can help. Also, I was going to try the new kryo version later today and see if there are any internal errors that are bubbling up to those test failures but not getting logged.

Thanks

On Tue, Jun 19, 2018 at 7:54 PM Aaron Bossert <[hidden email]> wrote:
P.S.  I am happy to roll up my sleeves, but from my initial attempt, errors
I was seeing were not obviously related to the Kryo version update (though
I am certain they are...at least indirectly because they only show up when
the version is changed), in terms of tracking down where the changes would
be required...I am not really familiar with the inner workings of Apex yet,
though this would definitely be a good excuse to learn...

I also saw a bunch of annoying warnings when building related to use of
deprecated methods/classes within the com.datatorrent.* namespace...I'd be
happy to start tracking some of those down too if it would be helpful...

On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert <[hidden email]> wrote:

> Pramod,
>
> In a nutshell, yes.  I can set a @FieldSerializer to the fields in
> question...the only one I have run into thus far is Instant...but it would
> be great to not need to annotate each field this way...Also, I could see
> this becoming problematic should I encounter more than a couple of fields
> that are not serializable using the older version of Kryo (but presumably
> would work with a more current one)...though admitedly, it would not be
> that big a deal in the interim.  It would be much cleaner to have an update
> to the most current version of Kryo, IMHO...though I understand that is
> something of a lift.  By the way, Kryo 5 just came out yesterday, so if
> there is to be effort expended, it might be good to go with the latest
> version...though, heads up, I tried it out and there seem to be quite a few
> changes that will be needed...new methods, and definitely some replaced
> ones...
>
>
>
> On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
> [hidden email]> wrote:
>
>> Hi Aaron,
>>
>> While we are debugging the test failures on dev, I didn't fully
>> understand the last question in your email. Looks like you mentioned that
>> the workaround for adding the @FieldSerializer to the field is working. Are
>> you looking for an alternative to this workaround and trying to set a
>> default serializer for all fields of type Instant (the field type you
>> mentioned in the earlier email) so that you don't have to set the
>> annotation each time?
>>
>> Thanks
>>
>> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <[hidden email]>
>> wrote:
>>
>>> I sent an e-mail to the dev list...but have not heard any responses.
>>> How active is that list?  Do you know?  Basically, I have a workaround that
>>> will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
>>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal.  I
>>> have two options left: go it alone and try to update Kryo in the apex-core,
>>> or simply change the default serializer.  The hiccup I am running into is
>>> that I don't see how (from the documentation) to set the default
>>> serializer...perhaps I am missing it.  Can someone point me to where that
>>> is in the docs?
>>>
>>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <[hidden email]>
>>> wrote:
>>>
>>>> Ah, you make a good point about the dev list...I am using IntelliJ on a
>>>> Mac.  When I tried building without any changes, I also encountered
>>>> failures.  I can re-run the non-modified sources to do a diff and see if
>>>> the errors were different...
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <[hidden email]>
>>>> wrote:
>>>>
>>>> Aaron,
>>>>
>>>> The question might be best served on the [hidden email] mailing
>>>> list as you are starting to make changes to the sources. Nothing obvious
>>>> jumps out to be, you don't need hadoop on your system to build the sources
>>>> successfully with all the tests passing. The test failures you are seeing
>>>> may or may not be related to the kryo change. What happens when you try to
>>>> build the original sources with changing the Kryo version? What is your
>>>> environment, mac, linux?
>>>>
>>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <[hidden email]>
>>>> wrote:
>>>>
>>>>> I know it has been a little while since I brought this up...I pulled
>>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change
>>>>> to Kryo v. 4.0.2 and encountered no problems while building, however, when
>>>>> I try to do mvn install, there are several failed tests.  I am hoping, that
>>>>> other than the obvious (to me, anyway) errors such as not finding
>>>>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>>>>> may be able to see what is going on?
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest)  Time
>>>>> elapsed: 0.115 sec  <<< FAILURE!
>>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>>>>> at
>>>>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.engine.StatsTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>>>>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>>>>>  Time elapsed: 3.266 sec  <<< FAILURE!
>>>>> java.lang.AssertionError: Validate input port queue size -1
>>>>> at
>>>>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>>>>> at
>>>>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>>>>> com.datatorrent.stram.StramMiniClusterTest  Time elapsed: 0.002 sec
>>>>>  <<< ERROR!
>>>>> java.io.FileNotFoundException:
>>>>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>>>>> (No such file or directory)
>>>>> at
>>>>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest)  Time
>>>>> elapsed: 0.138 sec  <<< FAILURE!
>>>>> java.lang.AssertionError: checkpoint path exists
>>>>> at
>>>>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>>>>
>>>>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>>>>>  Time elapsed: 5.191 sec  <<< FAILURE!
>>>>> java.lang.AssertionError:
>>>>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>>>>> exists
>>>>> at
>>>>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest)  Time
>>>>> elapsed: 0 sec  <<< FAILURE!
>>>>> java.lang.AssertionError: Number of applications expected:<2> but
>>>>> was:<0>
>>>>> at
>>>>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>>>>
>>>>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>>>>>  Time elapsed: 0.001 sec  <<< FAILURE!
>>>>> java.lang.AssertionError: Should consist of an app called
>>>>> MyFirstApplication
>>>>> at
>>>>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>>  Time elapsed: 0.104 sec  <<< ERROR!
>>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>>> Application Package
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>>>>
>>>>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>>>>>  Time elapsed: 0.045 sec  <<< FAILURE!
>>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>>> was:<[package]-default>
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>>>>
>>>>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>>  Time elapsed: 0.045 sec  <<< ERROR!
>>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>>> Application Package matching "MyFirstApplication"
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>>>>
>>>>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>>  Time elapsed: 0.058 sec  <<< FAILURE!
>>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>>> was:<[package]-default>
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>>>>
>>>>>
>>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> Thanks, Thomas!  I have implemented the fix you suggested
>>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>>>>
>>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <[hidden email]> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> It is not necessary to use a different stream codec when you have
>>>>>>> control over the class that is causing the error. You can annotate a
>>>>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>>>>> annotation. Here is an example:
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>>>>
>>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
>>>>>>> you like to create a JIRA for it? (
>>>>>>> http://apex.apache.org/contributing.html)
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
>>>>>>> [hidden email]> wrote:
>>>>>>>
>>>>>>>> I am writing an application that includes a custom object.  One of
>>>>>>>> the fields is of type Instant and seems to not agree with the version of
>>>>>>>> Kryo that is used in Apex (see exception below).  I now that kryo 4.X
>>>>>>>> supports serializing Instant fields, but for the life of me, cannot seem to
>>>>>>>> find a way to either get Apex to use a newer version of Kryo.  Can anyone
>>>>>>>> help?  Perhaps setting a different streamcodec that uses the newer version
>>>>>>>> of kryo would work, but I cannot seem to find an example of how to set a
>>>>>>>> different streamcodec despite references to the two (Kryo and Java) that
>>>>>>>> ship with Apex.
>>>>>>>>
>>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>>>>> (missing no-arg constructor): java.time.Instant
>>>>>>>> Serialization trace:
>>>>>>>> systemTime
>>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>>>>> at com.datatorrent.stram.codec.De
>>>>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> M. Aaron Bossert
>>>>>>>> (571) 242-4021
>>>>>>>> Punch Cyber Analytics Group
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> M. Aaron Bossert
>>>>>> (571) 242-4021
>>>>>> Punch Cyber Analytics Group
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (571) 242-4021
>>>>> Punch Cyber Analytics Group
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>

--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group