Re: Kryo version and default serializer

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Pramod Immaneni-3
Hi Aaron,

While we are debugging the test failures on dev, I didn't fully understand
the last question in your email. Looks like you mentioned that the
workaround for adding the @FieldSerializer to the field is working. Are you
looking for an alternative to this workaround and trying to set a default
serializer for all fields of type Instant (the field type you mentioned in
the earlier email) so that you don't have to set the annotation each time?

Thanks

On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <[hidden email]> wrote:

> I sent an e-mail to the dev list...but have not heard any responses.  How
> active is that list?  Do you know?  Basically, I have a workaround that
> will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal.  I
> have two options left: go it alone and try to update Kryo in the apex-core,
> or simply change the default serializer.  The hiccup I am running into is
> that I don't see how (from the documentation) to set the default
> serializer...perhaps I am missing it.  Can someone point me to where that
> is in the docs?
>
> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <[hidden email]>
> wrote:
>
>> Ah, you make a good point about the dev list...I am using IntelliJ on a
>> Mac.  When I tried building without any changes, I also encountered
>> failures.  I can re-run the non-modified sources to do a diff and see if
>> the errors were different...
>>
>> Sent from my iPhone
>>
>> On Jun 11, 2018, at 15:11, Pramod Immaneni <[hidden email]>
>> wrote:
>>
>> Aaron,
>>
>> The question might be best served on the [hidden email] mailing
>> list as you are starting to make changes to the sources. Nothing obvious
>> jumps out to be, you don't need hadoop on your system to build the sources
>> successfully with all the tests passing. The test failures you are seeing
>> may or may not be related to the kryo change. What happens when you try to
>> build the original sources with changing the Kryo version? What is your
>> environment, mac, linux?
>>
>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <[hidden email]>
>> wrote:
>>
>>> I know it has been a little while since I brought this up...I pulled
>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change
>>> to Kryo v. 4.0.2 and encountered no problems while building, however, when
>>> I try to do mvn install, there are several failed tests.  I am hoping, that
>>> other than the obvious (to me, anyway) errors such as not finding
>>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>>> may be able to see what is going on?
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.StramRecoveryTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest)  Time
>>> elapsed: 0.115 sec  <<< FAILURE!
>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>>> at
>>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>>
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.engine.StatsTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>>>  Time elapsed: 3.266 sec  <<< FAILURE!
>>> java.lang.AssertionError: Validate input port queue size -1
>>> at
>>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>>> at
>>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>>> com.datatorrent.stram.StramMiniClusterTest  Time elapsed: 0.002 sec  <<<
>>> ERROR!
>>> java.io.FileNotFoundException:
>>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>>> (No such file or directory)
>>> at
>>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest)  Time elapsed:
>>> 0.138 sec  <<< FAILURE!
>>> java.lang.AssertionError: checkpoint path exists
>>> at
>>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>>
>>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>>>  Time elapsed: 5.191 sec  <<< FAILURE!
>>> java.lang.AssertionError:
>>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>>> exists
>>> at
>>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.client.AppPackageTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest)  Time
>>> elapsed: 0 sec  <<< FAILURE!
>>> java.lang.AssertionError: Number of applications expected:<2> but was:<0>
>>> at
>>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>>
>>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>>>  Time elapsed: 0.001 sec  <<< FAILURE!
>>> java.lang.AssertionError: Should consist of an app called
>>> MyFirstApplication
>>> at
>>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>>
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>  Time elapsed: 0.104 sec  <<< ERROR!
>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>> Application Package
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>>
>>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>>>  Time elapsed: 0.045 sec  <<< FAILURE!
>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>> was:<[package]-default>
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>>
>>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>  Time elapsed: 0.045 sec  <<< ERROR!
>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>> Application Package matching "MyFirstApplication"
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>>
>>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>  Time elapsed: 0.058 sec  <<< FAILURE!
>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>> was:<[package]-default>
>>> at
>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>>
>>>
>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <[hidden email]>
>>> wrote:
>>>
>>>> Thanks, Thomas!  I have implemented the fix you suggested
>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>>
>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <[hidden email]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> It is not necessary to use a different stream codec when you have
>>>>> control over the class that is causing the error. You can annotate a
>>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>>> annotation. Here is an example:
>>>>>
>>>>>
>>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>>
>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you
>>>>> like to create a JIRA for it? (
>>>>> http://apex.apache.org/contributing.html)
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> I am writing an application that includes a custom object.  One of
>>>>>> the fields is of type Instant and seems to not agree with the version of
>>>>>> Kryo that is used in Apex (see exception below).  I now that kryo 4.X
>>>>>> supports serializing Instant fields, but for the life of me, cannot seem to
>>>>>> find a way to either get Apex to use a newer version of Kryo.  Can anyone
>>>>>> help?  Perhaps setting a different streamcodec that uses the newer version
>>>>>> of kryo would work, but I cannot seem to find an example of how to set a
>>>>>> different streamcodec despite references to the two (Kryo and Java) that
>>>>>> ship with Apex.
>>>>>>
>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>>> (missing no-arg constructor): java.time.Instant
>>>>>> Serialization trace:
>>>>>> systemTime
>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>>> at
>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>>> at com.datatorrent.stram.codec.De
>>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>>> at
>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>>> at
>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>>> at
>>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> M. Aaron Bossert
>>>>>> (571) 242-4021
>>>>>> Punch Cyber Analytics Group
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> M. Aaron Bossert
>>>> (571) 242-4021
>>>> Punch Cyber Analytics Group
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Aaron Bossert
Pramod,

In a nutshell, yes.  I can set a @FieldSerializer to the fields in
question...the only one I have run into thus far is Instant...but it would
be great to not need to annotate each field this way...Also, I could see
this becoming problematic should I encounter more than a couple of fields
that are not serializable using the older version of Kryo (but presumably
would work with a more current one)...though admitedly, it would not be
that big a deal in the interim.  It would be much cleaner to have an update
to the most current version of Kryo, IMHO...though I understand that is
something of a lift.  By the way, Kryo 5 just came out yesterday, so if
there is to be effort expended, it might be good to go with the latest
version...though, heads up, I tried it out and there seem to be quite a few
changes that will be needed...new methods, and definitely some replaced
ones...



On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <[hidden email]>
wrote:

> Hi Aaron,
>
> While we are debugging the test failures on dev, I didn't fully understand
> the last question in your email. Looks like you mentioned that the
> workaround for adding the @FieldSerializer to the field is working. Are you
> looking for an alternative to this workaround and trying to set a default
> serializer for all fields of type Instant (the field type you mentioned in
> the earlier email) so that you don't have to set the annotation each time?
>
> Thanks
>
> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <[hidden email]>
> wrote:
>
>> I sent an e-mail to the dev list...but have not heard any responses.  How
>> active is that list?  Do you know?  Basically, I have a workaround that
>> will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal.  I
>> have two options left: go it alone and try to update Kryo in the apex-core,
>> or simply change the default serializer.  The hiccup I am running into is
>> that I don't see how (from the documentation) to set the default
>> serializer...perhaps I am missing it.  Can someone point me to where that
>> is in the docs?
>>
>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <[hidden email]>
>> wrote:
>>
>>> Ah, you make a good point about the dev list...I am using IntelliJ on a
>>> Mac.  When I tried building without any changes, I also encountered
>>> failures.  I can re-run the non-modified sources to do a diff and see if
>>> the errors were different...
>>>
>>> Sent from my iPhone
>>>
>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <[hidden email]>
>>> wrote:
>>>
>>> Aaron,
>>>
>>> The question might be best served on the [hidden email] mailing
>>> list as you are starting to make changes to the sources. Nothing obvious
>>> jumps out to be, you don't need hadoop on your system to build the sources
>>> successfully with all the tests passing. The test failures you are seeing
>>> may or may not be related to the kryo change. What happens when you try to
>>> build the original sources with changing the Kryo version? What is your
>>> environment, mac, linux?
>>>
>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <[hidden email]>
>>> wrote:
>>>
>>>> I know it has been a little while since I brought this up...I pulled
>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change
>>>> to Kryo v. 4.0.2 and encountered no problems while building, however, when
>>>> I try to do mvn install, there are several failed tests.  I am hoping, that
>>>> other than the obvious (to me, anyway) errors such as not finding
>>>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>>>> may be able to see what is going on?
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest)  Time
>>>> elapsed: 0.115 sec  <<< FAILURE!
>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>>>> at
>>>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.engine.StatsTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>>>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>>>>  Time elapsed: 3.266 sec  <<< FAILURE!
>>>> java.lang.AssertionError: Validate input port queue size -1
>>>> at
>>>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>>>> at
>>>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>>>> com.datatorrent.stram.StramMiniClusterTest  Time elapsed: 0.002 sec
>>>>  <<< ERROR!
>>>> java.io.FileNotFoundException:
>>>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>>>> (No such file or directory)
>>>> at
>>>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest)  Time elapsed:
>>>> 0.138 sec  <<< FAILURE!
>>>> java.lang.AssertionError: checkpoint path exists
>>>> at
>>>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>>>
>>>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>>>>  Time elapsed: 5.191 sec  <<< FAILURE!
>>>> java.lang.AssertionError:
>>>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>>>> exists
>>>> at
>>>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest)  Time
>>>> elapsed: 0 sec  <<< FAILURE!
>>>> java.lang.AssertionError: Number of applications expected:<2> but
>>>> was:<0>
>>>> at
>>>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>>>
>>>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>>>>  Time elapsed: 0.001 sec  <<< FAILURE!
>>>> java.lang.AssertionError: Should consist of an app called
>>>> MyFirstApplication
>>>> at
>>>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>  Time elapsed: 0.104 sec  <<< ERROR!
>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>> Application Package
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>>>
>>>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>>>>  Time elapsed: 0.045 sec  <<< FAILURE!
>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>> was:<[package]-default>
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>>>
>>>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>  Time elapsed: 0.045 sec  <<< ERROR!
>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>> Application Package matching "MyFirstApplication"
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>>>
>>>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>  Time elapsed: 0.058 sec  <<< FAILURE!
>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>> was:<[package]-default>
>>>> at
>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>>>
>>>>
>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <[hidden email]>
>>>> wrote:
>>>>
>>>>> Thanks, Thomas!  I have implemented the fix you suggested
>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>>>
>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <[hidden email]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> It is not necessary to use a different stream codec when you have
>>>>>> control over the class that is causing the error. You can annotate a
>>>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>>>> annotation. Here is an example:
>>>>>>
>>>>>>
>>>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>>>
>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
>>>>>> you like to create a JIRA for it? (
>>>>>> http://apex.apache.org/contributing.html)
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <[hidden email]
>>>>>> > wrote:
>>>>>>
>>>>>>> I am writing an application that includes a custom object.  One of
>>>>>>> the fields is of type Instant and seems to not agree with the version of
>>>>>>> Kryo that is used in Apex (see exception below).  I now that kryo 4.X
>>>>>>> supports serializing Instant fields, but for the life of me, cannot seem to
>>>>>>> find a way to either get Apex to use a newer version of Kryo.  Can anyone
>>>>>>> help?  Perhaps setting a different streamcodec that uses the newer version
>>>>>>> of kryo would work, but I cannot seem to find an example of how to set a
>>>>>>> different streamcodec despite references to the two (Kryo and Java) that
>>>>>>> ship with Apex.
>>>>>>>
>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>>>> (missing no-arg constructor): java.time.Instant
>>>>>>> Serialization trace:
>>>>>>> systemTime
>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>>>> at
>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>>>> at com.datatorrent.stram.codec.De
>>>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>>>> at
>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>>>> at
>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>>>> at
>>>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> M. Aaron Bossert
>>>>>>> (571) 242-4021
>>>>>>> Punch Cyber Analytics Group
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (571) 242-4021
>>>>> Punch Cyber Analytics Group
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> M. Aaron Bossert
>>>> (571) 242-4021
>>>> Punch Cyber Analytics Group
>>>>
>>>>
>>>>
>>
>> --
>>
>> M. Aaron Bossert
>> (571) 242-4021
>> Punch Cyber Analytics Group
>>
>>
>>

--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Aaron Bossert
P.S.  I am happy to roll up my sleeves, but from my initial attempt, errors
I was seeing were not obviously related to the Kryo version update (though
I am certain they are...at least indirectly because they only show up when
the version is changed), in terms of tracking down where the changes would
be required...I am not really familiar with the inner workings of Apex yet,
though this would definitely be a good excuse to learn...

I also saw a bunch of annoying warnings when building related to use of
deprecated methods/classes within the com.datatorrent.* namespace...I'd be
happy to start tracking some of those down too if it would be helpful...

On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert <[hidden email]> wrote:

> Pramod,
>
> In a nutshell, yes.  I can set a @FieldSerializer to the fields in
> question...the only one I have run into thus far is Instant...but it would
> be great to not need to annotate each field this way...Also, I could see
> this becoming problematic should I encounter more than a couple of fields
> that are not serializable using the older version of Kryo (but presumably
> would work with a more current one)...though admitedly, it would not be
> that big a deal in the interim.  It would be much cleaner to have an update
> to the most current version of Kryo, IMHO...though I understand that is
> something of a lift.  By the way, Kryo 5 just came out yesterday, so if
> there is to be effort expended, it might be good to go with the latest
> version...though, heads up, I tried it out and there seem to be quite a few
> changes that will be needed...new methods, and definitely some replaced
> ones...
>
>
>
> On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
> [hidden email]> wrote:
>
>> Hi Aaron,
>>
>> While we are debugging the test failures on dev, I didn't fully
>> understand the last question in your email. Looks like you mentioned that
>> the workaround for adding the @FieldSerializer to the field is working. Are
>> you looking for an alternative to this workaround and trying to set a
>> default serializer for all fields of type Instant (the field type you
>> mentioned in the earlier email) so that you don't have to set the
>> annotation each time?
>>
>> Thanks
>>
>> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <[hidden email]>
>> wrote:
>>
>>> I sent an e-mail to the dev list...but have not heard any responses.
>>> How active is that list?  Do you know?  Basically, I have a workaround that
>>> will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2
>>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal.  I
>>> have two options left: go it alone and try to update Kryo in the apex-core,
>>> or simply change the default serializer.  The hiccup I am running into is
>>> that I don't see how (from the documentation) to set the default
>>> serializer...perhaps I am missing it.  Can someone point me to where that
>>> is in the docs?
>>>
>>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <[hidden email]>
>>> wrote:
>>>
>>>> Ah, you make a good point about the dev list...I am using IntelliJ on a
>>>> Mac.  When I tried building without any changes, I also encountered
>>>> failures.  I can re-run the non-modified sources to do a diff and see if
>>>> the errors were different...
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <[hidden email]>
>>>> wrote:
>>>>
>>>> Aaron,
>>>>
>>>> The question might be best served on the [hidden email] mailing
>>>> list as you are starting to make changes to the sources. Nothing obvious
>>>> jumps out to be, you don't need hadoop on your system to build the sources
>>>> successfully with all the tests passing. The test failures you are seeing
>>>> may or may not be related to the kryo change. What happens when you try to
>>>> build the original sources with changing the Kryo version? What is your
>>>> environment, mac, linux?
>>>>
>>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <[hidden email]>
>>>> wrote:
>>>>
>>>>> I know it has been a little while since I brought this up...I pulled
>>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change
>>>>> to Kryo v. 4.0.2 and encountered no problems while building, however, when
>>>>> I try to do mvn install, there are several failed tests.  I am hoping, that
>>>>> other than the obvious (to me, anyway) errors such as not finding
>>>>> yarn-site.xml on my laptop which does not have hadoop installed, someone
>>>>> may be able to see what is going on?
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest)  Time
>>>>> elapsed: 0.115 sec  <<< FAILURE!
>>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>>>>> at
>>>>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.engine.StatsTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>>>>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>>>>>  Time elapsed: 3.266 sec  <<< FAILURE!
>>>>> java.lang.AssertionError: Validate input port queue size -1
>>>>> at
>>>>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>>>>> at
>>>>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>>>>> com.datatorrent.stram.StramMiniClusterTest  Time elapsed: 0.002 sec
>>>>>  <<< ERROR!
>>>>> java.io.FileNotFoundException:
>>>>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>>>>> (No such file or directory)
>>>>> at
>>>>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest)  Time
>>>>> elapsed: 0.138 sec  <<< FAILURE!
>>>>> java.lang.AssertionError: checkpoint path exists
>>>>> at
>>>>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>>>>
>>>>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>>>>>  Time elapsed: 5.191 sec  <<< FAILURE!
>>>>> java.lang.AssertionError:
>>>>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>>>>> exists
>>>>> at
>>>>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest)  Time
>>>>> elapsed: 0 sec  <<< FAILURE!
>>>>> java.lang.AssertionError: Number of applications expected:<2> but
>>>>> was:<0>
>>>>> at
>>>>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>>>>
>>>>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>>>>>  Time elapsed: 0.001 sec  <<< FAILURE!
>>>>> java.lang.AssertionError: Should consist of an app called
>>>>> MyFirstApplication
>>>>> at
>>>>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>>>>
>>>>> -------------------------------------------------------------------------------
>>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>>  Time elapsed: 0.104 sec  <<< ERROR!
>>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>>> Application Package
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>>>>
>>>>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>>>>>  Time elapsed: 0.045 sec  <<< FAILURE!
>>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>>> was:<[package]-default>
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>>>>
>>>>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>>  Time elapsed: 0.045 sec  <<< ERROR!
>>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>>>>> Application Package matching "MyFirstApplication"
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>>>>
>>>>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>>  Time elapsed: 0.058 sec  <<< FAILURE!
>>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>>> was:<[package]-default>
>>>>> at
>>>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>>>>
>>>>>
>>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> Thanks, Thomas!  I have implemented the fix you suggested
>>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>>>>
>>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <[hidden email]> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> It is not necessary to use a different stream codec when you have
>>>>>>> control over the class that is causing the error. You can annotate a
>>>>>>> different field serializer instead, using the @FieldSerializer.Bind
>>>>>>> annotation. Here is an example:
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>>>>
>>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
>>>>>>> you like to create a JIRA for it? (
>>>>>>> http://apex.apache.org/contributing.html)
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
>>>>>>> [hidden email]> wrote:
>>>>>>>
>>>>>>>> I am writing an application that includes a custom object.  One of
>>>>>>>> the fields is of type Instant and seems to not agree with the version of
>>>>>>>> Kryo that is used in Apex (see exception below).  I now that kryo 4.X
>>>>>>>> supports serializing Instant fields, but for the life of me, cannot seem to
>>>>>>>> find a way to either get Apex to use a newer version of Kryo.  Can anyone
>>>>>>>> help?  Perhaps setting a different streamcodec that uses the newer version
>>>>>>>> of kryo would work, but I cannot seem to find an example of how to set a
>>>>>>>> different streamcodec despite references to the two (Kryo and Java) that
>>>>>>>> ship with Apex.
>>>>>>>>
>>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>>>>>>>> (missing no-arg constructor): java.time.Instant
>>>>>>>> Serialization trace:
>>>>>>>> systemTime
>>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>>>>> at
>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>>>>> at com.datatorrent.stram.codec.De
>>>>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>>>>> at
>>>>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> M. Aaron Bossert
>>>>>>>> (571) 242-4021
>>>>>>>> Punch Cyber Analytics Group
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> M. Aaron Bossert
>>>>>> (571) 242-4021
>>>>>> Punch Cyber Analytics Group
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (571) 242-4021
>>>>> Punch Cyber Analytics Group
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>

--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Aaron Bossert
cool! I'm on it.

On Tue, Jun 19, 2018 at 11:00 PM Pramod Immaneni <[hidden email]>
wrote:

> Hi Aaron,
>
> That would be great if you can pitch in wherever you can. Let us know how
> we can help. Also, I was going to try the new kryo version later today and
> see if there are any internal errors that are bubbling up to those test
> failures but not getting logged.
>
> Thanks
>
> On Tue, Jun 19, 2018 at 7:54 PM Aaron Bossert <[hidden email]>
> wrote:
>
>> P.S.  I am happy to roll up my sleeves, but from my initial attempt,
>> errors
>> I was seeing were not obviously related to the Kryo version update (though
>> I am certain they are...at least indirectly because they only show up when
>> the version is changed), in terms of tracking down where the changes would
>> be required...I am not really familiar with the inner workings of Apex
>> yet,
>> though this would definitely be a good excuse to learn...
>>
>> I also saw a bunch of annoying warnings when building related to use of
>> deprecated methods/classes within the com.datatorrent.* namespace...I'd be
>> happy to start tracking some of those down too if it would be helpful...
>>
>> On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert <[hidden email]>
>> wrote:
>>
>> > Pramod,
>> >
>> > In a nutshell, yes.  I can set a @FieldSerializer to the fields in
>> > question...the only one I have run into thus far is Instant...but it
>> would
>> > be great to not need to annotate each field this way...Also, I could see
>> > this becoming problematic should I encounter more than a couple of
>> fields
>> > that are not serializable using the older version of Kryo (but
>> presumably
>> > would work with a more current one)...though admitedly, it would not be
>> > that big a deal in the interim.  It would be much cleaner to have an
>> update
>> > to the most current version of Kryo, IMHO...though I understand that is
>> > something of a lift.  By the way, Kryo 5 just came out yesterday, so if
>> > there is to be effort expended, it might be good to go with the latest
>> > version...though, heads up, I tried it out and there seem to be quite a
>> few
>> > changes that will be needed...new methods, and definitely some replaced
>> > ones...
>> >
>> >
>> >
>> > On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
>> > [hidden email]> wrote:
>> >
>> >> Hi Aaron,
>> >>
>> >> While we are debugging the test failures on dev, I didn't fully
>> >> understand the last question in your email. Looks like you mentioned
>> that
>> >> the workaround for adding the @FieldSerializer to the field is
>> working. Are
>> >> you looking for an alternative to this workaround and trying to set a
>> >> default serializer for all fields of type Instant (the field type you
>> >> mentioned in the earlier email) so that you don't have to set the
>> >> annotation each time?
>> >>
>> >> Thanks
>> >>
>> >> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <[hidden email]>
>> >> wrote:
>> >>
>> >>> I sent an e-mail to the dev list...but have not heard any responses.
>> >>> How active is that list?  Do you know?  Basically, I have a
>> workaround that
>> >>> will deal with those "things" that Kryo v 2.24.0 will not handle (but
>> 4.0.2
>> >>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not
>> ideal.  I
>> >>> have two options left: go it alone and try to update Kryo in the
>> apex-core,
>> >>> or simply change the default serializer.  The hiccup I am running
>> into is
>> >>> that I don't see how (from the documentation) to set the default
>> >>> serializer...perhaps I am missing it.  Can someone point me to where
>> that
>> >>> is in the docs?
>> >>>
>> >>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <[hidden email]>
>> >>> wrote:
>> >>>
>> >>>> Ah, you make a good point about the dev list...I am using IntelliJ
>> on a
>> >>>> Mac.  When I tried building without any changes, I also encountered
>> >>>> failures.  I can re-run the non-modified sources to do a diff and
>> see if
>> >>>> the errors were different...
>> >>>>
>> >>>> Sent from my iPhone
>> >>>>
>> >>>> On Jun 11, 2018, at 15:11, Pramod Immaneni <
>> [hidden email]>
>> >>>> wrote:
>> >>>>
>> >>>> Aaron,
>> >>>>
>> >>>> The question might be best served on the [hidden email] mailing
>> >>>> list as you are starting to make changes to the sources. Nothing
>> obvious
>> >>>> jumps out to be, you don't need hadoop on your system to build the
>> sources
>> >>>> successfully with all the tests passing. The test failures you are
>> seeing
>> >>>> may or may not be related to the kryo change. What happens when you
>> try to
>> >>>> build the original sources with changing the Kryo version? What is
>> your
>> >>>> environment, mac, linux?
>> >>>>
>> >>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <[hidden email]
>> >
>> >>>> wrote:
>> >>>>
>> >>>>> I know it has been a little while since I brought this up...I pulled
>> >>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the
>> change
>> >>>>> to Kryo v. 4.0.2 and encountered no problems while building,
>> however, when
>> >>>>> I try to do mvn install, there are several failed tests.  I am
>> hoping, that
>> >>>>> other than the obvious (to me, anyway) errors such as not finding
>> >>>>> yarn-site.xml on my laptop which does not have hadoop installed,
>> someone
>> >>>>> may be able to see what is going on?
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
>> 6.372
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest
>> >>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest)  Time
>> >>>>> elapsed: 0.115 sec  <<< FAILURE!
>> >>>>> java.lang.AssertionError: flush count expected:<1> but was:<2>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.engine.StatsTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
>> 21.894
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>> >>>>>
>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>> >>>>>  Time elapsed: 3.266 sec  <<< FAILURE!
>> >>>>> java.lang.AssertionError: Validate input port queue size -1
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
>> 0.002
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest
>> >>>>> com.datatorrent.stram.StramMiniClusterTest  Time elapsed: 0.002 sec
>> >>>>>  <<< ERROR!
>> >>>>> java.io.FileNotFoundException:
>> >>>>>
>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>> >>>>> (No such file or directory)
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
>> 15.081
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest
>> >>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest)  Time
>> >>>>> elapsed: 0.138 sec  <<< FAILURE!
>> >>>>> java.lang.AssertionError: checkpoint path exists
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>> >>>>>
>> >>>>>
>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>> >>>>>  Time elapsed: 5.191 sec  <<< FAILURE!
>> >>>>> java.lang.AssertionError:
>> >>>>>
>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>> >>>>> exists
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
>> 0.105
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest
>> >>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest)  Time
>> >>>>> elapsed: 0 sec  <<< FAILURE!
>> >>>>> java.lang.AssertionError: Number of applications expected:<2> but
>> >>>>> was:<0>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>> >>>>>
>> >>>>>
>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>> >>>>>  Time elapsed: 0.001 sec  <<< FAILURE!
>> >>>>> java.lang.AssertionError: Should consist of an app called
>> >>>>> MyFirstApplication
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------
>> >>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed:
>> 0.938
>> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>> >>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>>  Time elapsed: 0.104 sec  <<< ERROR!
>> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>> >>>>> Application Package
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>> >>>>>
>> >>>>>
>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>>  Time elapsed: 0.045 sec  <<< FAILURE!
>> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>> >>>>> was:<[package]-default>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>> >>>>>
>> >>>>>
>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>>  Time elapsed: 0.045 sec  <<< ERROR!
>> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in
>> >>>>> Application Package matching "MyFirstApplication"
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>> >>>>>
>> >>>>>
>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>> >>>>>  Time elapsed: 0.058 sec  <<< FAILURE!
>> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>> >>>>> was:<[package]-default>
>> >>>>> at
>> >>>>>
>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>> >>>>>
>> >>>>>
>> >>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <
>> [hidden email]>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Thanks, Thomas!  I have implemented the fix you suggested
>> >>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>> >>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>> >>>>>>
>> >>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <[hidden email]>
>> wrote:
>> >>>>>>
>> >>>>>>> Hi,
>> >>>>>>>
>> >>>>>>> It is not necessary to use a different stream codec when you have
>> >>>>>>> control over the class that is causing the error. You can
>> annotate a
>> >>>>>>> different field serializer instead, using the
>> @FieldSerializer.Bind
>> >>>>>>> annotation. Here is an example:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>> >>>>>>>
>> >>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would
>> >>>>>>> you like to create a JIRA for it? (
>> >>>>>>> http://apex.apache.org/contributing.html)
>> >>>>>>>
>> >>>>>>> Thanks,
>> >>>>>>> Thomas
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
>> >>>>>>> [hidden email]> wrote:
>> >>>>>>>
>> >>>>>>>> I am writing an application that includes a custom object.  One
>> of
>> >>>>>>>> the fields is of type Instant and seems to not agree with the
>> version of
>> >>>>>>>> Kryo that is used in Apex (see exception below).  I now that
>> kryo 4.X
>> >>>>>>>> supports serializing Instant fields, but for the life of me,
>> cannot seem to
>> >>>>>>>> find a way to either get Apex to use a newer version of Kryo.
>> Can anyone
>> >>>>>>>> help?  Perhaps setting a different streamcodec that uses the
>> newer version
>> >>>>>>>> of kryo would work, but I cannot seem to find an example of how
>> to set a
>> >>>>>>>> different streamcodec despite references to the two (Kryo and
>> Java) that
>> >>>>>>>> ship with Apex.
>> >>>>>>>>
>> >>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>> >>>>>>>> Catastrophic Error: Execution halted due to Kryo exception!
>> >>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created
>> >>>>>>>> (missing no-arg constructor): java.time.Instant
>> >>>>>>>> Serialization trace:
>> >>>>>>>> systemTime
>> >>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>> >>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>> >>>>>>>> at
>> >>>>>>>>
>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>> >>>>>>>> at
>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>> >>>>>>>> at com.datatorrent.stram.codec.De
>> >>>>>>>>
>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>> >>>>>>>> at
>> >>>>>>>>
>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>>
>> >>>>>>>> M. Aaron Bossert
>> >>>>>>>> (571) 242-4021
>> >>>>>>>> Punch Cyber Analytics Group
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> M. Aaron Bossert
>> >>>>>> (571) 242-4021
>> >>>>>> Punch Cyber Analytics Group
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>>
>> >>>>> M. Aaron Bossert
>> >>>>> (571) 242-4021
>> >>>>> Punch Cyber Analytics Group
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>
>> >>> --
>> >>>
>> >>> M. Aaron Bossert
>> >>> (571) 242-4021
>> >>> Punch Cyber Analytics Group
>> >>>
>> >>>
>> >>>
>> >
>> > --
>> >
>> > M. Aaron Bossert
>> > (571) 242-4021
>> > Punch Cyber Analytics Group
>> >
>> >
>> >
>>
>> --
>>
>> M. Aaron Bossert
>> (571) 242-4021
>> Punch Cyber Analytics Group
>>
>

--

M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
Reply | Threaded
Open this post in threaded view
|

Re: Kryo version and default serializer

Vlad Rozov-2
Hi Aaron,

Nice to see more contributors to the Apex! Welcome.

I'd suggest that we stop this e-mail thread and continue discussion on
the other dev@apex only e-mail thread. Usually users@apex is not a good
choice to discuss dependency upgrades.

Please be patient, majority of Apex and Apache committers contribute at
their free time and there may be delays in response.

Thank you,

Vlad

On 6/19/18 20:02, Aaron Bossert wrote:

> cool! I'm on it.
>
> On Tue, Jun 19, 2018 at 11:00 PM Pramod Immaneni
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Hi Aaron,
>
>     That would be great if you can pitch in wherever you can. Let us
>     know how we can help. Also, I was going to try the
>     new kryo version later today and see if there are any internal
>     errors that are bubbling up to those test failures but not getting
>     logged.
>
>     Thanks
>
>     On Tue, Jun 19, 2018 at 7:54 PM Aaron Bossert
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         P.S.  I am happy to roll up my sleeves, but from my initial
>         attempt, errors
>         I was seeing were not obviously related to the Kryo version
>         update (though
>         I am certain they are...at least indirectly because they only
>         show up when
>         the version is changed), in terms of tracking down where the
>         changes would
>         be required...I am not really familiar with the inner workings
>         of Apex yet,
>         though this would definitely be a good excuse to learn...
>
>         I also saw a bunch of annoying warnings when building related
>         to use of
>         deprecated methods/classes within the com.datatorrent.*
>         namespace...I'd be
>         happy to start tracking some of those down too if it would be
>         helpful...
>
>         On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert
>         <[hidden email] <mailto:[hidden email]>> wrote:
>
>         > Pramod,
>         >
>         > In a nutshell, yes.  I can set a @FieldSerializer to the
>         fields in
>         > question...the only one I have run into thus far is
>         Instant...but it would
>         > be great to not need to annotate each field this way...Also,
>         I could see
>         > this becoming problematic should I encounter more than a
>         couple of fields
>         > that are not serializable using the older version of Kryo
>         (but presumably
>         > would work with a more current one)...though admitedly, it
>         would not be
>         > that big a deal in the interim.  It would be much cleaner to
>         have an update
>         > to the most current version of Kryo, IMHO...though I
>         understand that is
>         > something of a lift.  By the way, Kryo 5 just came out
>         yesterday, so if
>         > there is to be effort expended, it might be good to go with
>         the latest
>         > version...though, heads up, I tried it out and there seem to
>         be quite a few
>         > changes that will be needed...new methods, and definitely
>         some replaced
>         > ones...
>         >
>         >
>         >
>         > On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
>         > [hidden email]
>         <mailto:[hidden email]>> wrote:
>         >
>         >> Hi Aaron,
>         >>
>         >> While we are debugging the test failures on dev, I didn't fully
>         >> understand the last question in your email. Looks like you
>         mentioned that
>         >> the workaround for adding the @FieldSerializer to the field
>         is working. Are
>         >> you looking for an alternative to this workaround and
>         trying to set a
>         >> default serializer for all fields of type Instant (the
>         field type you
>         >> mentioned in the earlier email) so that you don't have to
>         set the
>         >> annotation each time?
>         >>
>         >> Thanks
>         >>
>         >> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert
>         <[hidden email] <mailto:[hidden email]>>
>         >> wrote:
>         >>
>         >>> I sent an e-mail to the dev list...but have not heard any
>         responses.
>         >>> How active is that list?  Do you know? Basically, I have a
>         workaround that
>         >>> will deal with those "things" that Kryo v 2.24.0 will not
>         handle (but 4.0.2
>         >>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this
>         is not ideal.  I
>         >>> have two options left: go it alone and try to update Kryo
>         in the apex-core,
>         >>> or simply change the default serializer.  The hiccup I am
>         running into is
>         >>> that I don't see how (from the documentation) to set the
>         default
>         >>> serializer...perhaps I am missing it.  Can someone point
>         me to where that
>         >>> is in the docs?
>         >>>
>         >>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert
>         <[hidden email] <mailto:[hidden email]>>
>         >>> wrote:
>         >>>
>         >>>> Ah, you make a good point about the dev list...I am using
>         IntelliJ on a
>         >>>> Mac.  When I tried building without any changes, I also
>         encountered
>         >>>> failures.  I can re-run the non-modified sources to do a
>         diff and see if
>         >>>> the errors were different...
>         >>>>
>         >>>> Sent from my iPhone
>         >>>>
>         >>>> On Jun 11, 2018, at 15:11, Pramod Immaneni
>         <[hidden email] <mailto:[hidden email]>>
>         >>>> wrote:
>         >>>>
>         >>>> Aaron,
>         >>>>
>         >>>> The question might be best served on the
>         [hidden email] <mailto:[hidden email]> mailing
>         >>>> list as you are starting to make changes to the sources.
>         Nothing obvious
>         >>>> jumps out to be, you don't need hadoop on your system to
>         build the sources
>         >>>> successfully with all the tests passing. The test
>         failures you are seeing
>         >>>> may or may not be related to the kryo change. What
>         happens when you try to
>         >>>> build the original sources with changing the Kryo
>         version? What is your
>         >>>> environment, mac, linux?
>         >>>>
>         >>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert
>         <[hidden email] <mailto:[hidden email]>>
>         >>>> wrote:
>         >>>>
>         >>>>> I know it has been a little while since I brought this
>         up...I pulled
>         >>>>> down the 3.7.1-RELEASE branch from GitHub in order to
>         test out the change
>         >>>>> to Kryo v. 4.0.2 and encountered no problems while
>         building, however, when
>         >>>>> I try to do mvn install, there are several failed
>         tests.  I am hoping, that
>         >>>>> other than the obvious (to me, anyway) errors such as
>         not finding
>         >>>>> yarn-site.xml on my laptop which does not have hadoop
>         installed, someone
>         >>>>> may be able to see what is going on?
>         >>>>>
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time
>         elapsed: 6.372
>         >>>>> sec <<< FAILURE! - in
>         com.datatorrent.stram.StramRecoveryTest
>         >>>>>
>         testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
>         >>>>> elapsed: 0.115 sec  <<< FAILURE!
>         >>>>> java.lang.AssertionError: flush count expected:<1> but
>         was:<2>
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>         >>>>>
>         >>>>>
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Test set: com.datatorrent.stram.engine.StatsTest
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time
>         elapsed: 21.894
>         >>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>         >>>>>
>         testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>         >>>>>  Time elapsed: 3.266 sec <<< FAILURE!
>         >>>>> java.lang.AssertionError: Validate input port queue size -1
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>         >>>>>
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
>         elapsed: 0.002
>         >>>>> sec <<< FAILURE! - in
>         com.datatorrent.stram.StramMiniClusterTest
>         >>>>> com.datatorrent.stram.StramMiniClusterTest  Time
>         elapsed: 0.002 sec
>         >>>>>  <<< ERROR!
>         >>>>> java.io.FileNotFoundException:
>         >>>>>
>         /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>         >>>>> (No such file or directory)
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>         >>>>>
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time
>         elapsed: 15.081
>         >>>>> sec <<< FAILURE! - in
>         com.datatorrent.stram.StramLocalClusterTest
>         >>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest)
>         Time
>         >>>>> elapsed: 0.138 sec  <<< FAILURE!
>         >>>>> java.lang.AssertionError: checkpoint path exists
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>         >>>>>
>         >>>>>
>         testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>         >>>>>  Time elapsed: 5.191 sec <<< FAILURE!
>         >>>>> java.lang.AssertionError:
>         >>>>>
>         ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>         >>>>> exists
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>         >>>>>
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time
>         elapsed: 0.105
>         >>>>> sec <<< FAILURE! - in
>         com.datatorrent.stram.client.AppPackageTest
>         >>>>>
>         testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>         >>>>> elapsed: 0 sec  <<< FAILURE!
>         >>>>> java.lang.AssertionError: Number of applications
>         expected:<2> but
>         >>>>> was:<0>
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>         >>>>>
>         >>>>>
>         testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>         >>>>>  Time elapsed: 0.001 sec <<< FAILURE!
>         >>>>> java.lang.AssertionError: Should consist of an app called
>         >>>>> MyFirstApplication
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>         >>>>>
>         >>>>>
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>         >>>>>
>         >>>>>
>         -------------------------------------------------------------------------------
>         >>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time
>         elapsed: 0.938
>         >>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>         >>>>>
>         testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>         >>>>>  Time elapsed: 0.104 sec <<< ERROR!
>         >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No
>         applications in
>         >>>>> Application Package
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>         >>>>>
>         >>>>>
>         testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>         >>>>>  Time elapsed: 0.045 sec <<< FAILURE!
>         >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>         >>>>> was:<[package]-default>
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>         >>>>>
>         >>>>>
>         testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>         >>>>>  Time elapsed: 0.045 sec <<< ERROR!
>         >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No
>         applications in
>         >>>>> Application Package matching "MyFirstApplication"
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>         >>>>>
>         >>>>>
>         testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>         >>>>>  Time elapsed: 0.058 sec <<< FAILURE!
>         >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>         >>>>> was:<[package]-default>
>         >>>>> at
>         >>>>>
>         com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>         >>>>>
>         >>>>>
>         >>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert
>         <[hidden email] <mailto:[hidden email]>>
>         >>>>> wrote:
>         >>>>>
>         >>>>>> Thanks, Thomas!  I have implemented the fix you suggested
>         >>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>         >>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>         >>>>>>
>         >>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise
>         <[hidden email] <mailto:[hidden email]>> wrote:
>         >>>>>>
>         >>>>>>> Hi,
>         >>>>>>>
>         >>>>>>> It is not necessary to use a different stream codec
>         when you have
>         >>>>>>> control over the class that is causing the error. You
>         can annotate a
>         >>>>>>> different field serializer instead, using the
>         @FieldSerializer.Bind
>         >>>>>>> annotation. Here is an example:
>         >>>>>>>
>         >>>>>>>
>         >>>>>>>
>         https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>         >>>>>>>
>         >>>>>>> It would be good to upgrade Kryo to 4.x in apex-core
>         master. Would
>         >>>>>>> you like to create a JIRA for it? (
>         >>>>>>> http://apex.apache.org/contributing.html)
>         >>>>>>>
>         >>>>>>> Thanks,
>         >>>>>>> Thomas
>         >>>>>>>
>         >>>>>>>
>         >>>>>>>
>         >>>>>>>
>         >>>>>>>
>         >>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
>         >>>>>>> [hidden email] <mailto:[hidden email]>> wrote:
>         >>>>>>>
>         >>>>>>>> I am writing an application that includes a custom
>         object.  One of
>         >>>>>>>> the fields is of type Instant and seems to not agree
>         with the version of
>         >>>>>>>> Kryo that is used in Apex (see exception below).  I
>         now that kryo 4.X
>         >>>>>>>> supports serializing Instant fields, but for the life
>         of me, cannot seem to
>         >>>>>>>> find a way to either get Apex to use a newer version
>         of Kryo.  Can anyone
>         >>>>>>>> help?  Perhaps setting a different streamcodec that
>         uses the newer version
>         >>>>>>>> of kryo would work, but I cannot seem to find an
>         example of how to set a
>         >>>>>>>> different streamcodec despite references to the two
>         (Kryo and Java) that
>         >>>>>>>> ship with Apex.
>         >>>>>>>>
>         >>>>>>>> ERROR
>         com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>         >>>>>>>> Catastrophic Error: Execution halted due to Kryo
>         exception!
>         >>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot
>         be created
>         >>>>>>>> (missing no-arg constructor): java.time.Instant
>         >>>>>>>> Serialization trace:
>         >>>>>>>> systemTime
>         >>>>>>>>
>         (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>         >>>>>>>> at
>         >>>>>>>>
>         com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>         >>>>>>>> at
>         com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>         >>>>>>>> at
>         com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>         >>>>>>>> at
>         >>>>>>>>
>         com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>         >>>>>>>> at
>         >>>>>>>>
>         com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>         >>>>>>>> at
>         com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>         >>>>>>>> at
>         >>>>>>>>
>         com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>         >>>>>>>> at
>         >>>>>>>>
>         com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>         >>>>>>>> at
>         com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>         >>>>>>>> at com.datatorrent.stram.codec.De
>         <http://com.datatorrent.stram.codec.De>
>         >>>>>>>>
>         faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>         >>>>>>>> at
>         >>>>>>>>
>         com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>         >>>>>>>> at
>         >>>>>>>>
>         com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>         >>>>>>>> at
>         >>>>>>>>
>         com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>         >>>>>>>> at
>         >>>>>>>>
>         com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>         >>>>>>>>
>         >>>>>>>>
>         >>>>>>>> --
>         >>>>>>>>
>         >>>>>>>> M. Aaron Bossert
>         >>>>>>>> (571) 242-4021
>         >>>>>>>> Punch Cyber Analytics Group
>         >>>>>>>>
>         >>>>>>>>
>         >>>>>>>>
>         >>>>>>>
>         >>>>>>
>         >>>>>>
>         >>>>>> --
>         >>>>>>
>         >>>>>> M. Aaron Bossert
>         >>>>>> (571) 242-4021
>         >>>>>> Punch Cyber Analytics Group
>         >>>>>>
>         >>>>>>
>         >>>>>>
>         >>>>>
>         >>>>>
>         >>>>> --
>         >>>>>
>         >>>>> M. Aaron Bossert
>         >>>>> (571) 242-4021
>         >>>>> Punch Cyber Analytics Group
>         >>>>>
>         >>>>>
>         >>>>>
>         >>>
>         >>> --
>         >>>
>         >>> M. Aaron Bossert
>         >>> (571) 242-4021
>         >>> Punch Cyber Analytics Group
>         >>>
>         >>>
>         >>>
>         >
>         > --
>         >
>         > M. Aaron Bossert
>         > (571) 242-4021
>         > Punch Cyber Analytics Group
>         >
>         >
>         >
>
>         --
>
>         M. Aaron Bossert
>         (571) 242-4021
>         Punch Cyber Analytics Group
>
>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>