[jira] [Commented] (APEXMALHAR-2034) Avro File Input Operator

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
Report Content as Inappropriate

[jira] [Commented] (APEXMALHAR-2034) Avro File Input Operator

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122764#comment-16122764 ]

Thomas Weise commented on APEXMALHAR-2034:

I would recommend to review the design of this overall and make sure the Avro spec is properly supported (including logical types from 1.8). Also, Avro is a serialization framework that by itself supports multiple wire representations. I'm not sure why GenericRecord would be exposed to a downstream operator. It seems that an abstract class should provide the GenericRecord and leave it to a derived class to provide a result.

> Avro File Input Operator
> ------------------------
>                 Key: APEXMALHAR-2034
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2034
>             Project: Apache Apex Malhar
>          Issue Type: New Feature
>            Reporter: devendra tagare
>            Assignee: Saumya Mohan
> This operator would extend the AbstractFileInputOperator to read Avro Container files.
> Input would be an Avro Container File.
> Output would be a GenericRecord.
> There would be 2 additional optional ports,
> 1.FilePort - for completed files.
> 2.FailedRecordsPort - this will capture fileName,Offset & error message as a string for handling by a downstream operator.
> This operator can be used in isolation or with the AvroToPojo operator to read an Avro record and convert it to a POJO.
> ---------------------------------------------------------------------------------------------------------------------
> This JIRA is used to create a Module on top of AvroFileInputOperator and AvroToPojo operator. The stream between the two operators will be set to CONTAINER_LOCAL which is required as Avro objects are not serialized by Kryo. This will help users to directly use the module which has the locality set to CONTAINER_LOCAL.

This message was sent by Atlassian JIRA