Proposal for UI component in stram

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Proposal for UI component in stram

chinmay
Administrator
Hi Community,

I want to propose a feature in apex-core to have a UI component in stram.
This is influenced by how basic runtime information is shown on UI by spark.

This includes following features:
1. Webpage will be hosted by stram and will be available for user to view
in one of the two ways (I'll prefer approach b):
   a. Hosted on a static port in which case we'll need to add a new servlet
to stram
   b. The webpage will be hosted on the same port as that of stram
webservice but at a different path. Say, http://
<stram_host>:<webservice_port>/ui

2. The webpage will be a static page and depending on the framework we
chose, it can show the realtime metric data from stram.

3. There will be a categories of readonly information (maybe dynamically
changing) that will be shown on the UI:
   a. Application level information
       i. Status
       ii. Number of tuples processed/emitted
       iii. Start time/ Running span
       iv. Stram events
       v. Total memory used/available
       vi. Number of container allocated/deployed
       v. Other information which is available in stram
   b. Logical Operator level information
       i. Status
       ii. Number of tuples processed/emitted
       iii. Important events related to logical operator
       iv. Container list in which the operator is deployed
       v. Any other information available in stram
       vi. Logical DAG View
   c. Container level information
       i. Status
       ii. Number of tuples processed/emitted
       iii. Important events related to logical operator
       iv. Any other information available in stram
       v. Physical DAG View

4. In second phase, there will be control related operations allowed from
the UI, as follows:
   a. Stop/Kill application
   b. Stop/Kill containers
   c. Stack trace dump of containers
   d. Logs related?
   d. etc...??

The above implementation can be done in phases as follows:
1. Have webpage display application level information only (read-only).
Static display first (i.e. page needs to refresh to see latest data), next
dynamically changing data.
2. Have jenkins build tools updated as needed by UI framework.
3. Update the backend (stram) to serve the UI pages
3. Extend the webage to display logical operator information.
4. Extend the webage to display physical operator information
5. Implement control operations on UI.


To implement above functionality, I propose ReactJS as UI framework and
MaterialUI (based on Google's material design) for theme/controls/CSS
support.

Please let me know what you think about the same.

Also, please let know if anyone is interested in contributing to this
feature.

Thanks,
Chinmay.
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Ambarish Pande
+1 For this feature. It would be really helpfull for users to visualize all
this information  and the DAG.

+1 for ReactJS + MaterialUI
 A great choice, given  its flexibility and performance.

I would like to contribute to this effort.

On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <[hidden email]> wrote:

> Hi Community,
>
> I want to propose a feature in apex-core to have a UI component in stram.
> This is influenced by how basic runtime information is shown on UI by
> spark.
>
> This includes following features:
> 1. Webpage will be hosted by stram and will be available for user to view
> in one of the two ways (I'll prefer approach b):
>    a. Hosted on a static port in which case we'll need to add a new servlet
> to stram
>    b. The webpage will be hosted on the same port as that of stram
> webservice but at a different path. Say, http://
> <stram_host>:<webservice_port>/ui
>
> 2. The webpage will be a static page and depending on the framework we
> chose, it can show the realtime metric data from stram.
>
> 3. There will be a categories of readonly information (maybe dynamically
> changing) that will be shown on the UI:
>    a. Application level information
>        i. Status
>        ii. Number of tuples processed/emitted
>        iii. Start time/ Running span
>        iv. Stram events
>        v. Total memory used/available
>        vi. Number of container allocated/deployed
>        v. Other information which is available in stram
>    b. Logical Operator level information
>        i. Status
>        ii. Number of tuples processed/emitted
>        iii. Important events related to logical operator
>        iv. Container list in which the operator is deployed
>        v. Any other information available in stram
>        vi. Logical DAG View
>    c. Container level information
>        i. Status
>        ii. Number of tuples processed/emitted
>        iii. Important events related to logical operator
>        iv. Any other information available in stram
>        v. Physical DAG View
>
> 4. In second phase, there will be control related operations allowed from
> the UI, as follows:
>    a. Stop/Kill application
>    b. Stop/Kill containers
>    c. Stack trace dump of containers
>    d. Logs related?
>    d. etc...??
>
> The above implementation can be done in phases as follows:
> 1. Have webpage display application level information only (read-only).
> Static display first (i.e. page needs to refresh to see latest data), next
> dynamically changing data.
> 2. Have jenkins build tools updated as needed by UI framework.
> 3. Update the backend (stram) to serve the UI pages
> 3. Extend the webage to display logical operator information.
> 4. Extend the webage to display physical operator information
> 5. Implement control operations on UI.
>
>
> To implement above functionality, I propose ReactJS as UI framework and
> MaterialUI (based on Google's material design) for theme/controls/CSS
> support.
>
> Please let me know what you think about the same.
>
> Also, please let know if anyone is interested in contributing to this
> feature.
>
> Thanks,
> Chinmay.
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Vlad Rozov-2
-0: stram provides REST API to query runtime information and IMO this is
sufficient to build an external webUI application(s).

Thank you,

Vlad

On 6/6/18 11:17, Ambarish Pande wrote:

> +1 For this feature. It would be really helpfull for users to visualize all
> this information  and the DAG.
>
> +1 for ReactJS + MaterialUI
>   A great choice, given  its flexibility and performance.
>
> I would like to contribute to this effort.
>
> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <[hidden email]> wrote:
>
>> Hi Community,
>>
>> I want to propose a feature in apex-core to have a UI component in stram.
>> This is influenced by how basic runtime information is shown on UI by
>> spark.
>>
>> This includes following features:
>> 1. Webpage will be hosted by stram and will be available for user to view
>> in one of the two ways (I'll prefer approach b):
>>     a. Hosted on a static port in which case we'll need to add a new servlet
>> to stram
>>     b. The webpage will be hosted on the same port as that of stram
>> webservice but at a different path. Say, http://
>> <stram_host>:<webservice_port>/ui
>>
>> 2. The webpage will be a static page and depending on the framework we
>> chose, it can show the realtime metric data from stram.
>>
>> 3. There will be a categories of readonly information (maybe dynamically
>> changing) that will be shown on the UI:
>>     a. Application level information
>>         i. Status
>>         ii. Number of tuples processed/emitted
>>         iii. Start time/ Running span
>>         iv. Stram events
>>         v. Total memory used/available
>>         vi. Number of container allocated/deployed
>>         v. Other information which is available in stram
>>     b. Logical Operator level information
>>         i. Status
>>         ii. Number of tuples processed/emitted
>>         iii. Important events related to logical operator
>>         iv. Container list in which the operator is deployed
>>         v. Any other information available in stram
>>         vi. Logical DAG View
>>     c. Container level information
>>         i. Status
>>         ii. Number of tuples processed/emitted
>>         iii. Important events related to logical operator
>>         iv. Any other information available in stram
>>         v. Physical DAG View
>>
>> 4. In second phase, there will be control related operations allowed from
>> the UI, as follows:
>>     a. Stop/Kill application
>>     b. Stop/Kill containers
>>     c. Stack trace dump of containers
>>     d. Logs related?
>>     d. etc...??
>>
>> The above implementation can be done in phases as follows:
>> 1. Have webpage display application level information only (read-only).
>> Static display first (i.e. page needs to refresh to see latest data), next
>> dynamically changing data.
>> 2. Have jenkins build tools updated as needed by UI framework.
>> 3. Update the backend (stram) to serve the UI pages
>> 3. Extend the webage to display logical operator information.
>> 4. Extend the webage to display physical operator information
>> 5. Implement control operations on UI.
>>
>>
>> To implement above functionality, I propose ReactJS as UI framework and
>> MaterialUI (based on Google's material design) for theme/controls/CSS
>> support.
>>
>> Please let me know what you think about the same.
>>
>> Also, please let know if anyone is interested in contributing to this
>> feature.
>>
>> Thanks,
>> Chinmay.
>>

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Thomas Weise-2
Administrator
In reply to this post by chinmay
strong +1 for adding a UI to Apex. This is actually something that users
expect and are accustomed to with similar projects. Thanks for taking the
initiative.

There are nuances to be discussed, but the overall plan LGTM.

To your question, I think this should use the existing web port with a
different path, this will make it easy to expose through the RM proxy. The
question whether the port is static or dynamic is a different concern,
there could be a separate feature that optionally allows the user to assign
a static port. On YARN, there is already a static address, the RM proxy
port.

I would propose to start this feature on a branch. The requirements for
build integration and source code structure should be fleshed out as you
go. It is conceivable that the UI build, which requires separate tooling,
runs as part of the release build, but not the usual CI.

Thomas



On Wed, Jun 6, 2018 at 7:43 AM, Chinmay Kolhatkar <[hidden email]>
wrote:

> Hi Community,
>
> I want to propose a feature in apex-core to have a UI component in stram.
> This is influenced by how basic runtime information is shown on UI by
> spark.
>
> This includes following features:
> 1. Webpage will be hosted by stram and will be available for user to view
> in one of the two ways (I'll prefer approach b):
>    a. Hosted on a static port in which case we'll need to add a new servlet
> to stram
>    b. The webpage will be hosted on the same port as that of stram
> webservice but at a different path. Say, http://
> <stram_host>:<webservice_port>/ui
>
> 2. The webpage will be a static page and depending on the framework we
> chose, it can show the realtime metric data from stram.
>
> 3. There will be a categories of readonly information (maybe dynamically
> changing) that will be shown on the UI:
>    a. Application level information
>        i. Status
>        ii. Number of tuples processed/emitted
>        iii. Start time/ Running span
>        iv. Stram events
>        v. Total memory used/available
>        vi. Number of container allocated/deployed
>        v. Other information which is available in stram
>    b. Logical Operator level information
>        i. Status
>        ii. Number of tuples processed/emitted
>        iii. Important events related to logical operator
>        iv. Container list in which the operator is deployed
>        v. Any other information available in stram
>        vi. Logical DAG View
>    c. Container level information
>        i. Status
>        ii. Number of tuples processed/emitted
>        iii. Important events related to logical operator
>        iv. Any other information available in stram
>        v. Physical DAG View
>
> 4. In second phase, there will be control related operations allowed from
> the UI, as follows:
>    a. Stop/Kill application
>    b. Stop/Kill containers
>    c. Stack trace dump of containers
>    d. Logs related?
>    d. etc...??
>
> The above implementation can be done in phases as follows:
> 1. Have webpage display application level information only (read-only).
> Static display first (i.e. page needs to refresh to see latest data), next
> dynamically changing data.
> 2. Have jenkins build tools updated as needed by UI framework.
> 3. Update the backend (stram) to serve the UI pages
> 3. Extend the webage to display logical operator information.
> 4. Extend the webage to display physical operator information
> 5. Implement control operations on UI.
>
>
> To implement above functionality, I propose ReactJS as UI framework and
> MaterialUI (based on Google's material design) for theme/controls/CSS
> support.
>
> Please let me know what you think about the same.
>
> Also, please let know if anyone is interested in contributing to this
> feature.
>
> Thanks,
> Chinmay.
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Ananth G
+1 for a UI for Apex.

-0 for making it part of the STRAM.

- Regarding same port, I was wondering if there will be usage patterns
based on firewall rules just based on ports ? This leads us to a situation
where we have to trust all users at the same level as the application
communication patterns.
- Deepak N mentioned of building metrics components. Assuming he is working
on this feature and is providing a REST API as well, there might be
overlaps in the functionality being implemented. Some part of the UI
features mentioned in the list seem to be definitely metrics related. Most
of the metrics UI views can be aggregated at external web applications like
Grafana of course with the help of other tooling like REST server from
atrato or Prometheus components.
- Are we considering serving metrics beyond application restarts? If yes,
visualising metrics is a better fit for tooling carved out for those
purposes given the amount of information will be huge and we will also need
potentially time series stores ?
- The highest value is from the points mentioned in the second phase and
DAG views implementation in the first phase.( assuming Deepak N proposal
for metrics is implementing a REST API).

There is definitely a huge value add in providing a UI and the need of the
hour for Apex. My concern is that we might be diluting the value add
provided to build such a functionality inside the STRAM and hence feel it
is better served as an external application as opposed to something
embedded in the STRAM.

Regards,
Ananth


On Sun, Jun 10, 2018 at 3:10 AM, Thomas Weise <[hidden email]> wrote:

> strong +1 for adding a UI to Apex. This is actually something that users
> expect and are accustomed to with similar projects. Thanks for taking the
> initiative.
>
> There are nuances to be discussed, but the overall plan LGTM.
>
> To your question, I think this should use the existing web port with a
> different path, this will make it easy to expose through the RM proxy. The
> question whether the port is static or dynamic is a different concern,
> there could be a separate feature that optionally allows the user to assign
> a static port. On YARN, there is already a static address, the RM proxy
> port.
>
> I would propose to start this feature on a branch. The requirements for
> build integration and source code structure should be fleshed out as you
> go. It is conceivable that the UI build, which requires separate tooling,
> runs as part of the release build, but not the usual CI.
>
> Thomas
>
>
>
> On Wed, Jun 6, 2018 at 7:43 AM, Chinmay Kolhatkar <[hidden email]>
> wrote:
>
> > Hi Community,
> >
> > I want to propose a feature in apex-core to have a UI component in stram.
> > This is influenced by how basic runtime information is shown on UI by
> > spark.
> >
> > This includes following features:
> > 1. Webpage will be hosted by stram and will be available for user to view
> > in one of the two ways (I'll prefer approach b):
> >    a. Hosted on a static port in which case we'll need to add a new
> servlet
> > to stram
> >    b. The webpage will be hosted on the same port as that of stram
> > webservice but at a different path. Say, http://
> > <stram_host>:<webservice_port>/ui
> >
> > 2. The webpage will be a static page and depending on the framework we
> > chose, it can show the realtime metric data from stram.
> >
> > 3. There will be a categories of readonly information (maybe dynamically
> > changing) that will be shown on the UI:
> >    a. Application level information
> >        i. Status
> >        ii. Number of tuples processed/emitted
> >        iii. Start time/ Running span
> >        iv. Stram events
> >        v. Total memory used/available
> >        vi. Number of container allocated/deployed
> >        v. Other information which is available in stram
> >    b. Logical Operator level information
> >        i. Status
> >        ii. Number of tuples processed/emitted
> >        iii. Important events related to logical operator
> >        iv. Container list in which the operator is deployed
> >        v. Any other information available in stram
> >        vi. Logical DAG View
> >    c. Container level information
> >        i. Status
> >        ii. Number of tuples processed/emitted
> >        iii. Important events related to logical operator
> >        iv. Any other information available in stram
> >        v. Physical DAG View
> >
> > 4. In second phase, there will be control related operations allowed from
> > the UI, as follows:
> >    a. Stop/Kill application
> >    b. Stop/Kill containers
> >    c. Stack trace dump of containers
> >    d. Logs related?
> >    d. etc...??
> >
> > The above implementation can be done in phases as follows:
> > 1. Have webpage display application level information only (read-only).
> > Static display first (i.e. page needs to refresh to see latest data),
> next
> > dynamically changing data.
> > 2. Have jenkins build tools updated as needed by UI framework.
> > 3. Update the backend (stram) to serve the UI pages
> > 3. Extend the webage to display logical operator information.
> > 4. Extend the webage to display physical operator information
> > 5. Implement control operations on UI.
> >
> >
> > To implement above functionality, I propose ReactJS as UI framework and
> > MaterialUI (based on Google's material design) for theme/controls/CSS
> > support.
> >
> > Please let me know what you think about the same.
> >
> > Also, please let know if anyone is interested in contributing to this
> > feature.
> >
> > Thanks,
> > Chinmay.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Vlad Rozov-2
In reply to this post by Vlad Rozov-2
Clarification: I am -0 on making UI part of the stram. I am +1 on
providing reference webUI application outside of the stram.

Thank you,

Vlad

On 6/9/18 07:05, Vlad Rozov wrote:

> -0: stram provides REST API to query runtime information and IMO this
> is sufficient to build an external webUI application(s).
>
> Thank you,
>
> Vlad
>
> On 6/6/18 11:17, Ambarish Pande wrote:
>> +1 For this feature. It would be really helpfull for users to
>> visualize all
>> this information  and the DAG.
>>
>> +1 for ReactJS + MaterialUI
>>   A great choice, given  its flexibility and performance.
>>
>> I would like to contribute to this effort.
>>
>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <[hidden email]>
>> wrote:
>>
>>> Hi Community,
>>>
>>> I want to propose a feature in apex-core to have a UI component in
>>> stram.
>>> This is influenced by how basic runtime information is shown on UI by
>>> spark.
>>>
>>> This includes following features:
>>> 1. Webpage will be hosted by stram and will be available for user to
>>> view
>>> in one of the two ways (I'll prefer approach b):
>>>     a. Hosted on a static port in which case we'll need to add a new
>>> servlet
>>> to stram
>>>     b. The webpage will be hosted on the same port as that of stram
>>> webservice but at a different path. Say, http://
>>> <stram_host>:<webservice_port>/ui
>>>
>>> 2. The webpage will be a static page and depending on the framework we
>>> chose, it can show the realtime metric data from stram.
>>>
>>> 3. There will be a categories of readonly information (maybe
>>> dynamically
>>> changing) that will be shown on the UI:
>>>     a. Application level information
>>>         i. Status
>>>         ii. Number of tuples processed/emitted
>>>         iii. Start time/ Running span
>>>         iv. Stram events
>>>         v. Total memory used/available
>>>         vi. Number of container allocated/deployed
>>>         v. Other information which is available in stram
>>>     b. Logical Operator level information
>>>         i. Status
>>>         ii. Number of tuples processed/emitted
>>>         iii. Important events related to logical operator
>>>         iv. Container list in which the operator is deployed
>>>         v. Any other information available in stram
>>>         vi. Logical DAG View
>>>     c. Container level information
>>>         i. Status
>>>         ii. Number of tuples processed/emitted
>>>         iii. Important events related to logical operator
>>>         iv. Any other information available in stram
>>>         v. Physical DAG View
>>>
>>> 4. In second phase, there will be control related operations allowed
>>> from
>>> the UI, as follows:
>>>     a. Stop/Kill application
>>>     b. Stop/Kill containers
>>>     c. Stack trace dump of containers
>>>     d. Logs related?
>>>     d. etc...??
>>>
>>> The above implementation can be done in phases as follows:
>>> 1. Have webpage display application level information only (read-only).
>>> Static display first (i.e. page needs to refresh to see latest
>>> data), next
>>> dynamically changing data.
>>> 2. Have jenkins build tools updated as needed by UI framework.
>>> 3. Update the backend (stram) to serve the UI pages
>>> 3. Extend the webage to display logical operator information.
>>> 4. Extend the webage to display physical operator information
>>> 5. Implement control operations on UI.
>>>
>>>
>>> To implement above functionality, I propose ReactJS as UI framework and
>>> MaterialUI (based on Google's material design) for theme/controls/CSS
>>> support.
>>>
>>> Please let me know what you think about the same.
>>>
>>> Also, please let know if anyone is interested in contributing to this
>>> feature.
>>>
>>> Thanks,
>>> Chinmay.
>>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Pramod Immaneni-3
I too think that a comprehensive UI works better standalone but given that
stram already runs a webapp server, provides web services and RM provides a
proxy with mapped url space for it, we can extend stram to provide better
visual output. Overall I am +1 on the idea.

On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <[hidden email]> wrote:

> Clarification: I am -0 on making UI part of the stram. I am +1 on
> providing reference webUI application outside of the stram.
>
> Thank you,
>
> Vlad
>
> On 6/9/18 07:05, Vlad Rozov wrote:
> > -0: stram provides REST API to query runtime information and IMO this
> > is sufficient to build an external webUI application(s).
> >
> > Thank you,
> >
> > Vlad
> >
> > On 6/6/18 11:17, Ambarish Pande wrote:
> >> +1 For this feature. It would be really helpfull for users to
> >> visualize all
> >> this information  and the DAG.
> >>
> >> +1 for ReactJS + MaterialUI
> >>   A great choice, given  its flexibility and performance.
> >>
> >> I would like to contribute to this effort.
> >>
> >> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <[hidden email]>
> >> wrote:
> >>
> >>> Hi Community,
> >>>
> >>> I want to propose a feature in apex-core to have a UI component in
> >>> stram.
> >>> This is influenced by how basic runtime information is shown on UI by
> >>> spark.
> >>>
> >>> This includes following features:
> >>> 1. Webpage will be hosted by stram and will be available for user to
> >>> view
> >>> in one of the two ways (I'll prefer approach b):
> >>>     a. Hosted on a static port in which case we'll need to add a new
> >>> servlet
> >>> to stram
> >>>     b. The webpage will be hosted on the same port as that of stram
> >>> webservice but at a different path. Say, http://
> >>> <stram_host>:<webservice_port>/ui
> >>>
> >>> 2. The webpage will be a static page and depending on the framework we
> >>> chose, it can show the realtime metric data from stram.
> >>>
> >>> 3. There will be a categories of readonly information (maybe
> >>> dynamically
> >>> changing) that will be shown on the UI:
> >>>     a. Application level information
> >>>         i. Status
> >>>         ii. Number of tuples processed/emitted
> >>>         iii. Start time/ Running span
> >>>         iv. Stram events
> >>>         v. Total memory used/available
> >>>         vi. Number of container allocated/deployed
> >>>         v. Other information which is available in stram
> >>>     b. Logical Operator level information
> >>>         i. Status
> >>>         ii. Number of tuples processed/emitted
> >>>         iii. Important events related to logical operator
> >>>         iv. Container list in which the operator is deployed
> >>>         v. Any other information available in stram
> >>>         vi. Logical DAG View
> >>>     c. Container level information
> >>>         i. Status
> >>>         ii. Number of tuples processed/emitted
> >>>         iii. Important events related to logical operator
> >>>         iv. Any other information available in stram
> >>>         v. Physical DAG View
> >>>
> >>> 4. In second phase, there will be control related operations allowed
> >>> from
> >>> the UI, as follows:
> >>>     a. Stop/Kill application
> >>>     b. Stop/Kill containers
> >>>     c. Stack trace dump of containers
> >>>     d. Logs related?
> >>>     d. etc...??
> >>>
> >>> The above implementation can be done in phases as follows:
> >>> 1. Have webpage display application level information only (read-only).
> >>> Static display first (i.e. page needs to refresh to see latest
> >>> data), next
> >>> dynamically changing data.
> >>> 2. Have jenkins build tools updated as needed by UI framework.
> >>> 3. Update the backend (stram) to serve the UI pages
> >>> 3. Extend the webage to display logical operator information.
> >>> 4. Extend the webage to display physical operator information
> >>> 5. Implement control operations on UI.
> >>>
> >>>
> >>> To implement above functionality, I propose ReactJS as UI framework and
> >>> MaterialUI (based on Google's material design) for theme/controls/CSS
> >>> support.
> >>>
> >>> Please let me know what you think about the same.
> >>>
> >>> Also, please let know if anyone is interested in contributing to this
> >>> feature.
> >>>
> >>> Thanks,
> >>> Chinmay.
> >>>
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Vlad Rozov-2
IMO it will be hard to provide UI that meets everyone expectation. An
external webUI can serve two goals

1. provide reference UI implementation
2. ensure that REST API have all necessary entry points

by embedding webUI directly into stram it will be tempting to shortcut
some calls to use stram classes directly.

Thank you,

Vlad

On 6/11/18 12:18, Pramod Immaneni wrote:

> I too think that a comprehensive UI works better standalone but given that
> stram already runs a webapp server, provides web services and RM provides a
> proxy with mapped url space for it, we can extend stram to provide better
> visual output. Overall I am +1 on the idea.
>
> On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <[hidden email]> wrote:
>
>> Clarification: I am -0 on making UI part of the stram. I am +1 on
>> providing reference webUI application outside of the stram.
>>
>> Thank you,
>>
>> Vlad
>>
>> On 6/9/18 07:05, Vlad Rozov wrote:
>>> -0: stram provides REST API to query runtime information and IMO this
>>> is sufficient to build an external webUI application(s).
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>> On 6/6/18 11:17, Ambarish Pande wrote:
>>>> +1 For this feature. It would be really helpfull for users to
>>>> visualize all
>>>> this information  and the DAG.
>>>>
>>>> +1 for ReactJS + MaterialUI
>>>>    A great choice, given  its flexibility and performance.
>>>>
>>>> I would like to contribute to this effort.
>>>>
>>>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <[hidden email]>
>>>> wrote:
>>>>
>>>>> Hi Community,
>>>>>
>>>>> I want to propose a feature in apex-core to have a UI component in
>>>>> stram.
>>>>> This is influenced by how basic runtime information is shown on UI by
>>>>> spark.
>>>>>
>>>>> This includes following features:
>>>>> 1. Webpage will be hosted by stram and will be available for user to
>>>>> view
>>>>> in one of the two ways (I'll prefer approach b):
>>>>>      a. Hosted on a static port in which case we'll need to add a new
>>>>> servlet
>>>>> to stram
>>>>>      b. The webpage will be hosted on the same port as that of stram
>>>>> webservice but at a different path. Say, http://
>>>>> <stram_host>:<webservice_port>/ui
>>>>>
>>>>> 2. The webpage will be a static page and depending on the framework we
>>>>> chose, it can show the realtime metric data from stram.
>>>>>
>>>>> 3. There will be a categories of readonly information (maybe
>>>>> dynamically
>>>>> changing) that will be shown on the UI:
>>>>>      a. Application level information
>>>>>          i. Status
>>>>>          ii. Number of tuples processed/emitted
>>>>>          iii. Start time/ Running span
>>>>>          iv. Stram events
>>>>>          v. Total memory used/available
>>>>>          vi. Number of container allocated/deployed
>>>>>          v. Other information which is available in stram
>>>>>      b. Logical Operator level information
>>>>>          i. Status
>>>>>          ii. Number of tuples processed/emitted
>>>>>          iii. Important events related to logical operator
>>>>>          iv. Container list in which the operator is deployed
>>>>>          v. Any other information available in stram
>>>>>          vi. Logical DAG View
>>>>>      c. Container level information
>>>>>          i. Status
>>>>>          ii. Number of tuples processed/emitted
>>>>>          iii. Important events related to logical operator
>>>>>          iv. Any other information available in stram
>>>>>          v. Physical DAG View
>>>>>
>>>>> 4. In second phase, there will be control related operations allowed
>>>>> from
>>>>> the UI, as follows:
>>>>>      a. Stop/Kill application
>>>>>      b. Stop/Kill containers
>>>>>      c. Stack trace dump of containers
>>>>>      d. Logs related?
>>>>>      d. etc...??
>>>>>
>>>>> The above implementation can be done in phases as follows:
>>>>> 1. Have webpage display application level information only (read-only).
>>>>> Static display first (i.e. page needs to refresh to see latest
>>>>> data), next
>>>>> dynamically changing data.
>>>>> 2. Have jenkins build tools updated as needed by UI framework.
>>>>> 3. Update the backend (stram) to serve the UI pages
>>>>> 3. Extend the webage to display logical operator information.
>>>>> 4. Extend the webage to display physical operator information
>>>>> 5. Implement control operations on UI.
>>>>>
>>>>>
>>>>> To implement above functionality, I propose ReactJS as UI framework and
>>>>> MaterialUI (based on Google's material design) for theme/controls/CSS
>>>>> support.
>>>>>
>>>>> Please let me know what you think about the same.
>>>>>
>>>>> Also, please let know if anyone is interested in contributing to this
>>>>> feature.
>>>>>
>>>>> Thanks,
>>>>> Chinmay.
>>>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Shubhrajyoti Mohapatra
Hi Community,

I would like to propose a slightly different approach on having an UI for
Apex.

The details are below:

   1. This involves having an external web-server installed along side of
   apexCli. We will basically mimic all the functionalities that apexCli
   currently provides.
   2. We will develop the necessary REST APIs for fetch/update operations
   which apexCli currently supports.
   3. Because most of the information and operations needed by a typical
   user are already there in apexCli, it will of great help to end users and
   which may increase the adoption rate.

About Chinmay's proposal of having the STRAM hosting the apps, please find
my comments below:

   1. It may increase load on STRAM if we were to provide real time display
   of information. I am aware that apexCli fetches info from STRAM apis but
   continuous polling can cause extra over-head.
   2. The UI will have limited scope for the particular application. Users
   would definitely love to have a comprehensive UI for the management of Apex
   Apps as a whole.
   3. UI might be able to affect the whole application through STRAM hence
   giving direct access to STRAM APIs might not be good idea.
   4. Because STRAM can't have a comprehensive UI hosted inside, anyways we
   will have to develop an external one.

So I would like propose the plan on action like this.

   1. We develop the wrapper UI for apexCli in the first phase and give
   user all the management options as REST APIs.
   2. If some of information we require are not in the current apexCli
   implementation, we can add those feature to apexCli and if required to
   STRAM APIs in subsequent phases.
   3. In future we can even remove the REST server from STRAM if we could
   provide something like MetricsStore functionality which was there in
   DataTorrent RTS.

--
Thanks and Regards,
Shubhrajyoti Mohapatra




On Tue, Jun 12, 2018 at 2:49 AM Vlad Rozov <[hidden email]> wrote:

> IMO it will be hard to provide UI that meets everyone expectation. An
> external webUI can serve two goals
>
> 1. provide reference UI implementation
> 2. ensure that REST API have all necessary entry points
>
> by embedding webUI directly into stram it will be tempting to shortcut
> some calls to use stram classes directly.
>
> Thank you,
>
> Vlad
>
> On 6/11/18 12:18, Pramod Immaneni wrote:
> > I too think that a comprehensive UI works better standalone but given
> that
> > stram already runs a webapp server, provides web services and RM
> provides a
> > proxy with mapped url space for it, we can extend stram to provide better
> > visual output. Overall I am +1 on the idea.
> >
> > On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <[hidden email]> wrote:
> >
> >> Clarification: I am -0 on making UI part of the stram. I am +1 on
> >> providing reference webUI application outside of the stram.
> >>
> >> Thank you,
> >>
> >> Vlad
> >>
> >> On 6/9/18 07:05, Vlad Rozov wrote:
> >>> -0: stram provides REST API to query runtime information and IMO this
> >>> is sufficient to build an external webUI application(s).
> >>>
> >>> Thank you,
> >>>
> >>> Vlad
> >>>
> >>> On 6/6/18 11:17, Ambarish Pande wrote:
> >>>> +1 For this feature. It would be really helpfull for users to
> >>>> visualize all
> >>>> this information  and the DAG.
> >>>>
> >>>> +1 for ReactJS + MaterialUI
> >>>>    A great choice, given  its flexibility and performance.
> >>>>
> >>>> I would like to contribute to this effort.
> >>>>
> >>>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <[hidden email]>
> >>>> wrote:
> >>>>
> >>>>> Hi Community,
> >>>>>
> >>>>> I want to propose a feature in apex-core to have a UI component in
> >>>>> stram.
> >>>>> This is influenced by how basic runtime information is shown on UI by
> >>>>> spark.
> >>>>>
> >>>>> This includes following features:
> >>>>> 1. Webpage will be hosted by stram and will be available for user to
> >>>>> view
> >>>>> in one of the two ways (I'll prefer approach b):
> >>>>>      a. Hosted on a static port in which case we'll need to add a new
> >>>>> servlet
> >>>>> to stram
> >>>>>      b. The webpage will be hosted on the same port as that of stram
> >>>>> webservice but at a different path. Say, http://
> >>>>> <stram_host>:<webservice_port>/ui
> >>>>>
> >>>>> 2. The webpage will be a static page and depending on the framework
> we
> >>>>> chose, it can show the realtime metric data from stram.
> >>>>>
> >>>>> 3. There will be a categories of readonly information (maybe
> >>>>> dynamically
> >>>>> changing) that will be shown on the UI:
> >>>>>      a. Application level information
> >>>>>          i. Status
> >>>>>          ii. Number of tuples processed/emitted
> >>>>>          iii. Start time/ Running span
> >>>>>          iv. Stram events
> >>>>>          v. Total memory used/available
> >>>>>          vi. Number of container allocated/deployed
> >>>>>          v. Other information which is available in stram
> >>>>>      b. Logical Operator level information
> >>>>>          i. Status
> >>>>>          ii. Number of tuples processed/emitted
> >>>>>          iii. Important events related to logical operator
> >>>>>          iv. Container list in which the operator is deployed
> >>>>>          v. Any other information available in stram
> >>>>>          vi. Logical DAG View
> >>>>>      c. Container level information
> >>>>>          i. Status
> >>>>>          ii. Number of tuples processed/emitted
> >>>>>          iii. Important events related to logical operator
> >>>>>          iv. Any other information available in stram
> >>>>>          v. Physical DAG View
> >>>>>
> >>>>> 4. In second phase, there will be control related operations allowed
> >>>>> from
> >>>>> the UI, as follows:
> >>>>>      a. Stop/Kill application
> >>>>>      b. Stop/Kill containers
> >>>>>      c. Stack trace dump of containers
> >>>>>      d. Logs related?
> >>>>>      d. etc...??
> >>>>>
> >>>>> The above implementation can be done in phases as follows:
> >>>>> 1. Have webpage display application level information only
> (read-only).
> >>>>> Static display first (i.e. page needs to refresh to see latest
> >>>>> data), next
> >>>>> dynamically changing data.
> >>>>> 2. Have jenkins build tools updated as needed by UI framework.
> >>>>> 3. Update the backend (stram) to serve the UI pages
> >>>>> 3. Extend the webage to display logical operator information.
> >>>>> 4. Extend the webage to display physical operator information
> >>>>> 5. Implement control operations on UI.
> >>>>>
> >>>>>
> >>>>> To implement above functionality, I propose ReactJS as UI framework
> and
> >>>>> MaterialUI (based on Google's material design) for theme/controls/CSS
> >>>>> support.
> >>>>>
> >>>>> Please let me know what you think about the same.
> >>>>>
> >>>>> Also, please let know if anyone is interested in contributing to this
> >>>>> feature.
> >>>>>
> >>>>> Thanks,
> >>>>> Chinmay.
> >>>>>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Thomas Weise-2
Administrator
Looks like there is some disconnect in this thread, a few replies express
concerns that the proposal IMO doesn't imply.

There is already a REST API that backs all of the CLI functionality. Modern
UIs run in the browser, and so what was proposed (unless I misunderstood)
would be based on the same  REST API and nothing extra. The only additional
thing the AM has to do is serve static files. And AFAIK it would actually
simplify the UI implementation when files and REST API have the same origin.

To me where files are coming from is actually a smaller piece and something
that can be augmented. But before proposing solutions outside of Apex or
additional deployment requirements, please consider the user experience. It
should be really simple to setup and use the UI.

I would expect that a large part of the discussion evolves around the UI
functionality that helps the target persona.

Thanks,
Thomas

On Tue, Jun 12, 2018, 8:05 PM Shubhrajyoti Mohapatra <[hidden email]>
wrote:

> Hi Community,
>
> I would like to propose a slightly different approach on having an UI for
> Apex.
>
> The details are below:
>
>    1. This involves having an external web-server installed along side of
>    apexCli. We will basically mimic all the functionalities that apexCli
>    currently provides.
>    2. We will develop the necessary REST APIs for fetch/update operations
>    which apexCli currently supports.
>    3. Because most of the information and operations needed by a typical
>    user are already there in apexCli, it will of great help to end users
> and
>    which may increase the adoption rate.
>
> About Chinmay's proposal of having the STRAM hosting the apps, please find
> my comments below:
>
>    1. It may increase load on STRAM if we were to provide real time display
>    of information. I am aware that apexCli fetches info from STRAM apis but
>    continuous polling can cause extra over-head.
>    2. The UI will have limited scope for the particular application. Users
>    would definitely love to have a comprehensive UI for the management of
> Apex
>    Apps as a whole.
>    3. UI might be able to affect the whole application through STRAM hence
>    giving direct access to STRAM APIs might not be good idea.
>    4. Because STRAM can't have a comprehensive UI hosted inside, anyways we
>    will have to develop an external one.
>
> So I would like propose the plan on action like this.
>
>    1. We develop the wrapper UI for apexCli in the first phase and give
>    user all the management options as REST APIs.
>    2. If some of information we require are not in the current apexCli
>    implementation, we can add those feature to apexCli and if required to
>    STRAM APIs in subsequent phases.
>    3. In future we can even remove the REST server from STRAM if we could
>    provide something like MetricsStore functionality which was there in
>    DataTorrent RTS.
>
> --
> Thanks and Regards,
> Shubhrajyoti Mohapatra
>
>
>
>
> On Tue, Jun 12, 2018 at 2:49 AM Vlad Rozov <[hidden email]> wrote:
>
> > IMO it will be hard to provide UI that meets everyone expectation. An
> > external webUI can serve two goals
> >
> > 1. provide reference UI implementation
> > 2. ensure that REST API have all necessary entry points
> >
> > by embedding webUI directly into stram it will be tempting to shortcut
> > some calls to use stram classes directly.
> >
> > Thank you,
> >
> > Vlad
> >
> > On 6/11/18 12:18, Pramod Immaneni wrote:
> > > I too think that a comprehensive UI works better standalone but given
> > that
> > > stram already runs a webapp server, provides web services and RM
> > provides a
> > > proxy with mapped url space for it, we can extend stram to provide
> better
> > > visual output. Overall I am +1 on the idea.
> > >
> > > On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <[hidden email]> wrote:
> > >
> > >> Clarification: I am -0 on making UI part of the stram. I am +1 on
> > >> providing reference webUI application outside of the stram.
> > >>
> > >> Thank you,
> > >>
> > >> Vlad
> > >>
> > >> On 6/9/18 07:05, Vlad Rozov wrote:
> > >>> -0: stram provides REST API to query runtime information and IMO this
> > >>> is sufficient to build an external webUI application(s).
> > >>>
> > >>> Thank you,
> > >>>
> > >>> Vlad
> > >>>
> > >>> On 6/6/18 11:17, Ambarish Pande wrote:
> > >>>> +1 For this feature. It would be really helpfull for users to
> > >>>> visualize all
> > >>>> this information  and the DAG.
> > >>>>
> > >>>> +1 for ReactJS + MaterialUI
> > >>>>    A great choice, given  its flexibility and performance.
> > >>>>
> > >>>> I would like to contribute to this effort.
> > >>>>
> > >>>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <
> [hidden email]>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Community,
> > >>>>>
> > >>>>> I want to propose a feature in apex-core to have a UI component in
> > >>>>> stram.
> > >>>>> This is influenced by how basic runtime information is shown on UI
> by
> > >>>>> spark.
> > >>>>>
> > >>>>> This includes following features:
> > >>>>> 1. Webpage will be hosted by stram and will be available for user
> to
> > >>>>> view
> > >>>>> in one of the two ways (I'll prefer approach b):
> > >>>>>      a. Hosted on a static port in which case we'll need to add a
> new
> > >>>>> servlet
> > >>>>> to stram
> > >>>>>      b. The webpage will be hosted on the same port as that of
> stram
> > >>>>> webservice but at a different path. Say, http://
> > >>>>> <stram_host>:<webservice_port>/ui
> > >>>>>
> > >>>>> 2. The webpage will be a static page and depending on the framework
> > we
> > >>>>> chose, it can show the realtime metric data from stram.
> > >>>>>
> > >>>>> 3. There will be a categories of readonly information (maybe
> > >>>>> dynamically
> > >>>>> changing) that will be shown on the UI:
> > >>>>>      a. Application level information
> > >>>>>          i. Status
> > >>>>>          ii. Number of tuples processed/emitted
> > >>>>>          iii. Start time/ Running span
> > >>>>>          iv. Stram events
> > >>>>>          v. Total memory used/available
> > >>>>>          vi. Number of container allocated/deployed
> > >>>>>          v. Other information which is available in stram
> > >>>>>      b. Logical Operator level information
> > >>>>>          i. Status
> > >>>>>          ii. Number of tuples processed/emitted
> > >>>>>          iii. Important events related to logical operator
> > >>>>>          iv. Container list in which the operator is deployed
> > >>>>>          v. Any other information available in stram
> > >>>>>          vi. Logical DAG View
> > >>>>>      c. Container level information
> > >>>>>          i. Status
> > >>>>>          ii. Number of tuples processed/emitted
> > >>>>>          iii. Important events related to logical operator
> > >>>>>          iv. Any other information available in stram
> > >>>>>          v. Physical DAG View
> > >>>>>
> > >>>>> 4. In second phase, there will be control related operations
> allowed
> > >>>>> from
> > >>>>> the UI, as follows:
> > >>>>>      a. Stop/Kill application
> > >>>>>      b. Stop/Kill containers
> > >>>>>      c. Stack trace dump of containers
> > >>>>>      d. Logs related?
> > >>>>>      d. etc...??
> > >>>>>
> > >>>>> The above implementation can be done in phases as follows:
> > >>>>> 1. Have webpage display application level information only
> > (read-only).
> > >>>>> Static display first (i.e. page needs to refresh to see latest
> > >>>>> data), next
> > >>>>> dynamically changing data.
> > >>>>> 2. Have jenkins build tools updated as needed by UI framework.
> > >>>>> 3. Update the backend (stram) to serve the UI pages
> > >>>>> 3. Extend the webage to display logical operator information.
> > >>>>> 4. Extend the webage to display physical operator information
> > >>>>> 5. Implement control operations on UI.
> > >>>>>
> > >>>>>
> > >>>>> To implement above functionality, I propose ReactJS as UI framework
> > and
> > >>>>> MaterialUI (based on Google's material design) for
> theme/controls/CSS
> > >>>>> support.
> > >>>>>
> > >>>>> Please let me know what you think about the same.
> > >>>>>
> > >>>>> Also, please let know if anyone is interested in contributing to
> this
> > >>>>> feature.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Chinmay.
> > >>>>>
> > >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Chinmay Kolhatkar-2
Hi Team,

I agree with Thomas. The main focus of this proposal was to provide a UI
layer using existing REST APIs.
I think we should first focus on providing a meaningful UI served by AM to
user using the same existing REST APIs.

If in general, there is a agreement for the proposal, I can proceed with
creating a JIRA and feature branch for the same.

Thanks,
Chinmay.


On Wed, Jun 13, 2018 at 12:21 AM Thomas Weise <[hidden email]> wrote:

> Looks like there is some disconnect in this thread, a few replies express
> concerns that the proposal IMO doesn't imply.
>
> There is already a REST API that backs all of the CLI functionality. Modern
> UIs run in the browser, and so what was proposed (unless I misunderstood)
> would be based on the same  REST API and nothing extra. The only additional
> thing the AM has to do is serve static files. And AFAIK it would actually
> simplify the UI implementation when files and REST API have the same
> origin.
>
> To me where files are coming from is actually a smaller piece and something
> that can be augmented. But before proposing solutions outside of Apex or
> additional deployment requirements, please consider the user experience. It
> should be really simple to setup and use the UI.
>
> I would expect that a large part of the discussion evolves around the UI
> functionality that helps the target persona.
>
> Thanks,
> Thomas
>
> On Tue, Jun 12, 2018, 8:05 PM Shubhrajyoti Mohapatra <
> [hidden email]>
> wrote:
>
> > Hi Community,
> >
> > I would like to propose a slightly different approach on having an UI for
> > Apex.
> >
> > The details are below:
> >
> >    1. This involves having an external web-server installed along side of
> >    apexCli. We will basically mimic all the functionalities that apexCli
> >    currently provides.
> >    2. We will develop the necessary REST APIs for fetch/update operations
> >    which apexCli currently supports.
> >    3. Because most of the information and operations needed by a typical
> >    user are already there in apexCli, it will of great help to end users
> > and
> >    which may increase the adoption rate.
> >
> > About Chinmay's proposal of having the STRAM hosting the apps, please
> find
> > my comments below:
> >
> >    1. It may increase load on STRAM if we were to provide real time
> display
> >    of information. I am aware that apexCli fetches info from STRAM apis
> but
> >    continuous polling can cause extra over-head.
> >    2. The UI will have limited scope for the particular application.
> Users
> >    would definitely love to have a comprehensive UI for the management of
> > Apex
> >    Apps as a whole.
> >    3. UI might be able to affect the whole application through STRAM
> hence
> >    giving direct access to STRAM APIs might not be good idea.
> >    4. Because STRAM can't have a comprehensive UI hosted inside, anyways
> we
> >    will have to develop an external one.
> >
> > So I would like propose the plan on action like this.
> >
> >    1. We develop the wrapper UI for apexCli in the first phase and give
> >    user all the management options as REST APIs.
> >    2. If some of information we require are not in the current apexCli
> >    implementation, we can add those feature to apexCli and if required to
> >    STRAM APIs in subsequent phases.
> >    3. In future we can even remove the REST server from STRAM if we could
> >    provide something like MetricsStore functionality which was there in
> >    DataTorrent RTS.
> >
> > --
> > Thanks and Regards,
> > Shubhrajyoti Mohapatra
> >
> >
> >
> >
> > On Tue, Jun 12, 2018 at 2:49 AM Vlad Rozov <[hidden email]> wrote:
> >
> > > IMO it will be hard to provide UI that meets everyone expectation. An
> > > external webUI can serve two goals
> > >
> > > 1. provide reference UI implementation
> > > 2. ensure that REST API have all necessary entry points
> > >
> > > by embedding webUI directly into stram it will be tempting to shortcut
> > > some calls to use stram classes directly.
> > >
> > > Thank you,
> > >
> > > Vlad
> > >
> > > On 6/11/18 12:18, Pramod Immaneni wrote:
> > > > I too think that a comprehensive UI works better standalone but given
> > > that
> > > > stram already runs a webapp server, provides web services and RM
> > > provides a
> > > > proxy with mapped url space for it, we can extend stram to provide
> > better
> > > > visual output. Overall I am +1 on the idea.
> > > >
> > > > On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <[hidden email]>
> wrote:
> > > >
> > > >> Clarification: I am -0 on making UI part of the stram. I am +1 on
> > > >> providing reference webUI application outside of the stram.
> > > >>
> > > >> Thank you,
> > > >>
> > > >> Vlad
> > > >>
> > > >> On 6/9/18 07:05, Vlad Rozov wrote:
> > > >>> -0: stram provides REST API to query runtime information and IMO
> this
> > > >>> is sufficient to build an external webUI application(s).
> > > >>>
> > > >>> Thank you,
> > > >>>
> > > >>> Vlad
> > > >>>
> > > >>> On 6/6/18 11:17, Ambarish Pande wrote:
> > > >>>> +1 For this feature. It would be really helpfull for users to
> > > >>>> visualize all
> > > >>>> this information  and the DAG.
> > > >>>>
> > > >>>> +1 for ReactJS + MaterialUI
> > > >>>>    A great choice, given  its flexibility and performance.
> > > >>>>
> > > >>>> I would like to contribute to this effort.
> > > >>>>
> > > >>>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <
> > [hidden email]>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi Community,
> > > >>>>>
> > > >>>>> I want to propose a feature in apex-core to have a UI component
> in
> > > >>>>> stram.
> > > >>>>> This is influenced by how basic runtime information is shown on
> UI
> > by
> > > >>>>> spark.
> > > >>>>>
> > > >>>>> This includes following features:
> > > >>>>> 1. Webpage will be hosted by stram and will be available for user
> > to
> > > >>>>> view
> > > >>>>> in one of the two ways (I'll prefer approach b):
> > > >>>>>      a. Hosted on a static port in which case we'll need to add a
> > new
> > > >>>>> servlet
> > > >>>>> to stram
> > > >>>>>      b. The webpage will be hosted on the same port as that of
> > stram
> > > >>>>> webservice but at a different path. Say, http://
> > > >>>>> <stram_host>:<webservice_port>/ui
> > > >>>>>
> > > >>>>> 2. The webpage will be a static page and depending on the
> framework
> > > we
> > > >>>>> chose, it can show the realtime metric data from stram.
> > > >>>>>
> > > >>>>> 3. There will be a categories of readonly information (maybe
> > > >>>>> dynamically
> > > >>>>> changing) that will be shown on the UI:
> > > >>>>>      a. Application level information
> > > >>>>>          i. Status
> > > >>>>>          ii. Number of tuples processed/emitted
> > > >>>>>          iii. Start time/ Running span
> > > >>>>>          iv. Stram events
> > > >>>>>          v. Total memory used/available
> > > >>>>>          vi. Number of container allocated/deployed
> > > >>>>>          v. Other information which is available in stram
> > > >>>>>      b. Logical Operator level information
> > > >>>>>          i. Status
> > > >>>>>          ii. Number of tuples processed/emitted
> > > >>>>>          iii. Important events related to logical operator
> > > >>>>>          iv. Container list in which the operator is deployed
> > > >>>>>          v. Any other information available in stram
> > > >>>>>          vi. Logical DAG View
> > > >>>>>      c. Container level information
> > > >>>>>          i. Status
> > > >>>>>          ii. Number of tuples processed/emitted
> > > >>>>>          iii. Important events related to logical operator
> > > >>>>>          iv. Any other information available in stram
> > > >>>>>          v. Physical DAG View
> > > >>>>>
> > > >>>>> 4. In second phase, there will be control related operations
> > allowed
> > > >>>>> from
> > > >>>>> the UI, as follows:
> > > >>>>>      a. Stop/Kill application
> > > >>>>>      b. Stop/Kill containers
> > > >>>>>      c. Stack trace dump of containers
> > > >>>>>      d. Logs related?
> > > >>>>>      d. etc...??
> > > >>>>>
> > > >>>>> The above implementation can be done in phases as follows:
> > > >>>>> 1. Have webpage display application level information only
> > > (read-only).
> > > >>>>> Static display first (i.e. page needs to refresh to see latest
> > > >>>>> data), next
> > > >>>>> dynamically changing data.
> > > >>>>> 2. Have jenkins build tools updated as needed by UI framework.
> > > >>>>> 3. Update the backend (stram) to serve the UI pages
> > > >>>>> 3. Extend the webage to display logical operator information.
> > > >>>>> 4. Extend the webage to display physical operator information
> > > >>>>> 5. Implement control operations on UI.
> > > >>>>>
> > > >>>>>
> > > >>>>> To implement above functionality, I propose ReactJS as UI
> framework
> > > and
> > > >>>>> MaterialUI (based on Google's material design) for
> > theme/controls/CSS
> > > >>>>> support.
> > > >>>>>
> > > >>>>> Please let me know what you think about the same.
> > > >>>>>
> > > >>>>> Also, please let know if anyone is interested in contributing to
> > this
> > > >>>>> feature.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Chinmay.
> > > >>>>>
> > > >>
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Vlad Rozov-2
If it is a set of static files that use existing REST API to display
information in a browser, where they will be hosted is irrelevant. They
can be hosted by stram, but it should be possible to host them
externally without making modificaitons. If this is the case, I don't
have any objections to the proposal.

Thank you,

Vlad

On 6/13/18 06:09, Chinmay Kolhatkar wrote:

> Hi Team,
>
> I agree with Thomas. The main focus of this proposal was to provide a UI
> layer using existing REST APIs.
> I think we should first focus on providing a meaningful UI served by AM to
> user using the same existing REST APIs.
>
> If in general, there is a agreement for the proposal, I can proceed with
> creating a JIRA and feature branch for the same.
>
> Thanks,
> Chinmay.
>
>
> On Wed, Jun 13, 2018 at 12:21 AM Thomas Weise <[hidden email]> wrote:
>
>> Looks like there is some disconnect in this thread, a few replies express
>> concerns that the proposal IMO doesn't imply.
>>
>> There is already a REST API that backs all of the CLI functionality. Modern
>> UIs run in the browser, and so what was proposed (unless I misunderstood)
>> would be based on the same  REST API and nothing extra. The only additional
>> thing the AM has to do is serve static files. And AFAIK it would actually
>> simplify the UI implementation when files and REST API have the same
>> origin.
>>
>> To me where files are coming from is actually a smaller piece and something
>> that can be augmented. But before proposing solutions outside of Apex or
>> additional deployment requirements, please consider the user experience. It
>> should be really simple to setup and use the UI.
>>
>> I would expect that a large part of the discussion evolves around the UI
>> functionality that helps the target persona.
>>
>> Thanks,
>> Thomas
>>
>> On Tue, Jun 12, 2018, 8:05 PM Shubhrajyoti Mohapatra <
>> [hidden email]>
>> wrote:
>>
>>> Hi Community,
>>>
>>> I would like to propose a slightly different approach on having an UI for
>>> Apex.
>>>
>>> The details are below:
>>>
>>>     1. This involves having an external web-server installed along side of
>>>     apexCli. We will basically mimic all the functionalities that apexCli
>>>     currently provides.
>>>     2. We will develop the necessary REST APIs for fetch/update operations
>>>     which apexCli currently supports.
>>>     3. Because most of the information and operations needed by a typical
>>>     user are already there in apexCli, it will of great help to end users
>>> and
>>>     which may increase the adoption rate.
>>>
>>> About Chinmay's proposal of having the STRAM hosting the apps, please
>> find
>>> my comments below:
>>>
>>>     1. It may increase load on STRAM if we were to provide real time
>> display
>>>     of information. I am aware that apexCli fetches info from STRAM apis
>> but
>>>     continuous polling can cause extra over-head.
>>>     2. The UI will have limited scope for the particular application.
>> Users
>>>     would definitely love to have a comprehensive UI for the management of
>>> Apex
>>>     Apps as a whole.
>>>     3. UI might be able to affect the whole application through STRAM
>> hence
>>>     giving direct access to STRAM APIs might not be good idea.
>>>     4. Because STRAM can't have a comprehensive UI hosted inside, anyways
>> we
>>>     will have to develop an external one.
>>>
>>> So I would like propose the plan on action like this.
>>>
>>>     1. We develop the wrapper UI for apexCli in the first phase and give
>>>     user all the management options as REST APIs.
>>>     2. If some of information we require are not in the current apexCli
>>>     implementation, we can add those feature to apexCli and if required to
>>>     STRAM APIs in subsequent phases.
>>>     3. In future we can even remove the REST server from STRAM if we could
>>>     provide something like MetricsStore functionality which was there in
>>>     DataTorrent RTS.
>>>
>>> --
>>> Thanks and Regards,
>>> Shubhrajyoti Mohapatra
>>>
>>>
>>>
>>>
>>> On Tue, Jun 12, 2018 at 2:49 AM Vlad Rozov <[hidden email]> wrote:
>>>
>>>> IMO it will be hard to provide UI that meets everyone expectation. An
>>>> external webUI can serve two goals
>>>>
>>>> 1. provide reference UI implementation
>>>> 2. ensure that REST API have all necessary entry points
>>>>
>>>> by embedding webUI directly into stram it will be tempting to shortcut
>>>> some calls to use stram classes directly.
>>>>
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>>> On 6/11/18 12:18, Pramod Immaneni wrote:
>>>>> I too think that a comprehensive UI works better standalone but given
>>>> that
>>>>> stram already runs a webapp server, provides web services and RM
>>>> provides a
>>>>> proxy with mapped url space for it, we can extend stram to provide
>>> better
>>>>> visual output. Overall I am +1 on the idea.
>>>>>
>>>>> On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <[hidden email]>
>> wrote:
>>>>>> Clarification: I am -0 on making UI part of the stram. I am +1 on
>>>>>> providing reference webUI application outside of the stram.
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Vlad
>>>>>>
>>>>>> On 6/9/18 07:05, Vlad Rozov wrote:
>>>>>>> -0: stram provides REST API to query runtime information and IMO
>> this
>>>>>>> is sufficient to build an external webUI application(s).
>>>>>>>
>>>>>>> Thank you,
>>>>>>>
>>>>>>> Vlad
>>>>>>>
>>>>>>> On 6/6/18 11:17, Ambarish Pande wrote:
>>>>>>>> +1 For this feature. It would be really helpfull for users to
>>>>>>>> visualize all
>>>>>>>> this information  and the DAG.
>>>>>>>>
>>>>>>>> +1 for ReactJS + MaterialUI
>>>>>>>>     A great choice, given  its flexibility and performance.
>>>>>>>>
>>>>>>>> I would like to contribute to this effort.
>>>>>>>>
>>>>>>>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <
>>> [hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Community,
>>>>>>>>>
>>>>>>>>> I want to propose a feature in apex-core to have a UI component
>> in
>>>>>>>>> stram.
>>>>>>>>> This is influenced by how basic runtime information is shown on
>> UI
>>> by
>>>>>>>>> spark.
>>>>>>>>>
>>>>>>>>> This includes following features:
>>>>>>>>> 1. Webpage will be hosted by stram and will be available for user
>>> to
>>>>>>>>> view
>>>>>>>>> in one of the two ways (I'll prefer approach b):
>>>>>>>>>       a. Hosted on a static port in which case we'll need to add a
>>> new
>>>>>>>>> servlet
>>>>>>>>> to stram
>>>>>>>>>       b. The webpage will be hosted on the same port as that of
>>> stram
>>>>>>>>> webservice but at a different path. Say, http://
>>>>>>>>> <stram_host>:<webservice_port>/ui
>>>>>>>>>
>>>>>>>>> 2. The webpage will be a static page and depending on the
>> framework
>>>> we
>>>>>>>>> chose, it can show the realtime metric data from stram.
>>>>>>>>>
>>>>>>>>> 3. There will be a categories of readonly information (maybe
>>>>>>>>> dynamically
>>>>>>>>> changing) that will be shown on the UI:
>>>>>>>>>       a. Application level information
>>>>>>>>>           i. Status
>>>>>>>>>           ii. Number of tuples processed/emitted
>>>>>>>>>           iii. Start time/ Running span
>>>>>>>>>           iv. Stram events
>>>>>>>>>           v. Total memory used/available
>>>>>>>>>           vi. Number of container allocated/deployed
>>>>>>>>>           v. Other information which is available in stram
>>>>>>>>>       b. Logical Operator level information
>>>>>>>>>           i. Status
>>>>>>>>>           ii. Number of tuples processed/emitted
>>>>>>>>>           iii. Important events related to logical operator
>>>>>>>>>           iv. Container list in which the operator is deployed
>>>>>>>>>           v. Any other information available in stram
>>>>>>>>>           vi. Logical DAG View
>>>>>>>>>       c. Container level information
>>>>>>>>>           i. Status
>>>>>>>>>           ii. Number of tuples processed/emitted
>>>>>>>>>           iii. Important events related to logical operator
>>>>>>>>>           iv. Any other information available in stram
>>>>>>>>>           v. Physical DAG View
>>>>>>>>>
>>>>>>>>> 4. In second phase, there will be control related operations
>>> allowed
>>>>>>>>> from
>>>>>>>>> the UI, as follows:
>>>>>>>>>       a. Stop/Kill application
>>>>>>>>>       b. Stop/Kill containers
>>>>>>>>>       c. Stack trace dump of containers
>>>>>>>>>       d. Logs related?
>>>>>>>>>       d. etc...??
>>>>>>>>>
>>>>>>>>> The above implementation can be done in phases as follows:
>>>>>>>>> 1. Have webpage display application level information only
>>>> (read-only).
>>>>>>>>> Static display first (i.e. page needs to refresh to see latest
>>>>>>>>> data), next
>>>>>>>>> dynamically changing data.
>>>>>>>>> 2. Have jenkins build tools updated as needed by UI framework.
>>>>>>>>> 3. Update the backend (stram) to serve the UI pages
>>>>>>>>> 3. Extend the webage to display logical operator information.
>>>>>>>>> 4. Extend the webage to display physical operator information
>>>>>>>>> 5. Implement control operations on UI.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> To implement above functionality, I propose ReactJS as UI
>> framework
>>>> and
>>>>>>>>> MaterialUI (based on Google's material design) for
>>> theme/controls/CSS
>>>>>>>>> support.
>>>>>>>>>
>>>>>>>>> Please let me know what you think about the same.
>>>>>>>>>
>>>>>>>>> Also, please let know if anyone is interested in contributing to
>>> this
>>>>>>>>> feature.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Chinmay.
>>>>>>>>>
>>>>

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

Pramod Immaneni-3
I don't think they would be reusable as is without any changes externally
as the application context would not be available. Also externally you
would want to have a different kind of application that aggregates across
multiple applications.

Thanks

On Thu, Jun 14, 2018 at 10:24 AM Vlad Rozov <[hidden email]> wrote:

> If it is a set of static files that use existing REST API to display
> information in a browser, where they will be hosted is irrelevant. They
> can be hosted by stram, but it should be possible to host them
> externally without making modificaitons. If this is the case, I don't
> have any objections to the proposal.
>
> Thank you,
>
> Vlad
>
> On 6/13/18 06:09, Chinmay Kolhatkar wrote:
> > Hi Team,
> >
> > I agree with Thomas. The main focus of this proposal was to provide a UI
> > layer using existing REST APIs.
> > I think we should first focus on providing a meaningful UI served by AM
> to
> > user using the same existing REST APIs.
> >
> > If in general, there is a agreement for the proposal, I can proceed with
> > creating a JIRA and feature branch for the same.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Wed, Jun 13, 2018 at 12:21 AM Thomas Weise <[hidden email]> wrote:
> >
> >> Looks like there is some disconnect in this thread, a few replies
> express
> >> concerns that the proposal IMO doesn't imply.
> >>
> >> There is already a REST API that backs all of the CLI functionality.
> Modern
> >> UIs run in the browser, and so what was proposed (unless I
> misunderstood)
> >> would be based on the same  REST API and nothing extra. The only
> additional
> >> thing the AM has to do is serve static files. And AFAIK it would
> actually
> >> simplify the UI implementation when files and REST API have the same
> >> origin.
> >>
> >> To me where files are coming from is actually a smaller piece and
> something
> >> that can be augmented. But before proposing solutions outside of Apex or
> >> additional deployment requirements, please consider the user
> experience. It
> >> should be really simple to setup and use the UI.
> >>
> >> I would expect that a large part of the discussion evolves around the UI
> >> functionality that helps the target persona.
> >>
> >> Thanks,
> >> Thomas
> >>
> >> On Tue, Jun 12, 2018, 8:05 PM Shubhrajyoti Mohapatra <
> >> [hidden email]>
> >> wrote:
> >>
> >>> Hi Community,
> >>>
> >>> I would like to propose a slightly different approach on having an UI
> for
> >>> Apex.
> >>>
> >>> The details are below:
> >>>
> >>>     1. This involves having an external web-server installed along
> side of
> >>>     apexCli. We will basically mimic all the functionalities that
> apexCli
> >>>     currently provides.
> >>>     2. We will develop the necessary REST APIs for fetch/update
> operations
> >>>     which apexCli currently supports.
> >>>     3. Because most of the information and operations needed by a
> typical
> >>>     user are already there in apexCli, it will of great help to end
> users
> >>> and
> >>>     which may increase the adoption rate.
> >>>
> >>> About Chinmay's proposal of having the STRAM hosting the apps, please
> >> find
> >>> my comments below:
> >>>
> >>>     1. It may increase load on STRAM if we were to provide real time
> >> display
> >>>     of information. I am aware that apexCli fetches info from STRAM
> apis
> >> but
> >>>     continuous polling can cause extra over-head.
> >>>     2. The UI will have limited scope for the particular application.
> >> Users
> >>>     would definitely love to have a comprehensive UI for the
> management of
> >>> Apex
> >>>     Apps as a whole.
> >>>     3. UI might be able to affect the whole application through STRAM
> >> hence
> >>>     giving direct access to STRAM APIs might not be good idea.
> >>>     4. Because STRAM can't have a comprehensive UI hosted inside,
> anyways
> >> we
> >>>     will have to develop an external one.
> >>>
> >>> So I would like propose the plan on action like this.
> >>>
> >>>     1. We develop the wrapper UI for apexCli in the first phase and
> give
> >>>     user all the management options as REST APIs.
> >>>     2. If some of information we require are not in the current apexCli
> >>>     implementation, we can add those feature to apexCli and if
> required to
> >>>     STRAM APIs in subsequent phases.
> >>>     3. In future we can even remove the REST server from STRAM if we
> could
> >>>     provide something like MetricsStore functionality which was there
> in
> >>>     DataTorrent RTS.
> >>>
> >>> --
> >>> Thanks and Regards,
> >>> Shubhrajyoti Mohapatra
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Jun 12, 2018 at 2:49 AM Vlad Rozov <[hidden email]> wrote:
> >>>
> >>>> IMO it will be hard to provide UI that meets everyone expectation. An
> >>>> external webUI can serve two goals
> >>>>
> >>>> 1. provide reference UI implementation
> >>>> 2. ensure that REST API have all necessary entry points
> >>>>
> >>>> by embedding webUI directly into stram it will be tempting to shortcut
> >>>> some calls to use stram classes directly.
> >>>>
> >>>> Thank you,
> >>>>
> >>>> Vlad
> >>>>
> >>>> On 6/11/18 12:18, Pramod Immaneni wrote:
> >>>>> I too think that a comprehensive UI works better standalone but given
> >>>> that
> >>>>> stram already runs a webapp server, provides web services and RM
> >>>> provides a
> >>>>> proxy with mapped url space for it, we can extend stram to provide
> >>> better
> >>>>> visual output. Overall I am +1 on the idea.
> >>>>>
> >>>>> On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <[hidden email]>
> >> wrote:
> >>>>>> Clarification: I am -0 on making UI part of the stram. I am +1 on
> >>>>>> providing reference webUI application outside of the stram.
> >>>>>>
> >>>>>> Thank you,
> >>>>>>
> >>>>>> Vlad
> >>>>>>
> >>>>>> On 6/9/18 07:05, Vlad Rozov wrote:
> >>>>>>> -0: stram provides REST API to query runtime information and IMO
> >> this
> >>>>>>> is sufficient to build an external webUI application(s).
> >>>>>>>
> >>>>>>> Thank you,
> >>>>>>>
> >>>>>>> Vlad
> >>>>>>>
> >>>>>>> On 6/6/18 11:17, Ambarish Pande wrote:
> >>>>>>>> +1 For this feature. It would be really helpfull for users to
> >>>>>>>> visualize all
> >>>>>>>> this information  and the DAG.
> >>>>>>>>
> >>>>>>>> +1 for ReactJS + MaterialUI
> >>>>>>>>     A great choice, given  its flexibility and performance.
> >>>>>>>>
> >>>>>>>> I would like to contribute to this effort.
> >>>>>>>>
> >>>>>>>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <
> >>> [hidden email]>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Community,
> >>>>>>>>>
> >>>>>>>>> I want to propose a feature in apex-core to have a UI component
> >> in
> >>>>>>>>> stram.
> >>>>>>>>> This is influenced by how basic runtime information is shown on
> >> UI
> >>> by
> >>>>>>>>> spark.
> >>>>>>>>>
> >>>>>>>>> This includes following features:
> >>>>>>>>> 1. Webpage will be hosted by stram and will be available for user
> >>> to
> >>>>>>>>> view
> >>>>>>>>> in one of the two ways (I'll prefer approach b):
> >>>>>>>>>       a. Hosted on a static port in which case we'll need to add
> a
> >>> new
> >>>>>>>>> servlet
> >>>>>>>>> to stram
> >>>>>>>>>       b. The webpage will be hosted on the same port as that of
> >>> stram
> >>>>>>>>> webservice but at a different path. Say, http://
> >>>>>>>>> <stram_host>:<webservice_port>/ui
> >>>>>>>>>
> >>>>>>>>> 2. The webpage will be a static page and depending on the
> >> framework
> >>>> we
> >>>>>>>>> chose, it can show the realtime metric data from stram.
> >>>>>>>>>
> >>>>>>>>> 3. There will be a categories of readonly information (maybe
> >>>>>>>>> dynamically
> >>>>>>>>> changing) that will be shown on the UI:
> >>>>>>>>>       a. Application level information
> >>>>>>>>>           i. Status
> >>>>>>>>>           ii. Number of tuples processed/emitted
> >>>>>>>>>           iii. Start time/ Running span
> >>>>>>>>>           iv. Stram events
> >>>>>>>>>           v. Total memory used/available
> >>>>>>>>>           vi. Number of container allocated/deployed
> >>>>>>>>>           v. Other information which is available in stram
> >>>>>>>>>       b. Logical Operator level information
> >>>>>>>>>           i. Status
> >>>>>>>>>           ii. Number of tuples processed/emitted
> >>>>>>>>>           iii. Important events related to logical operator
> >>>>>>>>>           iv. Container list in which the operator is deployed
> >>>>>>>>>           v. Any other information available in stram
> >>>>>>>>>           vi. Logical DAG View
> >>>>>>>>>       c. Container level information
> >>>>>>>>>           i. Status
> >>>>>>>>>           ii. Number of tuples processed/emitted
> >>>>>>>>>           iii. Important events related to logical operator
> >>>>>>>>>           iv. Any other information available in stram
> >>>>>>>>>           v. Physical DAG View
> >>>>>>>>>
> >>>>>>>>> 4. In second phase, there will be control related operations
> >>> allowed
> >>>>>>>>> from
> >>>>>>>>> the UI, as follows:
> >>>>>>>>>       a. Stop/Kill application
> >>>>>>>>>       b. Stop/Kill containers
> >>>>>>>>>       c. Stack trace dump of containers
> >>>>>>>>>       d. Logs related?
> >>>>>>>>>       d. etc...??
> >>>>>>>>>
> >>>>>>>>> The above implementation can be done in phases as follows:
> >>>>>>>>> 1. Have webpage display application level information only
> >>>> (read-only).
> >>>>>>>>> Static display first (i.e. page needs to refresh to see latest
> >>>>>>>>> data), next
> >>>>>>>>> dynamically changing data.
> >>>>>>>>> 2. Have jenkins build tools updated as needed by UI framework.
> >>>>>>>>> 3. Update the backend (stram) to serve the UI pages
> >>>>>>>>> 3. Extend the webage to display logical operator information.
> >>>>>>>>> 4. Extend the webage to display physical operator information
> >>>>>>>>> 5. Implement control operations on UI.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> To implement above functionality, I propose ReactJS as UI
> >> framework
> >>>> and
> >>>>>>>>> MaterialUI (based on Google's material design) for
> >>> theme/controls/CSS
> >>>>>>>>> support.
> >>>>>>>>>
> >>>>>>>>> Please let me know what you think about the same.
> >>>>>>>>>
> >>>>>>>>> Also, please let know if anyone is interested in contributing to
> >>> this
> >>>>>>>>> feature.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Chinmay.
> >>>>>>>>>
> >>>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

chinmay
Administrator
So here is how it'll be:
1. ReactJS application can be compiled for production builds. The output is
set of html+css+compressed js files. It can also output a war file directly
(I think).
2. The react app will have a configuration switch to allow user to set the
baseUrl. Default value will be "/ws/v2".
3. These files can be bundled with stram so that it can be hosted by stram.
In this case the base url for reading data will be "/ws/v2".. i.e. read
from localhost.
3. OR the config switch be set "http://path/to/stram/ws/v2" using which one
can host it seperately.

Basically, it can be reusable with configuration changes during/after build
time.

Hope that helps.

Thanks,
Chinmay.

On Thu, Jun 14, 2018 at 10:09 PM Pramod Immaneni <[hidden email]>
wrote:

> I don't think they would be reusable as is without any changes externally
> as the application context would not be available. Also externally you
> would want to have a different kind of application that aggregates across
> multiple applications.
>
> Thanks
>
> On Thu, Jun 14, 2018 at 10:24 AM Vlad Rozov <[hidden email]> wrote:
>
> > If it is a set of static files that use existing REST API to display
> > information in a browser, where they will be hosted is irrelevant. They
> > can be hosted by stram, but it should be possible to host them
> > externally without making modificaitons. If this is the case, I don't
> > have any objections to the proposal.
> >
> > Thank you,
> >
> > Vlad
> >
> > On 6/13/18 06:09, Chinmay Kolhatkar wrote:
> > > Hi Team,
> > >
> > > I agree with Thomas. The main focus of this proposal was to provide a
> UI
> > > layer using existing REST APIs.
> > > I think we should first focus on providing a meaningful UI served by AM
> > to
> > > user using the same existing REST APIs.
> > >
> > > If in general, there is a agreement for the proposal, I can proceed
> with
> > > creating a JIRA and feature branch for the same.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > > On Wed, Jun 13, 2018 at 12:21 AM Thomas Weise <[hidden email]> wrote:
> > >
> > >> Looks like there is some disconnect in this thread, a few replies
> > express
> > >> concerns that the proposal IMO doesn't imply.
> > >>
> > >> There is already a REST API that backs all of the CLI functionality.
> > Modern
> > >> UIs run in the browser, and so what was proposed (unless I
> > misunderstood)
> > >> would be based on the same  REST API and nothing extra. The only
> > additional
> > >> thing the AM has to do is serve static files. And AFAIK it would
> > actually
> > >> simplify the UI implementation when files and REST API have the same
> > >> origin.
> > >>
> > >> To me where files are coming from is actually a smaller piece and
> > something
> > >> that can be augmented. But before proposing solutions outside of Apex
> or
> > >> additional deployment requirements, please consider the user
> > experience. It
> > >> should be really simple to setup and use the UI.
> > >>
> > >> I would expect that a large part of the discussion evolves around the
> UI
> > >> functionality that helps the target persona.
> > >>
> > >> Thanks,
> > >> Thomas
> > >>
> > >> On Tue, Jun 12, 2018, 8:05 PM Shubhrajyoti Mohapatra <
> > >> [hidden email]>
> > >> wrote:
> > >>
> > >>> Hi Community,
> > >>>
> > >>> I would like to propose a slightly different approach on having an UI
> > for
> > >>> Apex.
> > >>>
> > >>> The details are below:
> > >>>
> > >>>     1. This involves having an external web-server installed along
> > side of
> > >>>     apexCli. We will basically mimic all the functionalities that
> > apexCli
> > >>>     currently provides.
> > >>>     2. We will develop the necessary REST APIs for fetch/update
> > operations
> > >>>     which apexCli currently supports.
> > >>>     3. Because most of the information and operations needed by a
> > typical
> > >>>     user are already there in apexCli, it will of great help to end
> > users
> > >>> and
> > >>>     which may increase the adoption rate.
> > >>>
> > >>> About Chinmay's proposal of having the STRAM hosting the apps, please
> > >> find
> > >>> my comments below:
> > >>>
> > >>>     1. It may increase load on STRAM if we were to provide real time
> > >> display
> > >>>     of information. I am aware that apexCli fetches info from STRAM
> > apis
> > >> but
> > >>>     continuous polling can cause extra over-head.
> > >>>     2. The UI will have limited scope for the particular application.
> > >> Users
> > >>>     would definitely love to have a comprehensive UI for the
> > management of
> > >>> Apex
> > >>>     Apps as a whole.
> > >>>     3. UI might be able to affect the whole application through STRAM
> > >> hence
> > >>>     giving direct access to STRAM APIs might not be good idea.
> > >>>     4. Because STRAM can't have a comprehensive UI hosted inside,
> > anyways
> > >> we
> > >>>     will have to develop an external one.
> > >>>
> > >>> So I would like propose the plan on action like this.
> > >>>
> > >>>     1. We develop the wrapper UI for apexCli in the first phase and
> > give
> > >>>     user all the management options as REST APIs.
> > >>>     2. If some of information we require are not in the current
> apexCli
> > >>>     implementation, we can add those feature to apexCli and if
> > required to
> > >>>     STRAM APIs in subsequent phases.
> > >>>     3. In future we can even remove the REST server from STRAM if we
> > could
> > >>>     provide something like MetricsStore functionality which was there
> > in
> > >>>     DataTorrent RTS.
> > >>>
> > >>> --
> > >>> Thanks and Regards,
> > >>> Shubhrajyoti Mohapatra
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Tue, Jun 12, 2018 at 2:49 AM Vlad Rozov <[hidden email]>
> wrote:
> > >>>
> > >>>> IMO it will be hard to provide UI that meets everyone expectation.
> An
> > >>>> external webUI can serve two goals
> > >>>>
> > >>>> 1. provide reference UI implementation
> > >>>> 2. ensure that REST API have all necessary entry points
> > >>>>
> > >>>> by embedding webUI directly into stram it will be tempting to
> shortcut
> > >>>> some calls to use stram classes directly.
> > >>>>
> > >>>> Thank you,
> > >>>>
> > >>>> Vlad
> > >>>>
> > >>>> On 6/11/18 12:18, Pramod Immaneni wrote:
> > >>>>> I too think that a comprehensive UI works better standalone but
> given
> > >>>> that
> > >>>>> stram already runs a webapp server, provides web services and RM
> > >>>> provides a
> > >>>>> proxy with mapped url space for it, we can extend stram to provide
> > >>> better
> > >>>>> visual output. Overall I am +1 on the idea.
> > >>>>>
> > >>>>> On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <[hidden email]>
> > >> wrote:
> > >>>>>> Clarification: I am -0 on making UI part of the stram. I am +1 on
> > >>>>>> providing reference webUI application outside of the stram.
> > >>>>>>
> > >>>>>> Thank you,
> > >>>>>>
> > >>>>>> Vlad
> > >>>>>>
> > >>>>>> On 6/9/18 07:05, Vlad Rozov wrote:
> > >>>>>>> -0: stram provides REST API to query runtime information and IMO
> > >> this
> > >>>>>>> is sufficient to build an external webUI application(s).
> > >>>>>>>
> > >>>>>>> Thank you,
> > >>>>>>>
> > >>>>>>> Vlad
> > >>>>>>>
> > >>>>>>> On 6/6/18 11:17, Ambarish Pande wrote:
> > >>>>>>>> +1 For this feature. It would be really helpfull for users to
> > >>>>>>>> visualize all
> > >>>>>>>> this information  and the DAG.
> > >>>>>>>>
> > >>>>>>>> +1 for ReactJS + MaterialUI
> > >>>>>>>>     A great choice, given  its flexibility and performance.
> > >>>>>>>>
> > >>>>>>>> I would like to contribute to this effort.
> > >>>>>>>>
> > >>>>>>>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <
> > >>> [hidden email]>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Community,
> > >>>>>>>>>
> > >>>>>>>>> I want to propose a feature in apex-core to have a UI component
> > >> in
> > >>>>>>>>> stram.
> > >>>>>>>>> This is influenced by how basic runtime information is shown on
> > >> UI
> > >>> by
> > >>>>>>>>> spark.
> > >>>>>>>>>
> > >>>>>>>>> This includes following features:
> > >>>>>>>>> 1. Webpage will be hosted by stram and will be available for
> user
> > >>> to
> > >>>>>>>>> view
> > >>>>>>>>> in one of the two ways (I'll prefer approach b):
> > >>>>>>>>>       a. Hosted on a static port in which case we'll need to
> add
> > a
> > >>> new
> > >>>>>>>>> servlet
> > >>>>>>>>> to stram
> > >>>>>>>>>       b. The webpage will be hosted on the same port as that of
> > >>> stram
> > >>>>>>>>> webservice but at a different path. Say, http://
> > >>>>>>>>> <stram_host>:<webservice_port>/ui
> > >>>>>>>>>
> > >>>>>>>>> 2. The webpage will be a static page and depending on the
> > >> framework
> > >>>> we
> > >>>>>>>>> chose, it can show the realtime metric data from stram.
> > >>>>>>>>>
> > >>>>>>>>> 3. There will be a categories of readonly information (maybe
> > >>>>>>>>> dynamically
> > >>>>>>>>> changing) that will be shown on the UI:
> > >>>>>>>>>       a. Application level information
> > >>>>>>>>>           i. Status
> > >>>>>>>>>           ii. Number of tuples processed/emitted
> > >>>>>>>>>           iii. Start time/ Running span
> > >>>>>>>>>           iv. Stram events
> > >>>>>>>>>           v. Total memory used/available
> > >>>>>>>>>           vi. Number of container allocated/deployed
> > >>>>>>>>>           v. Other information which is available in stram
> > >>>>>>>>>       b. Logical Operator level information
> > >>>>>>>>>           i. Status
> > >>>>>>>>>           ii. Number of tuples processed/emitted
> > >>>>>>>>>           iii. Important events related to logical operator
> > >>>>>>>>>           iv. Container list in which the operator is deployed
> > >>>>>>>>>           v. Any other information available in stram
> > >>>>>>>>>           vi. Logical DAG View
> > >>>>>>>>>       c. Container level information
> > >>>>>>>>>           i. Status
> > >>>>>>>>>           ii. Number of tuples processed/emitted
> > >>>>>>>>>           iii. Important events related to logical operator
> > >>>>>>>>>           iv. Any other information available in stram
> > >>>>>>>>>           v. Physical DAG View
> > >>>>>>>>>
> > >>>>>>>>> 4. In second phase, there will be control related operations
> > >>> allowed
> > >>>>>>>>> from
> > >>>>>>>>> the UI, as follows:
> > >>>>>>>>>       a. Stop/Kill application
> > >>>>>>>>>       b. Stop/Kill containers
> > >>>>>>>>>       c. Stack trace dump of containers
> > >>>>>>>>>       d. Logs related?
> > >>>>>>>>>       d. etc...??
> > >>>>>>>>>
> > >>>>>>>>> The above implementation can be done in phases as follows:
> > >>>>>>>>> 1. Have webpage display application level information only
> > >>>> (read-only).
> > >>>>>>>>> Static display first (i.e. page needs to refresh to see latest
> > >>>>>>>>> data), next
> > >>>>>>>>> dynamically changing data.
> > >>>>>>>>> 2. Have jenkins build tools updated as needed by UI framework.
> > >>>>>>>>> 3. Update the backend (stram) to serve the UI pages
> > >>>>>>>>> 3. Extend the webage to display logical operator information.
> > >>>>>>>>> 4. Extend the webage to display physical operator information
> > >>>>>>>>> 5. Implement control operations on UI.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> To implement above functionality, I propose ReactJS as UI
> > >> framework
> > >>>> and
> > >>>>>>>>> MaterialUI (based on Google's material design) for
> > >>> theme/controls/CSS
> > >>>>>>>>> support.
> > >>>>>>>>>
> > >>>>>>>>> Please let me know what you think about the same.
> > >>>>>>>>>
> > >>>>>>>>> Also, please let know if anyone is interested in contributing
> to
> > >>> this
> > >>>>>>>>> feature.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Chinmay.
> > >>>>>>>>>
> > >>>>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

chinmay
Administrator
Hi All,

I've created following jiras encompassing the requirements (To be picked up
in order):
https://issues.apache.org/jira/browse/APEXMALHAR-2558
https://issues.apache.org/jira/browse/APEXMALHAR-2559

I'll start adding subtasks for subitems mentioned in the jira.

I'll also create the feature branch shortly so that we can start working
against it.

-Chinmay.



On Sat, Jun 16, 2018 at 7:07 PM Chinmay Kolhatkar <[hidden email]>
wrote:

> So here is how it'll be:
> 1. ReactJS application can be compiled for production builds. The output
> is set of html+css+compressed js files. It can also output a war file
> directly (I think).
> 2. The react app will have a configuration switch to allow user to set the
> baseUrl. Default value will be "/ws/v2".
> 3. These files can be bundled with stram so that it can be hosted by
> stram. In this case the base url for reading data will be "/ws/v2".. i.e.
> read from localhost.
> 3. OR the config switch be set "http://path/to/stram/ws/v2" using which
> one can host it seperately.
>
> Basically, it can be reusable with configuration changes during/after
> build time.
>
> Hope that helps.
>
> Thanks,
> Chinmay.
>
> On Thu, Jun 14, 2018 at 10:09 PM Pramod Immaneni <
> [hidden email]> wrote:
>
>> I don't think they would be reusable as is without any changes externally
>> as the application context would not be available. Also externally you
>> would want to have a different kind of application that aggregates across
>> multiple applications.
>>
>> Thanks
>>
>> On Thu, Jun 14, 2018 at 10:24 AM Vlad Rozov <[hidden email]> wrote:
>>
>> > If it is a set of static files that use existing REST API to display
>> > information in a browser, where they will be hosted is irrelevant. They
>> > can be hosted by stram, but it should be possible to host them
>> > externally without making modificaitons. If this is the case, I don't
>> > have any objections to the proposal.
>> >
>> > Thank you,
>> >
>> > Vlad
>> >
>> > On 6/13/18 06:09, Chinmay Kolhatkar wrote:
>> > > Hi Team,
>> > >
>> > > I agree with Thomas. The main focus of this proposal was to provide a
>> UI
>> > > layer using existing REST APIs.
>> > > I think we should first focus on providing a meaningful UI served by
>> AM
>> > to
>> > > user using the same existing REST APIs.
>> > >
>> > > If in general, there is a agreement for the proposal, I can proceed
>> with
>> > > creating a JIRA and feature branch for the same.
>> > >
>> > > Thanks,
>> > > Chinmay.
>> > >
>> > >
>> > > On Wed, Jun 13, 2018 at 12:21 AM Thomas Weise <[hidden email]> wrote:
>> > >
>> > >> Looks like there is some disconnect in this thread, a few replies
>> > express
>> > >> concerns that the proposal IMO doesn't imply.
>> > >>
>> > >> There is already a REST API that backs all of the CLI functionality.
>> > Modern
>> > >> UIs run in the browser, and so what was proposed (unless I
>> > misunderstood)
>> > >> would be based on the same  REST API and nothing extra. The only
>> > additional
>> > >> thing the AM has to do is serve static files. And AFAIK it would
>> > actually
>> > >> simplify the UI implementation when files and REST API have the same
>> > >> origin.
>> > >>
>> > >> To me where files are coming from is actually a smaller piece and
>> > something
>> > >> that can be augmented. But before proposing solutions outside of
>> Apex or
>> > >> additional deployment requirements, please consider the user
>> > experience. It
>> > >> should be really simple to setup and use the UI.
>> > >>
>> > >> I would expect that a large part of the discussion evolves around
>> the UI
>> > >> functionality that helps the target persona.
>> > >>
>> > >> Thanks,
>> > >> Thomas
>> > >>
>> > >> On Tue, Jun 12, 2018, 8:05 PM Shubhrajyoti Mohapatra <
>> > >> [hidden email]>
>> > >> wrote:
>> > >>
>> > >>> Hi Community,
>> > >>>
>> > >>> I would like to propose a slightly different approach on having an
>> UI
>> > for
>> > >>> Apex.
>> > >>>
>> > >>> The details are below:
>> > >>>
>> > >>>     1. This involves having an external web-server installed along
>> > side of
>> > >>>     apexCli. We will basically mimic all the functionalities that
>> > apexCli
>> > >>>     currently provides.
>> > >>>     2. We will develop the necessary REST APIs for fetch/update
>> > operations
>> > >>>     which apexCli currently supports.
>> > >>>     3. Because most of the information and operations needed by a
>> > typical
>> > >>>     user are already there in apexCli, it will of great help to end
>> > users
>> > >>> and
>> > >>>     which may increase the adoption rate.
>> > >>>
>> > >>> About Chinmay's proposal of having the STRAM hosting the apps,
>> please
>> > >> find
>> > >>> my comments below:
>> > >>>
>> > >>>     1. It may increase load on STRAM if we were to provide real time
>> > >> display
>> > >>>     of information. I am aware that apexCli fetches info from STRAM
>> > apis
>> > >> but
>> > >>>     continuous polling can cause extra over-head.
>> > >>>     2. The UI will have limited scope for the particular
>> application.
>> > >> Users
>> > >>>     would definitely love to have a comprehensive UI for the
>> > management of
>> > >>> Apex
>> > >>>     Apps as a whole.
>> > >>>     3. UI might be able to affect the whole application through
>> STRAM
>> > >> hence
>> > >>>     giving direct access to STRAM APIs might not be good idea.
>> > >>>     4. Because STRAM can't have a comprehensive UI hosted inside,
>> > anyways
>> > >> we
>> > >>>     will have to develop an external one.
>> > >>>
>> > >>> So I would like propose the plan on action like this.
>> > >>>
>> > >>>     1. We develop the wrapper UI for apexCli in the first phase and
>> > give
>> > >>>     user all the management options as REST APIs.
>> > >>>     2. If some of information we require are not in the current
>> apexCli
>> > >>>     implementation, we can add those feature to apexCli and if
>> > required to
>> > >>>     STRAM APIs in subsequent phases.
>> > >>>     3. In future we can even remove the REST server from STRAM if we
>> > could
>> > >>>     provide something like MetricsStore functionality which was
>> there
>> > in
>> > >>>     DataTorrent RTS.
>> > >>>
>> > >>> --
>> > >>> Thanks and Regards,
>> > >>> Shubhrajyoti Mohapatra
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Tue, Jun 12, 2018 at 2:49 AM Vlad Rozov <[hidden email]>
>> wrote:
>> > >>>
>> > >>>> IMO it will be hard to provide UI that meets everyone expectation.
>> An
>> > >>>> external webUI can serve two goals
>> > >>>>
>> > >>>> 1. provide reference UI implementation
>> > >>>> 2. ensure that REST API have all necessary entry points
>> > >>>>
>> > >>>> by embedding webUI directly into stram it will be tempting to
>> shortcut
>> > >>>> some calls to use stram classes directly.
>> > >>>>
>> > >>>> Thank you,
>> > >>>>
>> > >>>> Vlad
>> > >>>>
>> > >>>> On 6/11/18 12:18, Pramod Immaneni wrote:
>> > >>>>> I too think that a comprehensive UI works better standalone but
>> given
>> > >>>> that
>> > >>>>> stram already runs a webapp server, provides web services and RM
>> > >>>> provides a
>> > >>>>> proxy with mapped url space for it, we can extend stram to provide
>> > >>> better
>> > >>>>> visual output. Overall I am +1 on the idea.
>> > >>>>>
>> > >>>>> On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <[hidden email]>
>> > >> wrote:
>> > >>>>>> Clarification: I am -0 on making UI part of the stram. I am +1 on
>> > >>>>>> providing reference webUI application outside of the stram.
>> > >>>>>>
>> > >>>>>> Thank you,
>> > >>>>>>
>> > >>>>>> Vlad
>> > >>>>>>
>> > >>>>>> On 6/9/18 07:05, Vlad Rozov wrote:
>> > >>>>>>> -0: stram provides REST API to query runtime information and IMO
>> > >> this
>> > >>>>>>> is sufficient to build an external webUI application(s).
>> > >>>>>>>
>> > >>>>>>> Thank you,
>> > >>>>>>>
>> > >>>>>>> Vlad
>> > >>>>>>>
>> > >>>>>>> On 6/6/18 11:17, Ambarish Pande wrote:
>> > >>>>>>>> +1 For this feature. It would be really helpfull for users to
>> > >>>>>>>> visualize all
>> > >>>>>>>> this information  and the DAG.
>> > >>>>>>>>
>> > >>>>>>>> +1 for ReactJS + MaterialUI
>> > >>>>>>>>     A great choice, given  its flexibility and performance.
>> > >>>>>>>>
>> > >>>>>>>> I would like to contribute to this effort.
>> > >>>>>>>>
>> > >>>>>>>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <
>> > >>> [hidden email]>
>> > >>>>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Hi Community,
>> > >>>>>>>>>
>> > >>>>>>>>> I want to propose a feature in apex-core to have a UI
>> component
>> > >> in
>> > >>>>>>>>> stram.
>> > >>>>>>>>> This is influenced by how basic runtime information is shown
>> on
>> > >> UI
>> > >>> by
>> > >>>>>>>>> spark.
>> > >>>>>>>>>
>> > >>>>>>>>> This includes following features:
>> > >>>>>>>>> 1. Webpage will be hosted by stram and will be available for
>> user
>> > >>> to
>> > >>>>>>>>> view
>> > >>>>>>>>> in one of the two ways (I'll prefer approach b):
>> > >>>>>>>>>       a. Hosted on a static port in which case we'll need to
>> add
>> > a
>> > >>> new
>> > >>>>>>>>> servlet
>> > >>>>>>>>> to stram
>> > >>>>>>>>>       b. The webpage will be hosted on the same port as that
>> of
>> > >>> stram
>> > >>>>>>>>> webservice but at a different path. Say, http://
>> > >>>>>>>>> <stram_host>:<webservice_port>/ui
>> > >>>>>>>>>
>> > >>>>>>>>> 2. The webpage will be a static page and depending on the
>> > >> framework
>> > >>>> we
>> > >>>>>>>>> chose, it can show the realtime metric data from stram.
>> > >>>>>>>>>
>> > >>>>>>>>> 3. There will be a categories of readonly information (maybe
>> > >>>>>>>>> dynamically
>> > >>>>>>>>> changing) that will be shown on the UI:
>> > >>>>>>>>>       a. Application level information
>> > >>>>>>>>>           i. Status
>> > >>>>>>>>>           ii. Number of tuples processed/emitted
>> > >>>>>>>>>           iii. Start time/ Running span
>> > >>>>>>>>>           iv. Stram events
>> > >>>>>>>>>           v. Total memory used/available
>> > >>>>>>>>>           vi. Number of container allocated/deployed
>> > >>>>>>>>>           v. Other information which is available in stram
>> > >>>>>>>>>       b. Logical Operator level information
>> > >>>>>>>>>           i. Status
>> > >>>>>>>>>           ii. Number of tuples processed/emitted
>> > >>>>>>>>>           iii. Important events related to logical operator
>> > >>>>>>>>>           iv. Container list in which the operator is deployed
>> > >>>>>>>>>           v. Any other information available in stram
>> > >>>>>>>>>           vi. Logical DAG View
>> > >>>>>>>>>       c. Container level information
>> > >>>>>>>>>           i. Status
>> > >>>>>>>>>           ii. Number of tuples processed/emitted
>> > >>>>>>>>>           iii. Important events related to logical operator
>> > >>>>>>>>>           iv. Any other information available in stram
>> > >>>>>>>>>           v. Physical DAG View
>> > >>>>>>>>>
>> > >>>>>>>>> 4. In second phase, there will be control related operations
>> > >>> allowed
>> > >>>>>>>>> from
>> > >>>>>>>>> the UI, as follows:
>> > >>>>>>>>>       a. Stop/Kill application
>> > >>>>>>>>>       b. Stop/Kill containers
>> > >>>>>>>>>       c. Stack trace dump of containers
>> > >>>>>>>>>       d. Logs related?
>> > >>>>>>>>>       d. etc...??
>> > >>>>>>>>>
>> > >>>>>>>>> The above implementation can be done in phases as follows:
>> > >>>>>>>>> 1. Have webpage display application level information only
>> > >>>> (read-only).
>> > >>>>>>>>> Static display first (i.e. page needs to refresh to see latest
>> > >>>>>>>>> data), next
>> > >>>>>>>>> dynamically changing data.
>> > >>>>>>>>> 2. Have jenkins build tools updated as needed by UI framework.
>> > >>>>>>>>> 3. Update the backend (stram) to serve the UI pages
>> > >>>>>>>>> 3. Extend the webage to display logical operator information.
>> > >>>>>>>>> 4. Extend the webage to display physical operator information
>> > >>>>>>>>> 5. Implement control operations on UI.
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> To implement above functionality, I propose ReactJS as UI
>> > >> framework
>> > >>>> and
>> > >>>>>>>>> MaterialUI (based on Google's material design) for
>> > >>> theme/controls/CSS
>> > >>>>>>>>> support.
>> > >>>>>>>>>
>> > >>>>>>>>> Please let me know what you think about the same.
>> > >>>>>>>>>
>> > >>>>>>>>> Also, please let know if anyone is interested in contributing
>> to
>> > >>> this
>> > >>>>>>>>> feature.
>> > >>>>>>>>>
>> > >>>>>>>>> Thanks,
>> > >>>>>>>>> Chinmay.
>> > >>>>>>>>>
>> > >>>>
>> >
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for UI component in stram

chinmay
Administrator
Added the subtasks. I just realized that by mistake I created the jiras in
APEXMALHAR instead of APEXCORE. Is there a way to move the jiras to
different project?
Let me know otherwise, I'll close these jiras and recreate new ones in
APEXCORE.

Thanks,
Chinmay.


On Sat, Jun 16, 2018 at 7:50 PM Chinmay Kolhatkar <[hidden email]>
wrote:

> Hi All,
>
> I've created following jiras encompassing the requirements (To be picked
> up in order):
> https://issues.apache.org/jira/browse/APEXMALHAR-2558
> https://issues.apache.org/jira/browse/APEXMALHAR-2559
>
> I'll start adding subtasks for subitems mentioned in the jira.
>
> I'll also create the feature branch shortly so that we can start working
> against it.
>
> -Chinmay.
>
>
>
> On Sat, Jun 16, 2018 at 7:07 PM Chinmay Kolhatkar <[hidden email]>
> wrote:
>
>> So here is how it'll be:
>> 1. ReactJS application can be compiled for production builds. The output
>> is set of html+css+compressed js files. It can also output a war file
>> directly (I think).
>> 2. The react app will have a configuration switch to allow user to set
>> the baseUrl. Default value will be "/ws/v2".
>> 3. These files can be bundled with stram so that it can be hosted by
>> stram. In this case the base url for reading data will be "/ws/v2".. i.e.
>> read from localhost.
>> 3. OR the config switch be set "http://path/to/stram/ws/v2" using which
>> one can host it seperately.
>>
>> Basically, it can be reusable with configuration changes during/after
>> build time.
>>
>> Hope that helps.
>>
>> Thanks,
>> Chinmay.
>>
>> On Thu, Jun 14, 2018 at 10:09 PM Pramod Immaneni <
>> [hidden email]> wrote:
>>
>>> I don't think they would be reusable as is without any changes externally
>>> as the application context would not be available. Also externally you
>>> would want to have a different kind of application that aggregates across
>>> multiple applications.
>>>
>>> Thanks
>>>
>>> On Thu, Jun 14, 2018 at 10:24 AM Vlad Rozov <[hidden email]> wrote:
>>>
>>> > If it is a set of static files that use existing REST API to display
>>> > information in a browser, where they will be hosted is irrelevant. They
>>> > can be hosted by stram, but it should be possible to host them
>>> > externally without making modificaitons. If this is the case, I don't
>>> > have any objections to the proposal.
>>> >
>>> > Thank you,
>>> >
>>> > Vlad
>>> >
>>> > On 6/13/18 06:09, Chinmay Kolhatkar wrote:
>>> > > Hi Team,
>>> > >
>>> > > I agree with Thomas. The main focus of this proposal was to provide
>>> a UI
>>> > > layer using existing REST APIs.
>>> > > I think we should first focus on providing a meaningful UI served by
>>> AM
>>> > to
>>> > > user using the same existing REST APIs.
>>> > >
>>> > > If in general, there is a agreement for the proposal, I can proceed
>>> with
>>> > > creating a JIRA and feature branch for the same.
>>> > >
>>> > > Thanks,
>>> > > Chinmay.
>>> > >
>>> > >
>>> > > On Wed, Jun 13, 2018 at 12:21 AM Thomas Weise <[hidden email]>
>>> wrote:
>>> > >
>>> > >> Looks like there is some disconnect in this thread, a few replies
>>> > express
>>> > >> concerns that the proposal IMO doesn't imply.
>>> > >>
>>> > >> There is already a REST API that backs all of the CLI functionality.
>>> > Modern
>>> > >> UIs run in the browser, and so what was proposed (unless I
>>> > misunderstood)
>>> > >> would be based on the same  REST API and nothing extra. The only
>>> > additional
>>> > >> thing the AM has to do is serve static files. And AFAIK it would
>>> > actually
>>> > >> simplify the UI implementation when files and REST API have the same
>>> > >> origin.
>>> > >>
>>> > >> To me where files are coming from is actually a smaller piece and
>>> > something
>>> > >> that can be augmented. But before proposing solutions outside of
>>> Apex or
>>> > >> additional deployment requirements, please consider the user
>>> > experience. It
>>> > >> should be really simple to setup and use the UI.
>>> > >>
>>> > >> I would expect that a large part of the discussion evolves around
>>> the UI
>>> > >> functionality that helps the target persona.
>>> > >>
>>> > >> Thanks,
>>> > >> Thomas
>>> > >>
>>> > >> On Tue, Jun 12, 2018, 8:05 PM Shubhrajyoti Mohapatra <
>>> > >> [hidden email]>
>>> > >> wrote:
>>> > >>
>>> > >>> Hi Community,
>>> > >>>
>>> > >>> I would like to propose a slightly different approach on having an
>>> UI
>>> > for
>>> > >>> Apex.
>>> > >>>
>>> > >>> The details are below:
>>> > >>>
>>> > >>>     1. This involves having an external web-server installed along
>>> > side of
>>> > >>>     apexCli. We will basically mimic all the functionalities that
>>> > apexCli
>>> > >>>     currently provides.
>>> > >>>     2. We will develop the necessary REST APIs for fetch/update
>>> > operations
>>> > >>>     which apexCli currently supports.
>>> > >>>     3. Because most of the information and operations needed by a
>>> > typical
>>> > >>>     user are already there in apexCli, it will of great help to end
>>> > users
>>> > >>> and
>>> > >>>     which may increase the adoption rate.
>>> > >>>
>>> > >>> About Chinmay's proposal of having the STRAM hosting the apps,
>>> please
>>> > >> find
>>> > >>> my comments below:
>>> > >>>
>>> > >>>     1. It may increase load on STRAM if we were to provide real
>>> time
>>> > >> display
>>> > >>>     of information. I am aware that apexCli fetches info from STRAM
>>> > apis
>>> > >> but
>>> > >>>     continuous polling can cause extra over-head.
>>> > >>>     2. The UI will have limited scope for the particular
>>> application.
>>> > >> Users
>>> > >>>     would definitely love to have a comprehensive UI for the
>>> > management of
>>> > >>> Apex
>>> > >>>     Apps as a whole.
>>> > >>>     3. UI might be able to affect the whole application through
>>> STRAM
>>> > >> hence
>>> > >>>     giving direct access to STRAM APIs might not be good idea.
>>> > >>>     4. Because STRAM can't have a comprehensive UI hosted inside,
>>> > anyways
>>> > >> we
>>> > >>>     will have to develop an external one.
>>> > >>>
>>> > >>> So I would like propose the plan on action like this.
>>> > >>>
>>> > >>>     1. We develop the wrapper UI for apexCli in the first phase and
>>> > give
>>> > >>>     user all the management options as REST APIs.
>>> > >>>     2. If some of information we require are not in the current
>>> apexCli
>>> > >>>     implementation, we can add those feature to apexCli and if
>>> > required to
>>> > >>>     STRAM APIs in subsequent phases.
>>> > >>>     3. In future we can even remove the REST server from STRAM if
>>> we
>>> > could
>>> > >>>     provide something like MetricsStore functionality which was
>>> there
>>> > in
>>> > >>>     DataTorrent RTS.
>>> > >>>
>>> > >>> --
>>> > >>> Thanks and Regards,
>>> > >>> Shubhrajyoti Mohapatra
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>> On Tue, Jun 12, 2018 at 2:49 AM Vlad Rozov <[hidden email]>
>>> wrote:
>>> > >>>
>>> > >>>> IMO it will be hard to provide UI that meets everyone
>>> expectation. An
>>> > >>>> external webUI can serve two goals
>>> > >>>>
>>> > >>>> 1. provide reference UI implementation
>>> > >>>> 2. ensure that REST API have all necessary entry points
>>> > >>>>
>>> > >>>> by embedding webUI directly into stram it will be tempting to
>>> shortcut
>>> > >>>> some calls to use stram classes directly.
>>> > >>>>
>>> > >>>> Thank you,
>>> > >>>>
>>> > >>>> Vlad
>>> > >>>>
>>> > >>>> On 6/11/18 12:18, Pramod Immaneni wrote:
>>> > >>>>> I too think that a comprehensive UI works better standalone but
>>> given
>>> > >>>> that
>>> > >>>>> stram already runs a webapp server, provides web services and RM
>>> > >>>> provides a
>>> > >>>>> proxy with mapped url space for it, we can extend stram to
>>> provide
>>> > >>> better
>>> > >>>>> visual output. Overall I am +1 on the idea.
>>> > >>>>>
>>> > >>>>> On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <[hidden email]>
>>> > >> wrote:
>>> > >>>>>> Clarification: I am -0 on making UI part of the stram. I am +1
>>> on
>>> > >>>>>> providing reference webUI application outside of the stram.
>>> > >>>>>>
>>> > >>>>>> Thank you,
>>> > >>>>>>
>>> > >>>>>> Vlad
>>> > >>>>>>
>>> > >>>>>> On 6/9/18 07:05, Vlad Rozov wrote:
>>> > >>>>>>> -0: stram provides REST API to query runtime information and
>>> IMO
>>> > >> this
>>> > >>>>>>> is sufficient to build an external webUI application(s).
>>> > >>>>>>>
>>> > >>>>>>> Thank you,
>>> > >>>>>>>
>>> > >>>>>>> Vlad
>>> > >>>>>>>
>>> > >>>>>>> On 6/6/18 11:17, Ambarish Pande wrote:
>>> > >>>>>>>> +1 For this feature. It would be really helpfull for users to
>>> > >>>>>>>> visualize all
>>> > >>>>>>>> this information  and the DAG.
>>> > >>>>>>>>
>>> > >>>>>>>> +1 for ReactJS + MaterialUI
>>> > >>>>>>>>     A great choice, given  its flexibility and performance.
>>> > >>>>>>>>
>>> > >>>>>>>> I would like to contribute to this effort.
>>> > >>>>>>>>
>>> > >>>>>>>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar <
>>> > >>> [hidden email]>
>>> > >>>>>>>> wrote:
>>> > >>>>>>>>
>>> > >>>>>>>>> Hi Community,
>>> > >>>>>>>>>
>>> > >>>>>>>>> I want to propose a feature in apex-core to have a UI
>>> component
>>> > >> in
>>> > >>>>>>>>> stram.
>>> > >>>>>>>>> This is influenced by how basic runtime information is shown
>>> on
>>> > >> UI
>>> > >>> by
>>> > >>>>>>>>> spark.
>>> > >>>>>>>>>
>>> > >>>>>>>>> This includes following features:
>>> > >>>>>>>>> 1. Webpage will be hosted by stram and will be available for
>>> user
>>> > >>> to
>>> > >>>>>>>>> view
>>> > >>>>>>>>> in one of the two ways (I'll prefer approach b):
>>> > >>>>>>>>>       a. Hosted on a static port in which case we'll need to
>>> add
>>> > a
>>> > >>> new
>>> > >>>>>>>>> servlet
>>> > >>>>>>>>> to stram
>>> > >>>>>>>>>       b. The webpage will be hosted on the same port as that
>>> of
>>> > >>> stram
>>> > >>>>>>>>> webservice but at a different path. Say, http://
>>> > >>>>>>>>> <stram_host>:<webservice_port>/ui
>>> > >>>>>>>>>
>>> > >>>>>>>>> 2. The webpage will be a static page and depending on the
>>> > >> framework
>>> > >>>> we
>>> > >>>>>>>>> chose, it can show the realtime metric data from stram.
>>> > >>>>>>>>>
>>> > >>>>>>>>> 3. There will be a categories of readonly information (maybe
>>> > >>>>>>>>> dynamically
>>> > >>>>>>>>> changing) that will be shown on the UI:
>>> > >>>>>>>>>       a. Application level information
>>> > >>>>>>>>>           i. Status
>>> > >>>>>>>>>           ii. Number of tuples processed/emitted
>>> > >>>>>>>>>           iii. Start time/ Running span
>>> > >>>>>>>>>           iv. Stram events
>>> > >>>>>>>>>           v. Total memory used/available
>>> > >>>>>>>>>           vi. Number of container allocated/deployed
>>> > >>>>>>>>>           v. Other information which is available in stram
>>> > >>>>>>>>>       b. Logical Operator level information
>>> > >>>>>>>>>           i. Status
>>> > >>>>>>>>>           ii. Number of tuples processed/emitted
>>> > >>>>>>>>>           iii. Important events related to logical operator
>>> > >>>>>>>>>           iv. Container list in which the operator is
>>> deployed
>>> > >>>>>>>>>           v. Any other information available in stram
>>> > >>>>>>>>>           vi. Logical DAG View
>>> > >>>>>>>>>       c. Container level information
>>> > >>>>>>>>>           i. Status
>>> > >>>>>>>>>           ii. Number of tuples processed/emitted
>>> > >>>>>>>>>           iii. Important events related to logical operator
>>> > >>>>>>>>>           iv. Any other information available in stram
>>> > >>>>>>>>>           v. Physical DAG View
>>> > >>>>>>>>>
>>> > >>>>>>>>> 4. In second phase, there will be control related operations
>>> > >>> allowed
>>> > >>>>>>>>> from
>>> > >>>>>>>>> the UI, as follows:
>>> > >>>>>>>>>       a. Stop/Kill application
>>> > >>>>>>>>>       b. Stop/Kill containers
>>> > >>>>>>>>>       c. Stack trace dump of containers
>>> > >>>>>>>>>       d. Logs related?
>>> > >>>>>>>>>       d. etc...??
>>> > >>>>>>>>>
>>> > >>>>>>>>> The above implementation can be done in phases as follows:
>>> > >>>>>>>>> 1. Have webpage display application level information only
>>> > >>>> (read-only).
>>> > >>>>>>>>> Static display first (i.e. page needs to refresh to see
>>> latest
>>> > >>>>>>>>> data), next
>>> > >>>>>>>>> dynamically changing data.
>>> > >>>>>>>>> 2. Have jenkins build tools updated as needed by UI
>>> framework.
>>> > >>>>>>>>> 3. Update the backend (stram) to serve the UI pages
>>> > >>>>>>>>> 3. Extend the webage to display logical operator information.
>>> > >>>>>>>>> 4. Extend the webage to display physical operator information
>>> > >>>>>>>>> 5. Implement control operations on UI.
>>> > >>>>>>>>>
>>> > >>>>>>>>>
>>> > >>>>>>>>> To implement above functionality, I propose ReactJS as UI
>>> > >> framework
>>> > >>>> and
>>> > >>>>>>>>> MaterialUI (based on Google's material design) for
>>> > >>> theme/controls/CSS
>>> > >>>>>>>>> support.
>>> > >>>>>>>>>
>>> > >>>>>>>>> Please let me know what you think about the same.
>>> > >>>>>>>>>
>>> > >>>>>>>>> Also, please let know if anyone is interested in
>>> contributing to
>>> > >>> this
>>> > >>>>>>>>> feature.
>>> > >>>>>>>>>
>>> > >>>>>>>>> Thanks,
>>> > >>>>>>>>> Chinmay.
>>> > >>>>>>>>>
>>> > >>>>
>>> >
>>> >
>>>
>>