Apache Apex with Apache slider

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache Apex with Apache slider

Vivek Bhide
Hi All,

Is there any resources for reference on implementing Apache Apex with Apache slider? Please let me know

Regards
Vivek
Reply | Threaded
Open this post in threaded view
|

Re: Apache Apex with Apache slider

Pramod Immaneni
Could you elaborate what your use case is as to why you would need to use slider when apex is already a native yarn application.

Thanks


On Tue, Aug 1, 2017 at 10:08 AM, Vivek Bhide <[hidden email]> wrote:
Hi All,

Is there any resources for reference on implementing Apache Apex with Apache
slider? Please let me know

Regards
Vivek



--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Apache-Apex-with-Apache-slider-tp1802.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Apache Apex with Apache slider

Vivek Bhide
Hi Pramod,

Main reason we are looking for marrying Apex with Slider is to make use of slider's ability of monitoring and restarting an application in case of failure. We have seen couple of times that Apex streaming application got killed because of some sporadic issues with cluster and there was no monitoring in place to alert about failure and restart

To avoid these oops situations and probably having some script that will monitor the job and alert/restart it we are thinking of using slider instead

Regards
Vivek
Reply | Threaded
Open this post in threaded view
|

Re: Apache Apex with Apache slider

Thomas Weise-2
Sounds like what you want is a Slider application that monitors the Apex application?

Yes, that would be possible. The Slider app/script could use the RM to locate the app and check the YARN status and then through the Apex AM REST API to poll the Apex metrics.

Thomas


On Tue, Aug 1, 2017 at 11:22 AM, Vivek Bhide <[hidden email]> wrote:
Hi Pramod,

Main reason we are looking for marrying Apex with Slider is to make use of
slider's ability of monitoring and restarting an application in case of
failure. We have seen couple of times that Apex streaming application got
killed because of some sporadic issues with cluster and there was no
monitoring in place to alert about failure and restart

To avoid these oops situations and probably having some script that will
monitor the job and alert/restart it we are thinking of using slider instead

Regards
Vivek



--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Apache-Apex-with-Apache-slider-tp1802p1804.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Apache Apex with Apache slider

Vivek Bhide
Yes Thomas thats exactly what we are looking for. But using slider is a bit trickier since slider manages the deployment of the application you give it to it. Also as far as I have read, you need to have your whole project structure as sliders expectation so that it can mange it by moving the code to containers. Since a regular deployment of Apex just consist of moving an .apa to some directory on any node in cluster (preferably edge node) and then launching the application through Apex CLI using launch, we are not sure as to what will change in terms of handling this task to slider

I am looking for some guidelines or possible sample implementation reference that can give headstart

While searching, found this below proposal about Apex which does says that 'Apache Slider is a YARN application to deploy existing distributed applications on YARN, monitor them, and make them larger or smaller as desired even when the application is running. Once Slider matures, we will take a look at close integration of Apex with Slider.'

which made me excited about using slider :)

Regards
Vivek
Reply | Threaded
Open this post in threaded view
|

Re: Apache Apex with Apache slider

Pramod Immaneni
You might be able to launch the application by running the apex cli as a separate process from your slider code and passing the apa. The apa could be in hdfs. This would however require apex cli to be present on all nodes as your slider code could be running on any node on the cluster.

Thanks

On Tue, Aug 1, 2017 at 11:49 AM, Vivek Bhide <[hidden email]> wrote:
Yes Thomas thats exactly what we are looking for. But using slider is a bit
trickier since slider manages the deployment of the application you give it
to it. Also as far as I have read, you need to have your whole project
structure as sliders expectation so that it can mange it by moving the code
to containers. Since a regular deployment of Apex just consist of moving an
.apa to some directory on any node in cluster (preferably edge node) and
then launching the application through Apex CLI using launch, we are not
sure as to what will change in terms of handling this task to slider

I am looking for some guidelines or possible sample implementation reference
that can give headstart

While searching, found this below proposal about Apex which does says that
/'Apache Slider is a YARN application to deploy existing distributed
applications on YARN, monitor them, and make them larger or smaller as
desired even when the application is running. Once Slider matures, we will
take a look at close integration of Apex with Slider.' /

which made me excited about using slider :)

Regards
Vivek



--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Apache-Apex-with-Apache-slider-tp1802p1806.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Apache Apex with Apache slider

Thomas Weise-2
Slider let's you package the binaries (along with the .apa), so that would eliminate the need for any extra install.


On Tue, Aug 1, 2017 at 11:54 AM, Pramod Immaneni <[hidden email]> wrote:
You might be able to launch the application by running the apex cli as a separate process from your slider code and passing the apa. The apa could be in hdfs. This would however require apex cli to be present on all nodes as your slider code could be running on any node on the cluster.

Thanks

On Tue, Aug 1, 2017 at 11:49 AM, Vivek Bhide <[hidden email]> wrote:
Yes Thomas thats exactly what we are looking for. But using slider is a bit
trickier since slider manages the deployment of the application you give it
to it. Also as far as I have read, you need to have your whole project
structure as sliders expectation so that it can mange it by moving the code
to containers. Since a regular deployment of Apex just consist of moving an
.apa to some directory on any node in cluster (preferably edge node) and
then launching the application through Apex CLI using launch, we are not
sure as to what will change in terms of handling this task to slider

I am looking for some guidelines or possible sample implementation reference
that can give headstart

While searching, found this below proposal about Apex which does says that
/'Apache Slider is a YARN application to deploy existing distributed
applications on YARN, monitor them, and make them larger or smaller as
desired even when the application is running. Once Slider matures, we will
take a look at close integration of Apex with Slider.' /

which made me excited about using slider :)

Regards
Vivek



--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Apache-Apex-with-Apache-slider-tp1802p1806.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.


Reply | Threaded
Open this post in threaded view
|

Re: Apache Apex with Apache slider

Vlad Rozov-2
What will be a reason to use Apex cli over Launcher API?

Thank you,

Vlad

On 8/1/17 11:59, Thomas Weise wrote:
Slider let's you package the binaries (along with the .apa), so that would eliminate the need for any extra install.


On Tue, Aug 1, 2017 at 11:54 AM, Pramod Immaneni <[hidden email]> wrote:
You might be able to launch the application by running the apex cli as a separate process from your slider code and passing the apa. The apa could be in hdfs. This would however require apex cli to be present on all nodes as your slider code could be running on any node on the cluster.

Thanks

On Tue, Aug 1, 2017 at 11:49 AM, Vivek Bhide <[hidden email]> wrote:
Yes Thomas thats exactly what we are looking for. But using slider is a bit
trickier since slider manages the deployment of the application you give it
to it. Also as far as I have read, you need to have your whole project
structure as sliders expectation so that it can mange it by moving the code
to containers. Since a regular deployment of Apex just consist of moving an
.apa to some directory on any node in cluster (preferably edge node) and
then launching the application through Apex CLI using launch, we are not
sure as to what will change in terms of handling this task to slider

I am looking for some guidelines or possible sample implementation reference
that can give headstart

While searching, found this below proposal about Apex which does says that
/'Apache Slider is a YARN application to deploy existing distributed
applications on YARN, monitor them, and make them larger or smaller as
desired even when the application is running. Once Slider matures, we will
take a look at close integration of Apex with Slider.' /

which made me excited about using slider :)

Regards
Vivek



--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Apache-Apex-with-Apache-slider-tp1802p1806.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.



Reply | Threaded
Open this post in threaded view
|

Re: Apache Apex with Apache slider

Thomas Weise-2
You can assemble it without writing code, since Slider is a script based environment.

Thomas

On Fri, Aug 4, 2017 at 4:51 AM, Vlad Rozov <[hidden email]> wrote:
What will be a reason to use Apex cli over Launcher API?

Thank you,

Vlad

On 8/1/17 11:59, Thomas Weise wrote:
Slider let's you package the binaries (along with the .apa), so that would eliminate the need for any extra install.


On Tue, Aug 1, 2017 at 11:54 AM, Pramod Immaneni <[hidden email]> wrote:
You might be able to launch the application by running the apex cli as a separate process from your slider code and passing the apa. The apa could be in hdfs. This would however require apex cli to be present on all nodes as your slider code could be running on any node on the cluster.

Thanks

On Tue, Aug 1, 2017 at 11:49 AM, Vivek Bhide <[hidden email]> wrote:
Yes Thomas thats exactly what we are looking for. But using slider is a bit
trickier since slider manages the deployment of the application you give it
to it. Also as far as I have read, you need to have your whole project
structure as sliders expectation so that it can mange it by moving the code
to containers. Since a regular deployment of Apex just consist of moving an
.apa to some directory on any node in cluster (preferably edge node) and
then launching the application through Apex CLI using launch, we are not
sure as to what will change in terms of handling this task to slider

I am looking for some guidelines or possible sample implementation reference
that can give headstart

While searching, found this below proposal about Apex which does says that
/'Apache Slider is a YARN application to deploy existing distributed
applications on YARN, monitor them, and make them larger or smaller as
desired even when the application is running. Once Slider matures, we will
take a look at close integration of Apex with Slider.' /

which made me excited about using slider :)

Regards
Vivek



--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Apache-Apex-with-Apache-slider-tp1802p1806.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.




Reply | Threaded
Open this post in threaded view
|

Re: Apache Apex with Apache slider

Vivek Bhide
Thanks you guys for you all your responses. With all your inputs and a sample implementation of slider within my organization, I was able to achieve the Apex streaming app monitoring and automatic restart through slider

Regards
Vivek