how to find application master URL ( non proxy )

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

how to find application master URL ( non proxy )

Sunil Parmar-2
We're using plain vanilla version of apex in our environment ( no datatorrent or dtgateway ). We're using STRAM API to access the application details. We are able to find the STRAM endpoints from YARN application API.

http://<RM>:8088/ws/v1/cluster/apps?state=RUNNING

The tracking URL property points to STRAM service but it's exposed by YARN web proxy. i.e.

http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/logicalPlan
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/physicalPlan
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/loggers

It works for monitoring but the limitation here though is that it only allows users to submit "GET" type requests on STRAM api. So it doesn't allow to run POST API  i.e. to change log level.

How to find direct application master tracking URL  ( no proxy ) ? Is there a YARN api / STRAM api to do so ? So far I have found a brute force way to scrap AM log to find it but I wanted to check with community experts if there is a better way.

Thanks,
Sunil Parmar
Reply | Threaded
Open this post in threaded view
|

Re: how to find application master URL ( non proxy )

Thomas Weise-2
Once the app is running you get the AM REST host/port from the app info, such as:

apex (application_1506456479682_0001) > get-app-info

...

  "appMasterTrackingUrl": "apex-sandbox:35463",

...

Have a look at the Apex CLI or here for details on how the REST calls are done.

Thomas


On Tue, Sep 26, 2017 at 11:53 AM, Sunil Parmar <[hidden email]> wrote:
We're using plain vanilla version of apex in our environment ( no datatorrent or dtgateway ). We're using STRAM API to access the application details. We are able to find the STRAM endpoints from YARN application API.

http://<RM>:8088/ws/v1/cluster/apps?state=RUNNING

The tracking URL property points to STRAM service but it's exposed by YARN web proxy. i.e.

http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/logicalPlan
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/physicalPlan
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/loggers

It works for monitoring but the limitation here though is that it only allows users to submit "GET" type requests on STRAM api. So it doesn't allow to run POST API  i.e. to change log level.

How to find direct application master tracking URL  ( no proxy ) ? Is there a YARN api / STRAM api to do so ? So far I have found a brute force way to scrap AM log to find it but I wanted to check with community experts if there is a better way.

Thanks,
Sunil Parmar

Reply | Threaded
Open this post in threaded view
|

Re: how to find application master URL ( non proxy )

Sunil Parmar-2
Thanks ! Is CLI the only way to get it ? Is there an API to find this ? We have a homegrown GUI built on STRAM API and setting log levels is one of the desirable feature. The limitation on prod environment is that we don't access to all the workers nodes where potentially the AppMaster could be running and GUI runs from edge node.

Sunil Parmar

On Tue, Sep 26, 2017 at 1:23 PM, Thomas Weise <[hidden email]> wrote:
Once the app is running you get the AM REST host/port from the app info, such as:

apex (application_1506456479682_0001) > get-app-info

...

  "appMasterTrackingUrl": "apex-sandbox:35463",

...

Have a look at the Apex CLI or here for details on how the REST calls are done.

Thomas


On Tue, Sep 26, 2017 at 11:53 AM, Sunil Parmar <[hidden email]> wrote:
We're using plain vanilla version of apex in our environment ( no datatorrent or dtgateway ). We're using STRAM API to access the application details. We are able to find the STRAM endpoints from YARN application API.

http://<RM>:8088/ws/v1/cluster/apps?state=RUNNING

The tracking URL property points to STRAM service but it's exposed by YARN web proxy. i.e.

http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/logicalPlan
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/physicalPlan
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/loggers

It works for monitoring but the limitation here though is that it only allows users to submit "GET" type requests on STRAM api. So it doesn't allow to run POST API  i.e. to change log level.

How to find direct application master tracking URL  ( no proxy ) ? Is there a YARN api / STRAM api to do so ? So far I have found a brute force way to scrap AM log to find it but I wanted to check with community experts if there is a better way.

Thanks,
Sunil Parmar


Reply | Threaded
Open this post in threaded view
|

Re: how to find application master URL ( non proxy )

Thomas Weise-2
The CLI is just another REST client (you see the calls it makes in the source code and repeat those same calls directly). That won't solve the problem of not being able to access the AM port directly though. Since the RM proxy does not support POST, you would need a separate proxy mechanism that does. Perhaps setting up an SSH tunnel with dynamic forwarding (SOCKS proxy) will help?

On Tue, Sep 26, 2017 at 4:18 PM, Sunil Parmar <[hidden email]> wrote:
Thanks ! Is CLI the only way to get it ? Is there an API to find this ? We have a homegrown GUI built on STRAM API and setting log levels is one of the desirable feature. The limitation on prod environment is that we don't access to all the workers nodes where potentially the AppMaster could be running and GUI runs from edge node.

Sunil Parmar

On Tue, Sep 26, 2017 at 1:23 PM, Thomas Weise <[hidden email]> wrote:
Once the app is running you get the AM REST host/port from the app info, such as:

apex (application_1506456479682_0001) > get-app-info

...

  "appMasterTrackingUrl": "apex-sandbox:35463",

...

Have a look at the Apex CLI or here for details on how the REST calls are done.

Thomas


On Tue, Sep 26, 2017 at 11:53 AM, Sunil Parmar <[hidden email]> wrote:
We're using plain vanilla version of apex in our environment ( no datatorrent or dtgateway ). We're using STRAM API to access the application details. We are able to find the STRAM endpoints from YARN application API.

http://<RM>:8088/ws/v1/cluster/apps?state=RUNNING

The tracking URL property points to STRAM service but it's exposed by YARN web proxy. i.e.

http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/logicalPlan
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/physicalPlan
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/loggers

It works for monitoring but the limitation here though is that it only allows users to submit "GET" type requests on STRAM api. So it doesn't allow to run POST API  i.e. to change log level.

How to find direct application master tracking URL  ( no proxy ) ? Is there a YARN api / STRAM api to do so ? So far I have found a brute force way to scrap AM log to find it but I wanted to check with community experts if there is a better way.

Thanks,
Sunil Parmar