3.5.0 apex core build failing with hadoop 2.7.3 dependency

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

3.5.0 apex core build failing with hadoop 2.7.3 dependency

chinmay
Administrator
Hi All,

I want to clarify whether my understanding is correct here.

1. I downloaded apache-apex-core-3.5.0-source-release.tar.gz
2. Extracted the tar
3. Ran following command to build apex core:
mvn clean package -DskipTests -Dhadoop.version=2.7.3
(NOTE: I have overridden hadoop version to 2.7.3)

I get following compilation error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project apex-engine: Compilation failure: Compilation failure:
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[159,87] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,120] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,173] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] -> [Help 1]

What should be the expected result while building apex-core when building with different version of hadoop?
Is this error expected OR something wrong in my environment?

Thanks,
Chinmay.

Reply | Threaded
Open this post in threaded view
|

Re: 3.5.0 apex core build failing with hadoop 2.7.3 dependency

Munagala Ramanath
Looks like this was a breaking change introduced in Hadoop 2.7.0:


static long DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String DELEGATION_TOKEN_MAX_LIFETIME_KEY 


static long RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY 

Ram

On Wed, Mar 22, 2017 at 11:59 PM, Chinmay Kolhatkar <[hidden email]> wrote:
Hi All,

I want to clarify whether my understanding is correct here.

1. I downloaded apache-apex-core-3.5.0-source-release.tar.gz
2. Extracted the tar
3. Ran following command to build apex core:
mvn clean package -DskipTests -Dhadoop.version=2.7.3
(NOTE: I have overridden hadoop version to 2.7.3)

I get following compilation error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project apex-engine: Compilation failure: Compilation failure:
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[159,87] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,120] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,173] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] -> [Help 1]

What should be the expected result while building apex-core when building with different version of hadoop?
Is this error expected OR something wrong in my environment?

Thanks,
Chinmay.




--

_______________________________________________________

Munagala V. Ramanath

Software Engineer

E: [hidden email] | M: (408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com  |  apex.apache.org


Reply | Threaded
Open this post in threaded view
|

Re: 3.5.0 apex core build failing with hadoop 2.7.3 dependency

Thomas Weise-2
Bummer. Thanks Ram, I was just going to look at it. The binaries work on 2.7.x though. Will the issue only show in secure mode? 

Please create a JIRA in APEXCORE.

Chinmay, you may need to apply a patch as part of your build to be able to use the 3.5.0 sources.

Thomas


On Thu, Mar 23, 2017 at 7:15 AM, Munagala Ramanath <[hidden email]> wrote:
Looks like this was a breaking change introduced in Hadoop 2.7.0:


static long DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String DELEGATION_TOKEN_MAX_LIFETIME_KEY 


static long RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY 

Ram

On Wed, Mar 22, 2017 at 11:59 PM, Chinmay Kolhatkar <[hidden email]> wrote:
Hi All,

I want to clarify whether my understanding is correct here.

1. I downloaded apache-apex-core-3.5.0-source-release.tar.gz
2. Extracted the tar
3. Ran following command to build apex core:
mvn clean package -DskipTests -Dhadoop.version=2.7.3
(NOTE: I have overridden hadoop version to 2.7.3)

I get following compilation error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project apex-engine: Compilation failure: Compilation failure:
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[159,87] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,120] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,173] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] -> [Help 1]

What should be the expected result while building apex-core when building with different version of hadoop?
Is this error expected OR something wrong in my environment?

Thanks,
Chinmay.




--

_______________________________________________________

Munagala V. Ramanath

Software Engineer

E: [hidden email] | M: <a href="tel:(408)%20331-5034" value="+14083315034" target="_blank">(408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com  |  apex.apache.org



Reply | Threaded
Open this post in threaded view
|

Re: 3.5.0 apex core build failing with hadoop 2.7.3 dependency

chinmay
Administrator
Cool.. I'll create a Jira...

On Thu, Mar 23, 2017 at 7:55 PM, Thomas Weise <[hidden email]> wrote:
Bummer. Thanks Ram, I was just going to look at it. The binaries work on 2.7.x though. Will the issue only show in secure mode? 

Please create a JIRA in APEXCORE.

Chinmay, you may need to apply a patch as part of your build to be able to use the 3.5.0 sources.

Thomas


On Thu, Mar 23, 2017 at 7:15 AM, Munagala Ramanath <[hidden email]> wrote:
Looks like this was a breaking change introduced in Hadoop 2.7.0:


static long DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String DELEGATION_TOKEN_MAX_LIFETIME_KEY 


static long RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY 

Ram

On Wed, Mar 22, 2017 at 11:59 PM, Chinmay Kolhatkar <[hidden email]> wrote:
Hi All,

I want to clarify whether my understanding is correct here.

1. I downloaded apache-apex-core-3.5.0-source-release.tar.gz
2. Extracted the tar
3. Ran following command to build apex core:
mvn clean package -DskipTests -Dhadoop.version=2.7.3
(NOTE: I have overridden hadoop version to 2.7.3)

I get following compilation error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project apex-engine: Compilation failure: Compilation failure:
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[159,87] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,120] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,173] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] -> [Help 1]

What should be the expected result while building apex-core when building with different version of hadoop?
Is this error expected OR something wrong in my environment?

Thanks,
Chinmay.




--

_______________________________________________________

Munagala V. Ramanath

Software Engineer

E: [hidden email] | M: <a href="tel:(408)%20331-5034" value="+14083315034" target="_blank">(408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com  |  apex.apache.org




Reply | Threaded
Open this post in threaded view
|

Re: 3.5.0 apex core build failing with hadoop 2.7.3 dependency

chinmay
Administrator

On Thu, Mar 23, 2017 at 7:59 PM, Chinmay Kolhatkar <[hidden email]> wrote:
Cool.. I'll create a Jira...

On Thu, Mar 23, 2017 at 7:55 PM, Thomas Weise <[hidden email]> wrote:
Bummer. Thanks Ram, I was just going to look at it. The binaries work on 2.7.x though. Will the issue only show in secure mode? 

Please create a JIRA in APEXCORE.

Chinmay, you may need to apply a patch as part of your build to be able to use the 3.5.0 sources.

Thomas


On Thu, Mar 23, 2017 at 7:15 AM, Munagala Ramanath <[hidden email]> wrote:
Looks like this was a breaking change introduced in Hadoop 2.7.0:


static long DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String DELEGATION_TOKEN_MAX_LIFETIME_KEY 


static long RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY 

Ram

On Wed, Mar 22, 2017 at 11:59 PM, Chinmay Kolhatkar <[hidden email]> wrote:
Hi All,

I want to clarify whether my understanding is correct here.

1. I downloaded apache-apex-core-3.5.0-source-release.tar.gz
2. Extracted the tar
3. Ran following command to build apex core:
mvn clean package -DskipTests -Dhadoop.version=2.7.3
(NOTE: I have overridden hadoop version to 2.7.3)

I get following compilation error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project apex-engine: Compilation failure: Compilation failure:
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[159,87] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,120] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,173] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] -> [Help 1]

What should be the expected result while building apex-core when building with different version of hadoop?
Is this error expected OR something wrong in my environment?

Thanks,
Chinmay.




--

_______________________________________________________

Munagala V. Ramanath

Software Engineer

E: [hidden email] | M: <a href="tel:(408)%20331-5034" value="+14083315034" target="_blank">(408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com  |  apex.apache.org





Reply | Threaded
Open this post in threaded view
|

Re: 3.5.0 apex core build failing with hadoop 2.7.3 dependency

Pramod Immaneni
In reply to this post by Munagala Ramanath
So much for semantic versioning..

On Thu, Mar 23, 2017 at 7:15 AM, Munagala Ramanath <[hidden email]> wrote:
Looks like this was a breaking change introduced in Hadoop 2.7.0:


static long DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String DELEGATION_TOKEN_MAX_LIFETIME_KEY 


static long RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY 

Ram

On Wed, Mar 22, 2017 at 11:59 PM, Chinmay Kolhatkar <[hidden email]> wrote:
Hi All,

I want to clarify whether my understanding is correct here.

1. I downloaded apache-apex-core-3.5.0-source-release.tar.gz
2. Extracted the tar
3. Ran following command to build apex core:
mvn clean package -DskipTests -Dhadoop.version=2.7.3
(NOTE: I have overridden hadoop version to 2.7.3)

I get following compilation error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project apex-engine: Compilation failure: Compilation failure:
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[159,87] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,120] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,173] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] -> [Help 1]

What should be the expected result while building apex-core when building with different version of hadoop?
Is this error expected OR something wrong in my environment?

Thanks,
Chinmay.




--

_______________________________________________________

Munagala V. Ramanath

Software Engineer

E: [hidden email] | M: <a href="tel:(408)%20331-5034" value="+14083315034" target="_blank">(408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com  |  apex.apache.org



Reply | Threaded
Open this post in threaded view
|

Re: 3.5.0 apex core build failing with hadoop 2.7.3 dependency

Vlad Rozov
YarnConfiguration is @Evolving :)

Moving discussion to dev@apex.

Thank you,

Vlad

On 3/23/17 08:43, Pramod Immaneni wrote:
So much for semantic versioning..

On Thu, Mar 23, 2017 at 7:15 AM, Munagala Ramanath <[hidden email]> wrote:
Looks like this was a breaking change introduced in Hadoop 2.7.0:


static long DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String DELEGATION_TOKEN_MAX_LIFETIME_KEY 


static long RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT 
static String RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY 

Ram

On Wed, Mar 22, 2017 at 11:59 PM, Chinmay Kolhatkar <[hidden email]> wrote:
Hi All,

I want to clarify whether my understanding is correct here.

1. I downloaded apache-apex-core-3.5.0-source-release.tar.gz
2. Extracted the tar
3. Ran following command to build apex core:
mvn clean package -DskipTests -Dhadoop.version=2.7.3
(NOTE: I have overridden hadoop version to 2.7.3)

I get following compilation error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project apex-engine: Compilation failure: Compilation failure:
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[159,87] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,120] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,173] cannot find symbol
[ERROR] symbol:   variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
[ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR] -> [Help 1]

What should be the expected result while building apex-core when building with different version of hadoop?
Is this error expected OR something wrong in my environment?

Thanks,
Chinmay.




--

_______________________________________________________

Munagala V. Ramanath

Software Engineer

E: [hidden email] | M: <a moz-do-not-send="true" href="tel:%28408%29%20331-5034" value="+14083315034" target="_blank">(408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com  |  apex.apache.org