Number of partitions

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Number of partitions

chiru

Hi Team,

Please help us understand, do we need to run the operator with 2 power N partitions (2,4,8,.. )or we can set to any number of partitions as per our requirement.

--
ur's
chiru
Reply | Threaded
Open this post in threaded view
|

Re: Number of partitions

Ambarish Pande-2
Hi,

There is no such binding to set the number of partitions as 2^n. It is just that in usual cases the partitions are set to be powers of 2 so that the load is distributed to each partition equally.
Here is an example.
Say the number of partitions is set to 3.
And you send data to respective partitions by looking at the last 2 bits of the tuple.

PART 1 - 00
PART 2 - 01
PART 3 - 10

Where will you send the tuple with 11 as last two bits?

So one of the partitions will have to bear tuples with 11 too, leading to load imbalance.

I hope this clears things.

Thank You


On Wed, Nov 8, 2017 at 12:45 PM, chiranjeevi vasupilli <[hidden email]> wrote:

Hi Team,

Please help us understand, do we need to run the operator with 2 power N partitions (2,4,8,.. )or we can set to any number of partitions as per our requirement.

--
ur's
chiru



--

_______________________________________________________

Ambarish Pande

Associate Software Engineer

E: [hidden email] | M: +91-9028293982

www.datatorrent.com  |  apex.apache.org


Reply | Threaded
Open this post in threaded view
|

Re: Number of partitions

vikram patil-2
In reply to this post by chiru
Hi Chiranjeevi,

No there is no such restriction on a number of partitions. But it may
create skew in tuple distribution for partitions of the operator as
few of the partitions will receive more tuples than others.

Thanks & Regards,
Vikram

On Wed, Nov 8, 2017 at 12:45 PM, chiranjeevi vasupilli
<[hidden email]> wrote:

>
> Hi Team,
>
> Please help us understand, do we need to run the operator with 2 power N
> partitions (2,4,8,.. )or we can set to any number of partitions as per our
> requirement.
>
> --
> ur's
> chiru