Simple random sampling and stratified sampling in PySpark

In this article, we will discuss simple random sampling and stratified sampling in PySpark.

Simple random sampling:

In simple random sampling, every element is not obtained in a particular order. In other words, they are obtained randomly. That is why the elements are equally likely to be selected.  In simple words, random sampling is defined as the process to select a subset randomly from a large dataset. Simple random sampling in PySpark can be obtained through the sample() function. Simple sampling is of two types: replacement and without replacement. These types of random sampling are discussed below in detail, 

Method 1: Random sampling with replacement

Random sampling with replacement is a type of random sampling in which the previous randomly chosen element is returned to the population and now a random element is picked up randomly.  

Syntax:

sample(True, fraction, seed)

Here,

• fraction: It represents the fraction of rows to be generated.  It might range from 0.0 to 1.0 (inclusive)
• seed: It represents the seed required sampling (By default it is a random seed). It is used to regenerate the same random sampling.

Example:

Output:

Method 2: Random sampling without replacement

Random sampling without replacement is a type of random sampling in which each group has only one chance to be picked up in the sample.

Syntax:

sample(False, fraction, seed)

Here,

fraction: It represents the fraction of rows to be generated.  It might range from 0.0 to 1.0 (inclusive)

seed: It represents the seed required sampling (By default it is a random seed). It is used to regenerate the same random sampling.

Example:

Output:

Method 3: Stratified sampling in pyspark

In the case of Stratified sampling each of the members is grouped into the groups having the same structure (homogeneous groups) known as strata and we choose the representative of each such subgroup (called strata). Stratified sampling in pyspark can be computed using sampleBy() function. The syntax is given below,

Syntax:

sampleBy(column, fractions, seed=None)

Here, 

• column: the column that defines strata
• fractions: It represents the sampling fraction for every stratum. When the stratum is not given, we assume fraction as zero.
• seed: It represents the seed required sampling (By default it is a random seed). It is used to regenerate the same random sampling.

Example:

In this example, we have three strata, 1000000, 400000, and 2000000 and they are selected according to the fractions, 0.2, 0.4, and 0.2 respectively.  

Output: