pyspark.RDD.withResources

RDD.withResources(profile)

Specify a pyspark.resource.ResourceProfile to use when calculating this RDD. This is only supported on certain cluster managers and currently requires dynamic allocation to be enabled. It will result in new executors with the resources specified being acquired to calculate the RDD.

Added in version 3.1.0.

Parameters:

profilepyspark.resource.ResourceProfile

a resource profile

Returns:

RDD

the same RDD with user specified profile

See also

RDD.getResourceProfile()

Notes

This API is experimental