MesosSchedulerUtils

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def checkPorts(conf: SparkConf, ports: List[(Long, Long)]): Boolean

   

   Checks executor ports if they are within some range of the offered list of ports ranges,

   Checks executor ports if they are within some range of the offered list of ports ranges,

   conf

   the Spark Config

   ports

   the list of ports to check

   returns

   true if ports are within range false otherwise

   Attributes

   protected

6. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

7. def createResource(resourceName: String, quantity: Double): Resource

   

   Build a Mesos resource protobuf object

   Build a Mesos resource protobuf object

   Attributes

   protected

8. def createResource(name: String, amount: Double, role: Option[String] = None): Resource

   

9. def createSchedulerDriver(masterUrl: String, scheduler: Scheduler, sparkUser: String, appName: String, conf: SparkConf, webuiUrl: Option[String] = None, checkpoint: Option[Boolean] = None, failoverTimeout: Option[Double] = None, frameworkId: Option[String] = None): SchedulerDriver

   

   Creates a new MesosSchedulerDriver that communicates to the Mesos master.

   Creates a new MesosSchedulerDriver that communicates to the Mesos master.

   masterUrl

   The url to connect to Mesos master

   scheduler

   the scheduler class to receive scheduler callbacks

   sparkUser

   User to impersonate with when running tasks

   appName

   The framework name to display on the Mesos UI

   conf

   Spark configuration

   webuiUrl

   The WebUI url to link from Mesos UI

   checkpoint

   Option to checkpoint tasks for failover

   failoverTimeout

   Duration Mesos master expect scheduler to reconnect on disconnect

   frameworkId

   The id of the new framework

   Attributes

   protected

10. final def eq(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

11. def equals(arg0: Any): Boolean

    

    Definition Classes

    AnyRef → Any

12. def executorMemory(sc: SparkContext): Int

    

    Return the amount of memory to allocate to each executor, taking into account container overheads.

    Return the amount of memory to allocate to each executor, taking into account container overheads.

    sc

    SparkContext to use to get spark.mesos.executor.memoryOverhead value

    returns

    memory requirement as (0.1 * <memoryOverhead>) or MEMORY_OVERHEAD_MINIMUM (whichever is larger)

13. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

14. def getAttribute(attr: Attribute): (String, Set[String])

    

    Helper method to get the key,value-set pair for a Mesos Attribute protobuf

    Helper method to get the key,value-set pair for a Mesos Attribute protobuf

    Attributes

    protected

15. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

16. def getRangeResource(res: List[Resource], name: String): List[(Long, Long)]

    

    Transforms a range resource to a list of ranges

    Transforms a range resource to a list of ranges

    res

    the mesos resource list

    name

    the name of the resource

    returns

    the list of ranges returned

    Attributes

    protected

17. def getRejectOfferDurationForReachedMaxCores(sc: SparkContext): Long

    

    Attributes

    protected

18. def getRejectOfferDurationForUnmetConstraints(sc: SparkContext): Long

    

    Attributes

    protected

19. def getResource(res: List[Resource], name: String): Double

    

20. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

21. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean = false): Boolean

    

    Attributes

    protected

    Definition Classes

    Logging

22. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

23. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

24. def isTraceEnabled(): Boolean

    

    Attributes

    protected

    Definition Classes

    Logging

25. def log: Logger

    

    Attributes

    protected

    Definition Classes

    Logging

26. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

27. def logDebug(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

28. def logError(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

29. def logError(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

30. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

31. def logInfo(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

32. def logName: String

    

    Attributes

    protected

    Definition Classes

    Logging

33. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

34. def logTrace(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

35. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

36. def logWarning(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

37. val managedPortNames: List[String]

    

38. def markErr(): Unit

    

    Attributes

    protected

39. def markRegistered(): Unit

    

    Signal that the scheduler has registered with Mesos.

    Signal that the scheduler has registered with Mesos.

    Attributes

    protected

40. def matchesAttributeRequirements(slaveOfferConstraints: Map[String, Set[String]], offerAttributes: Map[String, GeneratedMessage]): Boolean

    

    Match the requirements (if any) to the offer attributes.

    Match the requirements (if any) to the offer attributes. if attribute requirements are not specified - return true else if attribute is defined and no values are given, simple attribute presence is performed else if attribute name and value is specified, subset match is performed on slave attributes

41. var mesosDriver: SchedulerDriver

    

    Attributes

    protected

42. def mesosToTaskState(state: mesos.Protos.TaskState): TaskState.TaskState

    

43. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

44. def nonZeroPortValuesFromConfig(conf: SparkConf): List[Long]

    

    The values of the non-zero ports to be used by the executor process.

    The values of the non-zero ports to be used by the executor process.

    conf

    the spark config to use

    returns

    the ono-zero values of the ports

45. final def notify(): Unit

    

    Definition Classes

    AnyRef

46. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

47. def parseConstraintString(constraintsVal: String): Map[String, Set[String]]

    

    Parses the attributes constraints provided to spark and build a matching data struct: Map[<attribute-name>, Set[values-to-match]] The constraints are specified as ';' separated key-value pairs where keys and values are separated by ':'.

    Parses the attributes constraints provided to spark and build a matching data struct: Map[<attribute-name>, Set[values-to-match]] The constraints are specified as ';' separated key-value pairs where keys and values are separated by ':'. The ':' implies equality (for singular values) and "is one of" for multiple values (comma separated). For example:

    parseConstraintString("os:centos7;zone:us-east-1a,us-east-1b")
    // would result in
    <code>
    Map(
      "os" -> Set("centos7"),
      "zone":   -> Set("us-east-1a", "us-east-1b")
    )

    Mesos documentation: http://mesos.apache.org/documentation/attributes-resources/ https://github.com/apache/mesos/blob/master/src/common/values.cpp https://github.com/apache/mesos/blob/master/src/common/attributes.cpp

    constraintsVal

    constaints string consisting of ';' separated key-value pairs (separated by ':')

    returns

    Map of constraints to match resources offers.

48. def partitionPortResources(requestedPorts: List[Long], offeredResources: List[Resource]): (List[Resource], List[Resource])

    

    Partitions port resources.

    Partitions port resources.

    requestedPorts

    non-zero ports to assign

    offeredResources

    the resources offered

    returns

    resources left, port resources to be used.

49. def partitionResources(resources: List[Resource], resourceName: String, amountToUse: Double): (List[Resource], List[Resource])

    

    Partition the existing set of resources into two groups, those remaining to be scheduled and those requested to be used for a new task.

    Partition the existing set of resources into two groups, those remaining to be scheduled and those requested to be used for a new task.

    resources

    The full list of available resources

    resourceName

    The name of the resource to take from the available resources

    amountToUse

    The amount of resources to take from the available resources

    returns

    The remaining resources list and the used resources list.

50. def setupUris(uris: String, builder: Builder, useFetcherCache: Boolean = false): Unit

    

51. def startScheduler(newDriver: SchedulerDriver): Unit

    

    Starts the MesosSchedulerDriver and stores the current running driver to this new instance.

    Starts the MesosSchedulerDriver and stores the current running driver to this new instance. This driver is expected to not be running. This method returns only after the scheduler has registered with Mesos.

52. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

53. def taskStateToMesos(state: TaskState.TaskState): mesos.Protos.TaskState

    

54. def toAttributeMap(offerAttributes: List[Attribute]): Map[String, GeneratedMessage]

    

    Converts the attributes from the resource offer into a Map of name -> Attribute Value The attribute values are the mesos attribute types and they are

    Converts the attributes from the resource offer into a Map of name -> Attribute Value The attribute values are the mesos attribute types and they are

    offerAttributes

    the attributes offered

    Attributes

    protected

55. def toString(): String

    

    Definition Classes

    AnyRef → Any

56. def unsetFrameworkID(sc: SparkContext): Unit

    

    spark.mesos.driver.frameworkId is set by the cluster dispatcher to correlate driver submissions with frameworkIDs.

    spark.mesos.driver.frameworkId is set by the cluster dispatcher to correlate driver submissions with frameworkIDs. However, this causes issues when a driver process launches more than one framework (more than one SparkContext(, because they all try to register with the same frameworkID. To enforce that only the first driver registers with the configured framework ID, the driver calls this method after the first registration.

57. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

58. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

59. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )