Skip to content

Livy

Livy Server

Knox provides proxied access to Livy server for submitting Spark jobs. The gateway can be used to provide authentication and encryption for clients to servers like Livy.

Gateway configuration

The Gateway can be configured for Livy by modifying the topology XML file and providing a new service XML file.

In the topology XML file, add the following with the correct hostname:

<service>
  <role>LIVYSERVER</role>
  <url>http://<livy-server>:8998</url>
</service>

Livy server will use proxyUser to run the Spark session. To avoid that a user can provide here any user (e.g. a more privileged), Knox will need to rewrite the JSON body to replace what so ever is the value of proxyUser is with the username of the authenticated user.

{  
  "driverMemory":"2G",
  "executorCores":4,
  "executorMemory":"8G",
  "proxyUser":"bernhard",
  "conf":{  
    "spark.master":"yarn-cluster",
    "spark.jars.packages":"com.databricks:spark-csv_2.10:1.5.0"
  }
}

The above is an example request body to be used to create a Spark session via Livy server and illustrates the "proxyUser" that requires rewrite.