expressions
A function that get the absolute value of the numeric value.
A function that get the absolute value of the numeric value.
Returns the date that is num_months after start_date.
Returns the date that is num_months after start_date.
Used to assign a new name to a computation.
Used to assign a new name to a computation. For example the SQL expression "1 + 1 AS a" could be represented as follows: Alias(Add(Literal(1), Literal(1)), "a")()
Note that exprId and qualifiers are in a separate parameter list because we only pattern match on child and name.
The computation being performed
The name to be associated with the result of computing child.
A globally unique id used to check if an AttributeReference refers to this alias. Auto-assigned if left blank.
An optional string that can be used to referred to this attribute in a fully qualified way. Consider the examples tableName.name, subQueryAlias.name. tableName and subQueryAlias are possible qualifiers.
Explicit metadata associated with this alias that overwrites child's.
A flag to indicate if this alias is generated by Catalyst
Checks if the array (left) has the element (right)
Checks if the array (left) has the element (right)
Returns the numeric value of the first character of str.
Returns the numeric value of the first character of str.
A function throws an exception if 'condition' is not true.
A function throws an exception if 'condition' is not true.
A predicate that is evaluated to be true if there are at least n non-null and non-NaN values.
A reference to an attribute produced by another operator in the tree.
A reference to an attribute produced by another operator in the tree.
The name of this attribute, should only be used during analysis or for debugging.
The DataType of this attribute.
True if null is a valid value for this attribute.
The metadata of this attribute.
A globally unique id used to check if different AttributeReferences refer to the same attribute.
An optional string that can be used to referred to this attribute in a fully qualified way. Consider the examples tableName.name, subQueryAlias.name. tableName and subQueryAlias are possible qualifiers.
A flag to indicate if this reference is generated by Catalyst
Helper functions for working with Seq[Attribute].
A Set designed to hold AttributeReference objects, that performs equality checking using expression id instead of standard java equality.
A Set designed to hold AttributeReference objects, that performs equality checking using expression id instead of standard java equality. Using expression id means that these sets will correctly test for membership, even when the AttributeReferences in question differ cosmetically (e.g., the names have different capitalizations).
Note that we do not override equality for Attribute references as it is really weird when
AttributeReference("a"...) == AttributeReference("b", ...). This tactic leads to broken tests,
and also makes doing transformations hard (we always try keep older trees instead of new ones
when the transformation was a no-op).
Round an expression to d decimal places using HALF_EVEN rounding mode, also known as Gaussian rounding or bankers' rounding.
Round an expression to d decimal places using HALF_EVEN rounding mode, also known as Gaussian rounding or bankers' rounding. round(2.5) = 2.0, round(3.5) = 4.0.
Converts the argument from binary to a base 64 string.
Converts the argument from binary to a base 64 string.
An extended version of InternalRow that implements all special getters, toString
and equals/hashCode by genericGet.
An expression with two inputs and one output.
An expression with two inputs and one output. The output is by default evaluated to null if any input is evaluated to null.
A binary expression specifically for math functions that take two Doubles as input and returns
a Double.
A BinaryExpression that is an operator, with two properties:
A BinaryExpression that is an operator, with two properties:
1. The string representation is "x symbol y", rather than "funcName(x, y)". 2. Two inputs are expected to the be same type. If the two inputs have different types, the analyzer will find the tightest common type and do the proper type casting.
A function that calculates bitwise and(&) of two numbers.
A function that calculates bitwise and(&) of two numbers.
Code generation inherited from BinaryArithmetic.
A function that calculates bitwise not(~) of a number.
A function that calculates bitwise not(~) of a number.
A function that calculates bitwise or(|) of two numbers.
A function that calculates bitwise or(|) of two numbers.
Code generation inherited from BinaryArithmetic.
A function that calculates bitwise xor of two numbers.
A function that calculates bitwise xor of two numbers.
Code generation inherited from BinaryArithmetic.
A bound reference points to a specific slot in the input tuple, allowing the actual value to be retrieved more efficiently.
A bound reference points to a specific slot in the input tuple, allowing the actual value to be retrieved more efficiently. However, since operations like column pruning can change the layout of intermediate tuples, BindReferences should be run after all such transformations.
An expression that invokes a method on a class via reflection.
An expression that invokes a method on a class via reflection.
For now, only types defined in Reflect.typeMapping are supported (basically primitives
and string) as input types, and the output is turned automatically to a string.
Note that unlike Hive's reflect function, this expression calls only static methods (i.e. does not support calling non-static methods).
We should also look into how to consolidate this expression with org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke in the future.
the first element should be a literal string for the class name, and the second element should be a literal string for the method name, and the remaining are input arguments to the Java method.
Case statements of the form "CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END".
Case statements of the form "CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END". When a = true, returns b; when c = true, returns d; else returns e.
seq of (branch condition, branch value)
optional value for the else branch
Abstract parent class for common logic in CaseWhen and CaseWhenCodegen.
CaseWhen expression used when code generation condition is satisfied.
CaseWhen expression used when code generation condition is satisfied. OptimizeCodegen optimizer replaces CaseWhen into CaseWhenCodegen.
seq of (branch condition, branch value)
optional value for the else branch
Cast the child expression to the target data type.
Cast the child expression to the target data type.
Rounds the decimal to given scale and check whether the decimal can fit in provided precision or not, returns null if not.
An expression that is evaluated to the first non-null input.
An expression that is evaluated to the first non-null input.
coalesce(1, 2) => 1 coalesce(null, 1, 2) => 1 coalesce(null, null, 2) => 2 coalesce(null, null, null) => null
An expression that concatenates multiple input strings into a single string.
An expression that concatenates multiple input strings into a single string. If any input is null, concat returns null.
An expression that concatenates multiple input strings or array of strings into a single string, using a given separator (the first child).
An expression that concatenates multiple input strings or array of strings into a single string, using a given separator (the first child).
Returns null if the separator is null. Otherwise, concat_ws skips all null values.
A function that returns true if the string left contains the string right.
Convert a num from one base to another
Convert a num from one base to another
the number to be converted
from which base
to which base
A function that computes a cyclic redundancy check value and returns it as a bigint For input of type BinaryType
A function that computes a cyclic redundancy check value and returns it as a bigint For input of type BinaryType
Returns an Array containing the evaluation of all children expressions.
Returns an Array containing the evaluation of all children expressions.
Returns a catalyst Map containing the evaluation of all children expressions as keys and values.
Returns a catalyst Map containing the evaluation of all children expressions as keys and values. The children are a flatted sequence of kv pairs, e.g. (key1, value1, key2, value2, ...)
Creates a struct with the given field names and values
Creates a struct with the given field names and values
Seq(name1, val1, name2, val2, ...)
Common base class for both CreateNamedStruct and CreateNamedStructUnsafe.
Creates a struct with the given field names and values.
Creates a struct with the given field names and values. This is a variant that returns UnsafeRow directly. The unsafe projection operator replaces CreateStruct with this expression automatically at runtime.
Seq(name1, val1, name2, val2, ...)
The CumeDist function computes the position of a value relative to all values in the partition.
The CumeDist function computes the position of a value relative to all values in the partition. The result is the number of rows preceding or equal to the current row in the ordering of the partition divided by the total number of rows in the window partition. Any tie values in the ordering will evaluate to the same position.
This documentation has been based upon similar documentation for the Hive and Presto projects.
Expression representing the current batch time, which is used by StreamExecution to 1.
Expression representing the current batch time, which is used by StreamExecution to 1. prevent optimizer from pushing this expression below a stateful operator 2. allow IncrementalExecution to substitute this expression with a Literal(timestamp)
There is no code generation since this expression should be replaced with a literal.
Returns the current database of the SessionCatalog.
Returns the current database of the SessionCatalog.
Returns the current date at the start of query evaluation.
Returns the current date at the start of query evaluation. All calls of current_date within the same query return the same value.
There is no code generation since this expression should get constant folded by the optimizer.
Returns the current timestamp at the start of query evaluation.
Returns the current timestamp at the start of query evaluation. All calls of current_timestamp within the same query return the same value.
There is no code generation since this expression should get constant folded by the optimizer.
Adds a number of days to startdate.
Adds a number of days to startdate.
Returns the number of days from startDate to endDate.
Returns the number of days from startDate to endDate.
Subtracts a number of days to startdate.
Subtracts a number of days to startdate.
Decodes the first argument into a String using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
Decodes the first argument into a String using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.
The DenseRank function computes the rank of a value in a group of values.
The DenseRank function computes the rank of a value in a group of values. The result is one plus the previously assigned rank value. Unlike Rank, DenseRank will not produce gaps in the ranking sequence.
This documentation has been based upon similar documentation for the Hive and Presto projects.
to base the rank on; a change in the value of one the children will trigger a change in rank. This is an internal parameter and will be assigned by the Analyser.
Wrap any TokenizedLiteral expression with this so that we can invoke literal
initialization code within the .init() method of the generated class.
Wrap any TokenizedLiteral expression with this so that we can invoke literal
initialization code within the .init() method of the generated class.
This pushes itself as reference object and uses a call to eval() on itself for actual
evaluation and avoids embedding any generated code. This allows it to keep the
generated code identical regardless of the constant expression (and in addition
DynamicReplacableConstant trait casts to itself rather than actual object type).
We try to locate first foldable expression in a query tree such that all its child is foldable
but parent isn't. That way we locate the exact point where an expression is safe to evaluate
once instead of evaluating every row.
Expressions like select c from tab where
case col2 when 1 then col3 else 'y' end = 22
like queries don't convert literal evaluation into init method.
minimal expression tree that can be evaluated only once and turn into a constant.
Unlike Spark's InSet expression, this allows for TokenizedLiterals that can change dynamically in executions.
Encodes the first argument into a BINARY using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
Encodes the first argument into a BINARY using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.
A function that returns true if the string left ends with the string right.
This class is used to compute equality of (sub)expression trees.
This class is used to compute equality of (sub)expression trees. Expressions can be added to this class and they subsequently query for expression equality. Expression trees are considered equal if for the same input(s), the same result is produced.
Euler's number.
Euler's number. Note that there is no code generation because this is only evaluated by the optimizer during constant folding.
The Exists expression checks if a row exists in a subquery given some correlated condition.
The Exists expression checks if a row exists in a subquery given some correlated condition. For example (SQL):
SELECT *
FROM a
WHERE EXISTS (SELECT *
FROM b
WHERE b.id = a.id)
A trait that gets mixin to define the expected input types of an expression.
A trait that gets mixin to define the expected input types of an expression.
This trait is typically used by operator expressions (e.g. Add, Subtract) to define expected input types without any implicit casting.
Most function expressions (e.g. Substring should extends ImplicitCastInputTypes) instead.
Given an input array produces a sequence of rows for each value in the array.
Given an input array produces a sequence of rows for each value in the array.
SELECT explode(array(10,20)) -> 10 20
A base class for Explode and PosExplode
A globally unique id for a given named expression.
A globally unique id for a given named expression. Used to identify which attribute output by a relation is being referenced in a subsequent computation.
The id field is unique within a given JVM, while the uuid is used to uniquely identify JVMs.
An expression in Catalyst.
An expression in Catalyst.
If an expression wants to be exposed in the function registry (so users can call it with "name(arguments...)", the concrete implementation must be a case class whose constructor arguments are all Expressions types. See Substring for an example.
There are a few important traits:
- Nondeterministic: an expression that is not deterministic. - Unevaluable: an expression that is not supposed to be evaluated. - CodegenFallback: an expression that does not have code gen implemented and falls back to interpreted mode.
- LeafExpression: an expression that has no child. - UnaryExpression: an expression that has one child. - BinaryExpression: an expression that has two children. - TernaryExpression: an expression that has three children. - BinaryOperator: a special case of BinaryExpression that requires two children to have the same output data type.
A Set where membership is determined based on a canonical representation of an Expression (i.e.
A Set where membership is determined based on a canonical representation of an Expression (i.e. one that attempts to ignore cosmetic differences). See Canonicalize for more details.
Internally this set uses the canonical representation, but keeps also track of the original expressions to ease debugging. Since different expressions can share the same canonical representation, this means that operations that extract expressions from this set are only guaranteed to see at least one such expression. For example:
val set = AttributeSet(a + 1, 1 + a) set.iterator => Iterator(a + 1) set.contains(a + 1) => true set.contains(1 + a) => true set.contains(a + 2) => false
A function that returns the index (1-based) of the given string (left) in the comma- delimited list (right).
A function that returns the index (1-based) of the given string (left) in the comma- delimited list (right). Returns 0, if the string wasn't found or if the given string (left) contains a comma.
Formats the number X to a format like '#,###,###.##', rounded to D decimal places, and returns the result as a string.
Formats the number X to a format like '#,###,###.##', rounded to D decimal places, and returns the result as a string. If D is 0, the result has no decimal point or fractional part.
Returns the input formatted according do printf-style format strings
Returns the input formatted according do printf-style format strings
The trait used to represent the type of a Window Frame Boundary.
The trait used to represent the type of a Window Frame.
Given a timestamp, which corresponds to a certain time of day in UTC, returns another timestamp that corresponds to the same time of day in the given timezone.
Given a timestamp, which corresponds to a certain time of day in UTC, returns another timestamp that corresponds to the same time of day in the given timezone.
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format. If the format is missing, using format like "1970-01-01 00:00:00". Note that hive Language Manual says it returns 0 if fail, but in fact it returns null.
An expression that produces zero or more rows given a single input row.
An expression that produces zero or more rows given a single input row.
Generators produce multiple output rows instead of a single value like other expressions, and thus they must have a schema to associate with the rows that are output.
However, unlike row producing relational operators, which are either leaves or determine their
output schema functionally from their input, generators can contain other expressions that
might result in their modification by rules. This structure means that they might be copied
multiple times after first determining their output schema. If a new output schema is created for
each copy references up the tree might be rendered invalid. As a result generators must
instead define a function makeOutput which is called only once when the schema is first
requested. The attributes produced by this function will be automatically copied anytime rules
result in changes to the Generator or its children.
An internal row implementation that uses an array of objects as the underlying storage.
An internal row implementation that uses an array of objects as the underlying storage. Note that, while the array is not copied, and thus could technically be mutated after creation, this is not allowed.
A row implementation that uses an array of objects as the underlying storage.
A row implementation that uses an array of objects as the underlying storage. Note that, while the array is not copied, and thus could technically be mutated after creation, this is not allowed.
Returns the field at ordinal in the Array child.
Returns the field at ordinal in the Array child.
We need to do type checking here as ordinal expression maybe unresolved.
For a child whose data type is an array of structs, extracts the ordinal-th fields of all array
elements, and returns them as a new array.
For a child whose data type is an array of structs, extracts the ordinal-th fields of all array
elements, and returns them as a new array.
No need to do type checking since it is handled by ExtractValue.
Extracts json object from a json string based on json path specified, and returns json string of the extracted json object.
Extracts json object from a json string based on json path specified, and returns json string of the extracted json object. It will return null if the input json string is invalid.
Returns the value of key key in Map child.
Returns the value of key key in Map child.
We need to do type checking here as key expression maybe unresolved.
Returns the value of fields in the Struct child.
Returns the value of fields in the Struct child.
No need to do type checking since it is handled by ExtractValue.
Note that we can pass in the field name directly to keep case preserving in toString.
For example, when get field yEAr from <year: int, month: int>, we should pass in yEAr.
A function that returns the greatest value of all parameters, skipping null values.
A function that returns the greatest value of all parameters, skipping null values. It takes at least 2 parameters, and returns null iff all parameters are null.
Indicates whether a specified column expression in a GROUP BY list is aggregated or not.
Indicates whether a specified column expression in a GROUP BY list is aggregated or not. GROUPING returns 1 for aggregated or 0 for not aggregated in the result set.
GroupingID is a function that computes the level of grouping.
GroupingID is a function that computes the level of grouping.
If groupByExprs is empty, it means all grouping expressions in GroupingSets.
A placeholder expression for cube/rollup, which will be replaced by analyzer
A function that calculates hash value for a group of expressions.
A function that calculates hash value for a group of expressions. Note that the seed argument
is not exposed to users and should only be set inside spark SQL.
The hash value for an expression depends on its type and seed:
microseconds first, and use the result as seed to hash months.result starts with seed, then use result as seed, recursively
calculate hash value for each element, and assign the element hash value
to result.result starts with seed, then use result as seed, recursively
calculate hash value for each key-value, and assign the key-value hash
value to result.result starts with seed, then use result as seed, recursively
calculate hash value for each field, and assign the field hash value to
result.Finally we aggregate the hash values for each expression by the same way of struct.
If the argument is an INT or binary, hex returns the number as a STRING in hexadecimal format.
If the argument is an INT or binary, hex returns the number as a STRING in hexadecimal format. Otherwise if the number is a STRING, it converts each character into its hex representation and returns the resulting STRING. Negative numbers would be treated as two's complement.
Simulates Hive's hashing function at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils#hashcode() in Hive
Simulates Hive's hashing function at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils#hashcode() in Hive
We should use this hash function for both shuffle and bucket of Hive tables, so that we can guarantee shuffle and bucketing have same data distribution
TODO: Support Decimal and date related types
A mixin for the analyzer to perform implicit type casting using org.apache.spark.sql.catalyst.analysis.TypeCoercion.ImplicitTypeCasts.
Evaluates to true if list contains value.
Evaluates to true if list contains value.
Optimized version of In clause, when all filter values of In clause are static.
Returns string, with the first letter of each word in uppercase, all other letters in lowercase.
Returns string, with the first letter of each word in uppercase, all other letters in lowercase. Words are delimited by whitespace.
Explodes an array of structs into a table.
Explodes an array of structs into a table.
Expression that returns the name of the current file being read.
Expression that returns the name of the current file being read.
Base class for interpreted hash functions.
A MutableProjection that is calculated by calling eval on each of the specified
expressions.
A MutableProjection that is calculated by calling eval on each of the specified
expressions.
a sequence of expressions that determine the value of each column of the output row.
An interpreted row ordering comparator.
A Projection that is calculated by calling the eval of each of the specified expressions.
Evaluates to true iff it's NaN.
Evaluates to true iff it's NaN.
An expression that is evaluated to true if the input is not null.
An expression that is evaluated to true if the input is not null.
An expression that is evaluated to true if the input is null.
An expression that is evaluated to true if the input is null.
A mutable wrapper that makes two rows appear as a single concatenated row.
A mutable wrapper that makes two rows appear as a single concatenated row. Designed to be instantiated once per thread and reused.
Converts an json input string to a StructType with the specified schema.
The Lag function returns the value of input at the offsetth row before the current row in
the window.
The Lag function returns the value of input at the offsetth row before the current row in
the window. Offsets start at 0, which is the current row. The offset must be constant
integer value. The default offset is 1. When the value of input is null at the offsetth row,
null is returned. If there is no such offset row, the default expression is evaluated.
expression to evaluate offset rows before the current row.
rows to jump back in the partition.
to use when the offset row does not exist.
Returns the last day of the month which the date belongs to.
Returns the last day of the month which the date belongs to.
The Lead function returns the value of input at the offsetth row after the current row in
the window.
The Lead function returns the value of input at the offsetth row after the current row in
the window. Offsets start at 0, which is the current row. The offset must be constant
integer value. The default offset is 1. When the value of input is null at the offsetth row,
null is returned. If there is no such offset row, the default expression is evaluated.
expression to evaluate offset rows after the current row.
rows to jump ahead in the partition.
to use when the offset is larger than the window. The default value is null.
A leaf expression, i.e.
A leaf expression, i.e. one without any child expressions.
A leaf expression specifically for math constants.
A leaf expression specifically for math constants. Math constants expect no input.
There is no code generation because they should get constant folded by the optimizer.
A function that returns the least value of all parameters, skipping null values.
A function that returns the least value of all parameters, skipping null values. It takes at least 2 parameters, and returns null iff all parameters are null.
A function that return the length of the given string or binary expression.
A function that return the length of the given string or binary expression.
A function that return the Levenshtein distance between the two given strings.
A function that return the Levenshtein distance between the two given strings.
Simple RegEx pattern matching function
Simple RegEx pattern matching function
A ListQuery expression defines the query which we want to search in an IN subquery expression.
A ListQuery expression defines the query which we want to search in an IN subquery expression. It should and can only be used in conjunction with an IN expression.
For example (SQL):
SELECT *
FROM a
WHERE a.id IN (SELECT id
FROM b)
In order to do type checking, use Literal.create() instead of constructor
Computes the logarithm of a number.
Computes the logarithm of a number.
the logarithm base, default to e.
the number to compute the logarithm of.
A function that converts the characters of a string to lowercase.
A function that converts the characters of a string to lowercase.
Create a Decimal from an unscaled Long value.
Create a Decimal from an unscaled Long value. Note: this expression is internal and created only by the optimizer, we don't need to do type check for it.
Returns an unordered array containing the keys of the map.
Returns an unordered array containing the keys of the map.
Returns an unordered array containing the values of the map.
Returns an unordered array containing the values of the map.
A function that calculates an MD5 128-bit checksum and returns it as a hex string For input of type BinaryType
A function that calculates an MD5 128-bit checksum and returns it as a hex string For input of type BinaryType
Returns monotonically increasing 64-bit integers.
Returns monotonically increasing 64-bit integers.
The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the lower 33 bits represent the record number within each partition. The assumption is that the data frame has less than 1 billion partitions, and each partition has less than 8 billion records.
Since this expression is stateful, it cannot be a case object.
Returns number of months between dates date1 and date2.
Returns number of months between dates date1 and date2.
A MurMur3 Hash expression.
A MurMur3 Hash expression.
We should use this hash function for both shuffle and bucket, so that we can guarantee shuffle and bucketing have same data distribution.
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row.
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row. If the schema of the input row is specified, then the given expression will be bound to that schema.
In contrast to a normal projection, a MutableProjection reuses the same underlying row object
each time an input row is added. This significantly reduces the cost of calculating the
projection, but means that it is not safe to hold on to a reference to a InternalRow after
next() has been called on the Iterator that produced it. Instead, the user must call
InternalRow.copy() and hold on to the returned InternalRow before calling next().
A parent class for mutable container objects that are reused when the values are changed, resulting in less garbage.
A parent class for mutable container objects that are reused when the values are changed, resulting in less garbage. These values are held by a SpecificInternalRow.
The following code was roughly used to generate these objects:
val types = "Int,Float,Boolean,Double,Short,Long,Byte,Any".split(",") types.map {tpe => s""" final class Mutable$tpe extends MutableValue { var value: $tpe = 0 def boxed = if (isNull) null else value def update(v: Any) = value = { isNull = false v.asInstanceOf[$tpe] } def copy() = { val newCopy = new Mutable$tpe newCopy.isNull = isNull newCopy.value = value newCopy } }""" }.foreach(println) types.map { tpe => s""" override def set$tpe(ordinal: Int, value: $tpe): Unit = { val currentValue = values(ordinal).asInstanceOf[Mutable$tpe] currentValue.isNull = false currentValue.value = value } override def get$tpe(i: Int): $tpe = { values(i).asInstanceOf[Mutable$tpe].value }""" }.foreach(println)
The NTile function divides the rows for each window partition into n buckets ranging from 1 to
at most n.
The NTile function divides the rows for each window partition into n buckets ranging from 1 to
at most n. Bucket values will differ by at most 1. If the number of rows in the partition does
not divide evenly into the number of buckets, then the remainder values are distributed one per
bucket, starting with the first bucket.
The NTile function is particularly useful for the calculation of tertiles, quartiles, deciles and other common summary statistics
The function calculates two variables during initialization: The size of a regular bucket, and the number of buckets that will have one extra row added to it (when the rows do not evenly fit into the number of buckets); both variables are based on the size of the current partition. During the calculation process the function keeps track of the current row number, the current bucket number, and the row number at which the bucket will change (bucketThreshold). When the current row number reaches bucket threshold, the bucket value is increased by one and the the threshold is increased by the bucket size (plus one extra if the current bucket is padded).
This documentation has been based upon similar documentation for the Hive and Presto projects.
number of buckets to divide the rows in. Default value is 1.
An Expression evaluates to left iff it's not NaN, or evaluates to right otherwise.
An Expression evaluates to left iff it's not NaN, or evaluates to right otherwise.
This Expression is useful for mapping NaN values to null.
An Expression that is named.
Returns the first date which is later than startDate and named as dayOfWeek.
Returns the first date which is later than startDate and named as dayOfWeek. For example, NextDay(2015-07-27, Sunday) would return 2015-08-02, which is the first Sunday later than 2015-07-27.
Allowed "dayOfWeek" is defined in DateTimeUtils.getDayOfWeekFromString.
Expressions that don't have SQL representation should extend this trait.
Expressions that don't have SQL representation should extend this trait. Examples are
ScalaUDF, ScalaUDAF, and object expressions like MapObjects and Invoke.
An expression that is nondeterministic.
When an expression inherits this, meaning the expression is null intolerant (i.e.
When an expression inherits this, meaning the expression is null intolerant (i.e. any null input will result in null output). We will use this information during constructing IsNotNull constraints.
An offset window function is a window function that returns the value of the input column offset by a number of rows within the partition.
An offset window function is a window function that returns the value of the input column offset by a number of rows within the partition. For instance: an OffsetWindowfunction for value x with offset -2, will get the value of x 2 rows back in the partition.
A place holder used to hold a reference that has been resolved to a field outside of the current plan.
A place holder used to hold a reference that has been resolved to a field outside of the current plan. This is used for correlated subqueries.
In addition to TokenLiteral, this class can also be used in plan caching so allows for internal value to be updated in subsequent runs when the plan is re-used with different constants.
In addition to TokenLiteral, this class can also be used in plan caching so allows for internal value to be updated in subsequent runs when the plan is re-used with different constants. For that reason this does not extend Literal (to avoid Analyzer/Optimizer etc doing constant propagation for example) and its hash/equals ignores the value matching and only the position of the literal in the plan is used with the data type.
Where ever ParamLiteral case matching is required, it must match for DynamicReplacableConstant and use .eval(..) for code generation. see SNAP-1597 for more details. For cases of common-subexpression elimination that depend on constant values being equal in different parts of the tree, a new RefParamLiteral has been added that points to a ParamLiteral and is always equal to it, see SNAP-2462 for more details.
Extracts a part from a URL
Extracts a part from a URL
The PercentRank function computes the percentage ranking of a value in a group of values.
The PercentRank function computes the percentage ranking of a value in a group of values. The result the rank of the minus one divided by the total number of rows in the partition minus one: (r - 1) / (n - 1). If a partition only contains one row, the function will return 0.
The PercentRank function is similar to the CumeDist function, but it uses rank values instead of row counts in the its numerator.
This documentation has been based upon similar documentation for the Hive and Presto projects.
to base the rank on; a change in the value of one the children will trigger a change in rank. This is an internal parameter and will be assigned by the Analyser.
Pi.
Pi. Note that there is no code generation because this is only evaluated by the optimizer during constant folding.
An interface for expressions that contain a QueryPlan.
Given an input array produces a sequence of rows for each position and value in the array.
Given an input array produces a sequence of rows for each position and value in the array.
SELECT posexplode(array(10,20)) -> 0 10 1 20
Expression used internally to convert the TimestampType to Long without losing precision, i.e.
Expression used internally to convert the TimestampType to Long without losing precision, i.e. in microseconds. Used in time windowing.
An Expression that returns a boolean value.
A predicate subquery checks the existence of a value in a sub-query.
A predicate subquery checks the existence of a value in a sub-query. We currently only allow PredicateSubquery expressions within a Filter plan (i.e. WHERE or a HAVING clause). This will be rewritten into a left semi/anti join during analysis.
A place holder used when printing expressions without debugging information such as the expression id or the unresolved indicator.
Print the result of an expression to stderr (used for debugging codegen).
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row.
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row. If the schema of the input row is specified, then the given expression will be bound to that schema.
An expression used to wrap the children when promote the precision of DecimalType to avoid promote multiple times.
A Random distribution generating expression.
A Random distribution generating expression. TODO: This can be made generic to generate any type of random distribution, or any type of StructType.
Since this expression is stateful, it cannot be a case object.
Generate a random column with i.i.d.
Generate a random column with i.i.d. uniformly distributed values in [0, 1).
Generate a random column with i.i.d.
Generate a random column with i.i.d. values drawn from the standard normal distribution.
The Rank function computes the rank of a value in a group of values.
The Rank function computes the rank of a value in a group of values. The result is one plus the number of rows preceding or equal to the current row in the ordering of the partition. The values will produce gaps in the sequence.
This documentation has been based upon similar documentation for the Hive and Presto projects.
to base the rank on; a change in the value of one the children will trigger a change in rank. This is an internal parameter and will be assigned by the Analyser.
A RankLike function is a WindowFunction that changes its value based on a change in the value of the order of the window in which is processed.
A RankLike function is a WindowFunction that changes its value based on a change in the value of
the order of the window in which is processed. For instance, when the value of input changes
in a window ordered by input the rank function also changes. The size of the change of the
rank function is (typically) not dependent on the size of the change in input.
This documentation has been based upon similar documentation for the Hive and Presto projects.
This class is used as a substitution for ParamLiteral when two ParamLiterals have same constant values during parsing.
This class is used as a substitution for ParamLiteral when two ParamLiterals have same constant values during parsing. This behaves like being equal to the ParamLiteral it points to in all respects but will be different from other ParamLiterals. Two RefParamLiterals will be equal iff their respective ParamLiterals are.
The above policy allows an expression like "a = 4 and b = 4" to be equal to "a = 5 and b = 5" after tokenization but will be different from "a = 5 and b = 6". This distinction is required because former can lead to a different execution plan after common-subexpression processing etc that can apply on because the actual values for the two tokenized values are equal in this instance. Hence it can lead to a different plan in case where actual constants are different, so after tokenization they should act as different expressions. See TPCH Q19 for an example where equal values in two different positions lead to an optimized plan due to common-subexpression being pulled out of OR conditions as a separate AND condition which leads to further filter push down which is not possible if the actual values are different.
Note: This class maintains its own copy of value since it can change in execution (e.g. ROUND can change precision of underlying Decimal value) which should not lead to a change of value of referenced ParamLiteral or vice-versa. However, during planning, code generation and other phases before runJob, the value and dataType should match exactly which is checked by referenceEquals. After deserialization on remote executor, the class no longer maintains a reference and falls back to behaving like a regular ParamLiteral since the required analysis and other phases are already done, and final code generation requires a copy of the values.
A special expression that evaluates BoundReferences by given expressions instead of the input row.
A special expression that evaluates BoundReferences by given expressions instead of the input row.
The expression that contains BoundReference and produces the final output.
The expressions that used as input values for BoundReference.
Extract a specific(idx) group identified by a Java regex.
Extract a specific(idx) group identified by a Java regex.
NOTE: this expression is not THREAD-SAFE, as it has some internal mutable status.
Replace all substrings of str that match regexp with rep.
Replace all substrings of str that match regexp with rep.
NOTE: this expression is not THREAD-SAFE, as it has some internal mutable status.
Round an expression to d decimal places using HALF_UP rounding mode.
Round an expression to d decimal places using HALF_UP rounding mode. round(2.5) == 3.0, round(3.5) == 4.0.
Round the child's result to scale decimal place when scale >= 0
or round at integral part when scale < 0.
Round the child's result to scale decimal place when scale >= 0
or round at integral part when scale < 0.
Child of IntegralType would round to itself when scale >= 0.
Child of FractionalType whose value is NaN or Infinite would always round to itself.
Round's dataType would always equal to child's dataType except for DecimalType,
which would lead scale decrease from the origin DecimalType.
The RowNumber function computes a unique, sequential number to each row, starting with one, according to the ordering of rows within the window partition.
The RowNumber function computes a unique, sequential number to each row, starting with one, according to the ordering of rows within the window partition.
This documentation has been based upon similar documentation for the Hive and Presto projects.
An expression that gets replaced at runtime (currently by the optimizer) into a different expression for evaluation.
An expression that gets replaced at runtime (currently by the optimizer) into a different expression for evaluation. This is mainly used to provide compatibility with other databases. For example, we use this to support "nvl" by replacing it with "coalesce".
A RuntimeReplaceable should have the original parameters along with a "child" expression in the case class constructor, and define a normal constructor that accepts only the original parameters. For an example, see Nvl. To make sure the explain plan and expression SQL works correctly, the implementation should also override flatArguments method and sql method.
User-defined function.
User-defined function. Note that the user-defined functions must be deterministic.
The user defined scala function to run. Note that if you use primitive parameters, you are not able to check if it is null or not, and the UDF will return null for you if the primitive input is null. Use boxed type or Option if you wanna do the null-handling yourself.
Return type of function.
The input expressions of this UDF.
The expected input types of this UDF, used to perform type coercion. If we do not want to perform coercion, simply use "Nil". Note that it would've been better to use Option of Seq[DataType] so we can use "None" as the case for no type coercion. However, that would require more refactoring of the codebase.
The user-specified name of this UDF.
A subquery that will return only one row and one column.
A subquery that will return only one row and one column. This will be converted into a physical scalar subquery during planning.
Note: exprId is used to have a unique name in explain string output.
Splits a string into arrays of sentences, where each sentence is an array of words.
Splits a string into arrays of sentences, where each sentence is an array of words. The 'lang' and 'country' arguments are optional, and if omitted, the default locale is used.
A function that calculates a sha1 hash value and returns it as a hex string For input of type BinaryType or StringType
A function that calculates a sha1 hash value and returns it as a hex string For input of type BinaryType or StringType
A function that calculates the SHA-2 family of functions (SHA-224, SHA-256, SHA-384, and SHA-512) and returns it as a hex string.
A function that calculates the SHA-2 family of functions (SHA-224, SHA-256, SHA-384, and SHA-512) and returns it as a hex string. The first argument is the string or binary to be hashed. The second argument indicates the desired bit length of the result, which must have a value of 224, 256, 384, 512, or 0 (which is equivalent to 256). SHA-224 is supported starting from Java 8. If asking for an unsupported SHA function, the return value is NULL. If either argument is NULL or the hash length is not one of the permitted values, the return value is NULL.
Bitwise left shift.
Bitwise left shift.
the base number to shift.
number of bits to left shift.
Bitwise (signed) right shift.
Bitwise (signed) right shift.
the base number to shift.
number of bits to right shift.
Bitwise unsigned right shift, for integer and long data type.
Bitwise unsigned right shift, for integer and long data type.
the base number.
the number of bits to right shift.
Given an array or map, returns its size.
Given an array or map, returns its size. Returns -1 if null.
A SizeBasedWindowFunction needs the size of the current window for its calculation.
Sorts the input array in ascending / descending order according to the natural ordering of the array elements and returns it.
Sorts the input array in ascending / descending order according to the natural ordering of the array elements and returns it.
An expression that can be used to sort a tuple.
An expression that can be used to sort a tuple. This class extends expression primarily so that transformations over expression will descend into its child.
An expression to generate a 64-bit long prefix used in sorting.
An expression to generate a 64-bit long prefix used in sorting. If the sort must operate over null keys as well, this.nullValue can be used in place of emitted null prefixes in the sort.
A function that return Soundex code of the given string expression.
A function that return Soundex code of the given string expression.
Expression that returns the current partition id.
Expression that returns the current partition id.
A row type that holds an array specialized container objects, of type MutableValue, chosen based on the dataTypes of each column.
A row type that holds an array specialized container objects, of type MutableValue, chosen based on the dataTypes of each column. The intent is to decrease garbage when modifying the values of primitive columns.
A specified Window Frame.
Separate v1, ..., vk into n rows.
Separate v1, ..., vk into n rows. Each row will have k/n columns. n must be constant.
SELECT stack(2, 1, 2, 3) -> 1 2 3 NULL
A function that returns true if the string left starts with the string right.
A function that returns the position of the first occurrence of substr in the given string.
A function that returns the position of the first occurrence of substr in the given string. Returns null if either of the arguments are null and returns 0 if substr could not be found in str.
NOTE: that this is not zero based, but 1-based index. The first character in str has index 1.
Returns str, left-padded with pad to a length of len.
Returns str, left-padded with pad to a length of len.
A function that returns the position of the first occurrence of substr in given string after position pos.
A function that returns the position of the first occurrence of substr in given string after position pos.
A base trait for functions that compare two strings, returning a boolean.
Returns str, right-padded with pad to a length of len.
Returns str, right-padded with pad to a length of len.
Returns the string which repeat the given string value n times.
Returns the string which repeat the given string value n times.
Returns the reversed given string.
Returns the reversed given string.
Returns a string consisting of n spaces.
Returns a string consisting of n spaces.
Splits str around pat (pattern is a regular expression).
Splits str around pat (pattern is a regular expression).
Creates a map after splitting the input text into key/value pairs using delimiters
Creates a map after splitting the input text into key/value pairs using delimiters
A function translate any character in the srcExpr by a character in replaceExpr.
A function translate any character in the srcExpr by a character in replaceExpr.
The characters in replaceExpr is corresponding to the characters in matchingExpr.
The translate will happen when any character in the string matching with the character
in the matchingExpr.
A function that trim the spaces from both ends for the specified string.
A function that trim the spaces from both ends for the specified string.
A function that trim the spaces from left end for given string.
A function that trim the spaces from left end for given string.
A function that trim the spaces from right end for given string.
A function that trim the spaces from right end for given string.
Converts a StructType to a json output string.
A base interface for expressions that contain a LogicalPlan.
A function that takes a substring of its first argument starting at a given position.
A function that takes a substring of its first argument starting at a given position. Defined for String and Binary types.
NOTE: that this is not zero based, but 1-based index. The first character in str has index 1.
Returns the substring from string str before count occurrences of the delimiter delim.
Returns the substring from string str before count occurrences of the delimiter delim. If count is positive, everything the left of the final delimiter (counting from left) is returned. If count is negative, every to the right of the final delimiter (counting from the right) is returned. substring_index performs a case-sensitive match when searching for delim.
An expression with three inputs and one output.
An expression with three inputs and one output. The output is by default evaluated to null if any input is evaluated to null.
Adds an interval to timestamp.
Subtracts an interval from timestamp.
Returns the date part of a timestamp or string.
Returns the date part of a timestamp or string.
Given a timestamp, which corresponds to a certain time of day in the given timezone, returns another timestamp that corresponds to the same time of day in UTC.
Given a timestamp, which corresponds to a certain time of day in the given timezone, returns another timestamp that corresponds to the same time of day in UTC.
Converts time string with given pattern.
Converts time string with given pattern. Deterministic version of UnixTimestamp, must have at least one parameter.
A Literal that passes its value as a reference object in generated code instead of embedding as a constant to allow generated code reuse.
Returns date truncated to the unit specified by the format.
Returns date truncated to the unit specified by the format.
Converts the argument from a base 64 string to BINARY.
Converts the argument from a base 64 string to BINARY.
An expression with one input and one output.
An expression with one input and one output. The output is by default evaluated to null if the input is evaluated to null.
A unary expression specifically for math functions.
A unary expression specifically for math functions. Math Functions expect a specific type of
input format, therefore these functions extend ExpectsInputTypes.
An expression that cannot be evaluated.
An expression that cannot be evaluated. Some expressions don't live past analysis or optimization time (e.g. Star). This trait is used by those expressions.
Performs the inverse operation of HEX.
Performs the inverse operation of HEX. Resulting characters are returned as a byte array.
Converts time string with given pattern.
Converts time string with given pattern. (see [http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html]) to Unix time stamp (in seconds), returns null if fail. Note that hive Language Manual says it returns 0 if fail, but in fact it returns null. If the second parameter is missing, use "yyyy-MM-dd HH:mm:ss". If no parameters provided, the first parameter will be current_timestamp. If the first parameter is a Date or Timestamp instead of String, we will ignore the second parameter.
A projection that returns UnsafeRow.
Return the unscaled Long value of a Decimal, assuming it fits in a Long.
Return the unscaled Long value of a Decimal, assuming it fits in a Long. Note: this expression is internal and created only by the optimizer, we don't need to do type check for it.
Cast the child expression to the target data type, but will throw error if the cast might truncate, e.g.
Cast the child expression to the target data type, but will throw error if the cast might truncate, e.g. long -> int, timestamp -> data.
A function that converts the characters of a string to uppercase.
A function that converts the characters of a string to uppercase.
A generator that produces its output using the provided lambda function.
<value> FOLLOWING boundary.
<value> PRECEDING boundary.
The trait used to represent the a Window Frame.
A window function is a function that can only be evaluated in the context of a window operator.
The trait of the Window Specification (specified in the OVER clause or WINDOW clause) for Window Functions.
The specification for a window function.
The specification for a window function.
It defines the way that input rows are partitioned.
It defines the ordering of rows in a partition.
It defines the window frame in a partition.
A Window specification reference that refers to the WindowSpecDefinition defined
under the name name.
A xxHash64 64-bit hash expression.
Builds a map that is keyed by an Attribute's expression id.
Builds a map that is keyed by an Attribute's expression id. Using the expression id allows values to be looked up even when the attributes used differ cosmetically (i.e., the capitalization of the name, or the expected nullability).
Rewrites an expression using rules that are guaranteed preserve the result while attempting to remove cosmetic variations.
Rewrites an expression using rules that are guaranteed preserve the result while attempting
to remove cosmetic variations. Deterministic expressions that are equal after canonicalization
will always return the same answer given the same input (i.e. false positives should not be
possible). However, it is possible that two canonical expressions that are not equal will in fact
return the same answer given any input (i.e. false negatives are possible).
The following rules are applied:
hashCode.hashCode.hashCode.
Case statements of the form "CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END".
Case statements of the form "CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END". When a = b, returns c; when a = d, returns e; else returns f.
Factory methods for CaseWhen.
Returns a Row containing the evaluation of all children expressions.
CURRENT ROW boundary.
Extractor for and other utility methods for decimal literals.
Used as input into expressions whose output does not depend on any input value.
An extractor that matches both standard 3VL equality and null-safe equality.
Extractor for making working with frame boundaries easier.
A projection that could turn UnsafeRow into GenericInternalRow
Extractor for retrieving Int literals.
An expression representing a not yet available attribute name.
An expression representing a not yet available attribute name. This expression is unevaluable and as its name suggests it is a temporary place holder until we're able to determine the actual attribute name.
An extractor that matches non-null literal values
RangeFrame treats rows in a partition as groups of peers.
RangeFrame treats rows in a partition as groups of peers.
All rows having the same ORDER BY ordering are considered as peers.
When a ValuePreceding or a ValueFollowing is used as its FrameBoundary,
the value is considered as a logical offset.
For example, assuming the value of the current row's ORDER BY expression expr is v,
RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING represents a frame containing rows whose values
expr are in the range of [v-1, v+1].
If ORDER BY clause is not defined, all rows in the partition is considered as peers
of the current row.
RowFrame treats rows in a partition individually.
RowFrame treats rows in a partition individually. When a ValuePreceding
or a ValueFollowing is used as its FrameBoundary, the value is considered
as a physical offset.
For example, ROW BETWEEN 1 PRECEDING AND 1 FOLLOWING represents a 3-row frame,
from the row precedes the current row to the row follows the current row.
UNBOUNDED FOLLOWING boundary.
UNBOUNDED PRECEDING boundary.
Used as a place holder when a frame specification is not defined.
A collection of generators that build custom bytecode at runtime for performing the evaluation of catalyst expression.
A set of classes that can be used to represent trees of relational expressions. A key goal of the expression library is to hide the details of naming and scoping from developers who want to manipulate trees of relational operators. As such, the library defines a special type of expression, a NamedExpression in addition to the standard collection of expressions.
Standard Expressions
A library of standard expressions (e.g., Add, EqualTo), aggregates (e.g., SUM, COUNT), and other computations (e.g. UDFs). Each expression type is capable of determining its output schema as a function of its children's output schema.
Named Expressions
Some expression are named and thus can be referenced by later operators in the dataflow graph. The two types of named expressions are AttributeReferences and Aliases. AttributeReferences refer to attributes of the input tuple for a given operator and form the leaves of some expression trees. Aliases assign a name to intermediate computations. For example, in the SQL statement
SELECT a+b AS c FROM ..., the expressionsaandbwould be represented byAttributeReferencesandcwould be represented by anAlias.During analysis, all named expressions are assigned a globally unique expression id, which can be used for equality comparisons. While the original names are kept around for debugging purposes, they should never be used to check if two attributes refer to the same value, as plan transformations can result in the introduction of naming ambiguity. For example, consider a plan that contains subqueries, both of which are reading from the same table. If an optimization removes the subqueries, scoping information would be destroyed, eliminating the ability to reason about which subquery produced a given attribute.
Evaluation
The result of expressions can be evaluated using the
Expression.apply(Row)method.