Intro­duc­tion to frayed tensors

Share post:


We discuss the concept of Tensor­flow

ragged tensors

which was intro­duced at the end of 2018. We explain their basic struc­ture and why they are useful.

Problem defini­tion

Due to the standar­diza­tion of tradi­tional machine learning problems, many models are very easy to imple­ment: Read a table of features from a database, use pandas and numpy for prepro­ces­sing, create a model using one of the well-known libra­ries, and type In many cases – that’s it! Done!

But wait – what if the data is not in tabular form and is irregular by nature? What if there are instances with diffe­rent dimen­sions? Let’s consider a scenario for the classi­fi­ca­tion of time series and assume we have a data set consis­ting of four diffe­rent short time series:

Introduction to ragged tensors

As you can see, the series differ both in the number and the time of the measu­re­ments. Since machine learning models usually require a fixed input size, it is somewhat more compli­cated to fit such data into our models.

There are a number of ways to deal with this type of input. For example, we could inter­po­late the series and take virtual measu­re­ments at the same points in time for each series:

Introduction to ragged tensors

Here we take the values of the time stamps 0, 2, 4, 6, 8 and 10, so that each series consists of 6 values. At this stage, however, we already have to select hyper­pa­ra­me­ters such as the type of inter­po­la­tion, the number of values, etc. However, we cannot rely on the accuracy of the inter­po­la­tion, especi­ally for extra­po­lated values and for values within large gaps between conse­cu­tive measu­re­ments (see the orange and green rows at time 10).

If we want to feed the data into a Tensor­Flow Keras model and do not want to use inter­po­la­tion techni­ques, a common practice is to pad the rows, e.g. with zeros at the end. This is neces­sary because Tensor­Flow combines data stacks that must have the same shape in every dimen­sion. A batch with the above 4 series would have the form (4, 6), where 4 is the number of series (= batch size) and 6 is the number of measu­re­ments per series.

However, the 6 results from artifi­cial data, either inter­po­lated measu­re­ments or fill values. To overcome the uncer­tainty and overhead of these two techni­ques, we can process the original data with ragged tensors.

Concept of frayed tensors

The concept of ragged tensors is surpri­singly simple once you under­stand the inten­tion behind it. Let’s stick to our example above with 4 time series. As you can see, the minimum number of measu­re­ments per series is 3, while the maximum number is 5. With padding, we would have to fill each row with zeros at the end (or sometimes at the begin­ning) to achieve a common length of 5.

In contrast, a ragged tensor consists of the conca­te­na­tion of all values from all series together with metadata indica­ting where the conca­te­na­tion should be split into the indivi­dual series. Let’s define our data frame df and then our ragged tensor rt:

Time Value
0 3
3 1
6 8
8 0
10 9
0 15
5 11
8 7
0 12
2 7
4 8
9 2
0 9
4 0
6 13
10 4
row_splits = [0, 5, 8, 12, 16]
rt = tf.RaggedTensor.from_row_splits(values=df.values, row_splits=row_splits)

<tf.RaggedTensor [[[0, 3], [3, 1], [6, 8], [8, 0], [10, 9]], [[0, 15], [5, 11], [8, 7]], [[0, 12], [2, 7], [4, 8], [9, 2]], [[0, 9], [4, 0], [6, 13], [10, 4]]]>

As we can see, the row_splits array defines the indivi­dual rows by speci­fying their start row (inclu­sive) and end row (exclu­sive).

That’s it. This is the really simple struc­ture of the ragged tensors. As an alter­na­tive to speci­fying row_splits, we can also create the same ragged tensor using one of the follo­wing methods:

  • value_rowids: for each row in the conca­te­n­ated series, we specify an ID number that indexes the indivi­dual row:
value_rowids = [0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3]
rt_1 = tf.RaggedTensor.from_value_rowids(values=df.values, value_rowids=value_rowids)

<tf.RaggedTensor [[[0, 3], [3, 1], [6, 8], [8, 0], [10, 9]], [[0, 15], [5, 11], [8, 7]], [[0, 12], [2, 7], [4, 8], [9, 2]], [[0, 9], [4, 0], [6, 13], [10, 4]]]>
  • row_lengths: we specify the length of each indivi­dual row:
row_lengths = [5, 3, 4, 4]
rt_2 = tf.RaggedTensor.from_row_lengths(values=df.values, row_lengths=row_lengths)

<tf.RaggedTensor [[[0, 3], [3, 1], [6, 8], [8, 0], [10, 9]], [[0, 15], [5, 11], [8, 7]], [[0, 12], [2, 7], [4, 8], [9, 2]], [[0, 9], [4, 0], [6, 13], [10, 4]]]>
  • Constant: We can define the ragged tensor as a “constant” by directly speci­fying a list of arrays:
rt_3 = tf.ragged.constant([df.loc[0:4, :].values, df.loc[5:7, :].values, df.loc[8:11, :].values, df.loc[12:15, :].values])

<tf.RaggedTensor [[[0, 3], [3, 1], [6, 8], [8, 0], [10, 9]], [[0, 15], [5, 11], [8, 7]], [[0, 12], [2, 7], [4, 8], [9, 2]], [[0, 9], [4, 0], [6, 13], [10, 4]]]>

Intern­ally, it does not matter which method we choose to create a ragged tensor, the results are all equiva­lent. Next, let’s look at how to perform mathe­ma­tical opera­tions with ragged tensors.

Working with frayed tensors

Tensor­Flow offers a very handy function to perform opera­tions on ragged tensors: tf.ragged.map_flat_values(op, *args, **kwargs). It does what the function name says – each ragged tensor in args is replaced by its conca­te­n­ated (=flat) version, omitting the stack dimen­sion. In our example, this is the same as if we were working directly with the df.values. The only diffe­rence is that the output of the opera­tion is again a Ragged Tensor with the same metadata infor­ma­tion about where to split. Let us consider an example in which we calcu­late the matrix product of the ragged tensor with a matrix m of the form (2, 5). Each indivi­dual series in our ragged tensor has the form (k, 2), where k corre­sponds to the number of measu­re­ments in the respec­tive series. Make sure that you start the floats first:

m = tf.random.uniform(shape=[2, 5])

(2, 5)

rt = tf.cast(rt, tf.float32)
result = tf.ragged.map_flat_values(tf.matmul, rt, m)
print(*(t.shape for t in result), sep='\n')

(5, 5)
(3, 5)
(4, 5)
(4, 5)

Perfect! The resul­ting ragged tensor has the same row splits as the input, but the inner dimen­sion has changed from 2 to 5 due to the matrix multi­pli­ca­tion. We could perform some more compli­cated opera­tions, for example if m is not a 2-dimen­sional matrix but a 3-dimen­sional tensor:

m = tf.random.uniform(shape=[2, 5, 4])

(2, 5, 4)

rt = tf.cast(rt, tf.float32)
result = tf.ragged.map_flat_values(tf.einsum, "bi, ijk -> bjk", rt, m)
print(*(t.shape for t in result), sep='\n')

(5, 5, 4)
(3, 5, 4)
(4, 5, 4)
(4, 5, 4)

As expected, the batch dimen­sion b corre­sponds to the length of the indivi­dual series, while the other dimen­sions come from m. Inciden­tally, tf.einsum refers to the Einstein summa­tion conven­tion, which is extre­mely practical when we are working with higher-dimen­sional tensors. Read more about it



One last point: It is also very easy to perform aggre­ga­tions over frayed tensors. For example, if we want to know the column-wise sum, we can use reduc­tion functions:

tf.reduce_sum(rt, axis=1)

<tf.Tensor: shape=(4, 2), dtype=float32, numpy=
array([[27., 21.],
       [13., 33.],
       [15., 29.],
       [20., 26.]], dtype=float32)>

There are many more opera­tions for frayed tensors, which are listed here.


We have learned about the struc­ture of Tensor­Flow Ragged Tensors and how to perform basic mathe­ma­tical opera­tions with them. They make it unneces­sary to use unnatural pre-proces­sing techni­ques such as inter­po­la­tion or padding. This is parti­cu­larly useful for irregular time series data sets, although there are many other appli­ca­tions. Imagine a data set with images of diffe­rent sizes – ragged tensors can even handle multiple ragged dimen­sions, perfect for this.

In a later post, I’ll dive a little deeper into working with ragged tensors as input types for a Keras model by treating the indivi­dual time series as sets and drawing atten­tion directly to the ragged tensors. Stay tuned!

Picture of Torben Windler

Torben Windler

Project request

Thank you for your interest in m²hycon’s services. We look forward to hearing about your project and attach great importance to providing you with detailed advice.

We store and use the data you enter in the form exclusively for processing your request. Your data is transmitted in encrypted form. We process your personal data in accordance with our privacy policy.