2 years ago

#42319

test-img

HyeonPhil Youn

Tensorflow tf.data.Dataset interleave() returns Unexpected result

I was following some another example from this link which is about the tf.data.Dataset.interleave() method.

import tensorflow as tf
import tensorflow.keras as keras

def do_range(i):
    for j in tf.range(i):
        yield j

ds = tf.data.Dataset.range(10).interleave(
    lambda ind: tf.data.Dataset.from_generator(do_range, args=(ind,), output_types=tf.int64))

I tried to understand this example, so I started with the range(3), range(4), ..., and so on.
For example,

ds = tf.data.Dataset.range(3).interleave(
    lambda ind: tf.data.Dataset.from_generator(do_range, args=(ind,), output_types=tf.int64))
res = [x for x in ds.as_numpy_iterator()]
# This returns [0, 0, 1]
ds = tf.data.Dataset.range(4).interleave(
    lambda ind: tf.data.Dataset.from_generator(do_range, args=(ind,), output_types=tf.int64))
res = [x for x in ds.as_numpy_iterator()]
# This returns [0, 0, 0, 1, 1, 2]
ds = tf.data.Dataset.range(5).interleave(
    lambda ind: tf.data.Dataset.from_generator(do_range, args=(ind,), output_types=tf.int64))
res = [x for x in ds.as_numpy_iterator()]
# This returns [0, 0, 0, 0, 1, 1, 1, 2, 2, 3]

Until range(9), it returns the dataset I expected.
But from range(10), it doesn't return the dataset I expected.

ds = tf.data.Dataset.range(10).interleave(
    lambda ind: tf.data.Dataset.from_generator(do_range, args=(ind,), output_types=tf.int64))
res = [x for x in ds.as_numpy_iterator()]
# This returns [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 2, 2, 2, 2, 2, 2, 1, 3, 3, 3, 3, 3, 2, 4, 4, 4, 4, 3, 5, 5, 5, 4, 6, 6, 5, 7, 6, 7, 8]
# Notice that 0s, 1s, 2s are not gathered..

What is going on here? Isn't this should be [0, ..., 0, 1, ..., 1, 2, ..., 2, ..., 8] ?

python

tensorflow

tensorflow-datasets

interleave

0 Answers

Your Answer

Accepted video resources