Python Generators

Yield Keywords, iterables & iterators concepts will be applied in this tutorial.

A Generator in python programming is said to be nothing but a function that returns a value using the yield keyword and not using the return statement.

If a custom iterator is to be written, the __iter__() and __next__() method will have to be implemented, and the mechanism of returning the next value must be defined by maintaining the state and raising the StopIteration exception when no more values are left for iterating.

Generator aids in creating an iterator in which state it is and it is maintained by using the yield keyword to return data.

Any function that contains the yield keyword will return an iterator object which will be iterated over using a for loop after it has automatically become a generator.

Creating a Python Generator

To convert a simple function into a generator function requires a magic recipe known as the yield keyword. There is a single or multiple yield statement needed to return some data from the generator where each time the generator is called the yield statement stores the state of the local variables and a result is yielded.

# A simple generator function
def new_gen():
    n = 1
    print('First execution...')
    # Generator function contains yield statements
    yield n

    n += 1
    print('Second execution...')
    yield n

    n += 1
    print('Third execution...')
    yield n

# main method
if __name__=='__main__':
  
  # new_gen function will not execute, but return a generator
  x = new_gen()
  # calling next on variable x
  print(next(x))
  # calling next on variable x
  print(next(x))
  # calling next on variable x
  print(next(x))

Output:

First execution...
1
Second execution...
2
Third execution...
3

A python generator called new gen() has been defined in the example above. In new gen(), a variable n is assigned some value, allowing it to print something and then yield a value. Inside the function, we have repeated it 3 times.

If a generator function is called like an ordinary function, the programming won't get executed because an iterator object is returned. Hence a variable x is taken to store the iterator object and then the next() method is used to execute the method. And every time the execution of the generator starts from where it last left i.e. the last yield statement.

Yielding into Python List

A generator in python programming can be directly used to form a list by generating values as shown in the example below, where a generator is used to generate a series of values and stored the values yielded by the generator into the list called mylist.

# defining a generator
def generator():
    x = 0
    while x < 5:
        yield x
        x += 1

# yielding values directly into a list
mylist = list(generator())

# use for loop to print mylist items
for x in mylist:
    print(x)

Output:

0
1
2
3
4

Generator vs. Normal Function vs. Python List

The major difference between a generator and a simple function is that a generator yields values instead of returning values. In simpler words, a simple function just returns a single variable and then it ends but a generator can yield a series of values by providing an iterator object on which we can iterate.

A generator must have a yield statement; else it will just be an ordinary function like the rest functions.

The generator stores its local variable state and returns the control to the caller as it yields. And when called upon again, it starts from where it left in the last call. A normal function doesn't support this behavior. Hence, a generator is more than an ordinary function.

There are differences between a generator and a list but the biggest difference, as well as the biggest advantage of a generator, is that the generator never stores any values; it just yields values one by one, therefore, saving space as well as time while the list does none of these.

Python Generators with a Loop

As an iterator object is returned by a generator, loops like for loop can be used to iterate over it and print the yielded values.

# generator to reverse a string
def reverse_string(my_str):
    length = len(my_str)
    for i in range(length-1, -1, -1):
        yield my_str[i]

# using for loop to reverse the string
for char in reverse_string("tutorialwithexample.com"):
    print(char)

Output:

m
o
c
.
e
l
p
m
a
x
e
h
t
i
w
l
a
i
r
o
t
u
t

Generator Expressions in Python

Generator Expressions can be used just like list comprehension to create a python generator's shorthand. For instance,

data = [0, 1, 2, 3, 4]
# creating a generator using generator expression
new_generator = (x*x for x in range(5))

for each in new_generator:
    print(each)

Output:

0
1
4
9
16

From the example above, an iterator that is similar to the list data is generated by the generator expression with each of its values squared. Rather than creating a list of squares, the result has been printed directly without the need to store a single value.