How to Read a File Line by Line in Python

Using a file line by line in Python is a memory-friendly way to process text. This approach is helpful when dealing with large text files, which would not be an efficient way to load the entire file to memory. It offers the best speed and also provides a great deal of control regarding how each line is processed.

Reading files line by line is an essential part of any file manipulation you will do with Python, especially when doing log file analysis, filtering data, and scanning content. Python provides different methods to read lines from files, including using a standard for loop, the readlines() method, and the readline() method. 

In this article, we will explain each method step by step, demonstrate how they work with examples, and provide comparison scenarios so you can determine the best one to use for your needs.

Why Read a File Line by Line?

Reading a file line by line is a great way to process a file in Python if the file itself is large and you don’t want to use a lot of memory. It is good when working with log files, big text documents, or data files where you are processing through the file like a stream and don’t want to load everything.

It allows for better performance and control especially when the processing involves filtering or analyzing a subset of data lines. This means the program is more responsive and stays in memory while only processing small fragments instead of trying to load everything at once.

Common reasons to read a file line by line include:

• Low memory usage — only one line is loaded at a time
• Easy line-by-line data processing — each line can be evaluated or modified separately
• Suitable for log file analysis — ideal for monitoring or extracting information from logs

This method allows for memory efficient reading of files, which is extremely common in data processing, automating workflows, and other script-based applications. 

Method 1 – Using a for Loop

The most straightforward method in Python to read data from a text file one line only at a time is using a for loop. This provides execution from the head to tail of the file and does not require the entire file to be loaded into memory, which makes it memory efficient and useful for very large files.

To use the for loop you must first establish that the file exists in the correct location. If your script is running from the same folder as the text file, the file should be in that folder. First we will assume the filename is named filename.txt, and that it contains the following text:

Line one
Line two

Here is how you would read and print each line:

with open('filename.txt') as file:
    for line in file:
        print(line.strip())

In the above code, open(‘filename.txt’) opens ‘filename.txt’ to read mode (read operation). The with keyword is used to ensure that the file is closed when the operation is complete. Then we do for line in file: to read the file one line at a time. Finally, print (line.strip())  prints the line after processing it to remove extra whitespace or newline characters. 

After that you can see the output by opening a command prompt and type:

$ python read_file.py

If your file is in a different directory, provide the full path:

with open('/path/to/your/file/filename.txt') as file:
    for line in file:
        print(line.strip())

This method works nicely for virtually all practical purposes, such as to process log files as before, to pre-filter a data file or to print one line of output at a time. It is clean, concise, and “pythonically correct” (insofar as the method of looping through a file is a common procedure). 

Method 2 – Using readlines()

The readlines() method reads all the lines from a text file at once, and stores each line as a element in a list. Below the previous example is a read operation using readlines(). To test if this code works for you, place the text file named filename.txt in the same directory/folder as your python script file. Then save the script, it should have a .py file extension, of something like read_file.py. 

The content of filename.txt is:

Apple
Banana
Cherry

The python script of read_file.py is:

with open('filename.txt') as file:
    lines = file.readlines()
    for line in lines:
        print(line.strip())

Next, after saving the code, you need to execute it to see the output using:

$ python read_file.py 

In this example, we read the entire data file at once, then stored each line as an element in a list. Then we looped through each line to process it. This method is easy to implement, but you are loading all data into memory at once. This method is not the best approach for processing the data if you are reading a large file.

You can also use the readline() method with loop as well:

with open('filename.txt') as file:
    line = file.readline()
    while line:
        print(line.strip())
        line = file.readline()

Here, only one line will be loaded in memory at a time. The iteration will continue until there are no more lines to read. In this way, you are using less memory, and processing will work better for files that are larger or streaming data sources.

Both approaches are acceptable ways to retrieve text formatted data, and the approach you choose should consider the size of the file and your specific application’s memory constraints.

Method 3 – Using fileinput Module

The fileinput module is useful when you want to process multiple files with a single iteration. This is often more efficient and cleaner than having multiple nested iterations, and tends to be very useful when processing logs or datasets stored using two or more text files.

To check out this method, create two sample text files called file1.txt and file2.txt, and place them in the same directory as your Python script. The following text is to be included in file1.txt:

Line one
Line two
Line three

Whereas the content for file2.txt is: 

Apple
Banana
Cherry

Now the python script to read multiple files line by line, you can use the following Python script:

import fileinput

for line in fileinput.input(files=('file1.txt', 'file2.txt')):
    print(line.strip())

Conclusion

Reading a file line by line in Python is a common task is file handling. In this article I have outlined several usable methods to do this, using for loops, readlines(), readline(), and the fileinput module. Each method has specific needs that need to be addressed according to the size of the file, memory constraints and the number of files that need to be processed. 

With small to medium sized files, readlines() is the easiest method, while readline() and loop method provide more flexibility with memory when handling larger files. The fileinput module is the best route when dealing with multiple files in turn. The method you select will depend on your use case. Testing each one out (providing sample outputs) will allow you verify it’s accuracy and gain confidence before implementing in practice.

FAQ