How to Get All Keys in Redis?

If you are managing data in Redis, you are frequently required to see what keys are available in the database. This is particularly the case in scenarios such as debugging, monitoring, or merely having an overview of data. Although there are several options to achieve this goal, it is essential to be aware of the consequences of each method because some of them may affect performance.

On the other hand, knowing how to effectively retrieve keys is a must-have skill for a developer or a system administrator who works with Redis. It enables you to check your data, resolve issues, and get an understanding of the data structure of your application’s data. In view of this, the present guide is aimed at making the users knowledgeable about the most common methods, giving a description of their advantages as well as drawbacks.

We are going to reveal the fundamental commands which refer to key retrieval as well as talk about their performance features, and share the best practices for you to be sure that you do not harm the performance of your Redis server by mistake. So, let us get familiar with key management in Redis.The Redis database can grow to a considerable size over time. Sometimes you may need to know all the database records to remember some elements or to understand the logic. It is not a problem in Redis. However, before I give you a ready-made command, I will present you with the basics of Redis and commands.

Key Takeaways

• Knowledge of the differences between KEYS and SCAN is a prerequisite for a procedure that maintains a healthy Redis instance.
• The KEYS command is the easiest, but at the same time the most risky means for obtaining all keys. It is capable of stopping the server for a long time.
• The SCAN command is a better non-blocking alternative to the production environment, which is recommended. In it, keys are retrieved stepwise.
• If you want to make better use of the SCAN or KEYS commands, you should apply certain patterns to them so that you can get only those filtered results that efficiently.
• Do not run the command KEYS on your production server if you have a large dataset. It can lead to severe performance issues.

What is Redis?

Redis is the one who with the least effort finds the most needed books from the slow and big library and puts them right in front of you. So, you don’t have to go and look for the information every time in the huge and slow library, Redis just gets the data you use the most and keeps it right on the desk. That’s why websites and apps that use Redis are super fast.

Basically, it’s a desk that is designed to keep various things:

• Just small records (in the form of a string)
• Groups of things (like a shopping list)
• Sets of things (e.g., a list of your all-time favorite movies)

Now, because Redis is so adept at keeping everything on the fast desk, it servers over the lights too and in less time than expected. 

Key Features of Redis

The data structures and speed are just the tip of the iceberg when it comes to Redis, as it is also full-featured and versatile for use as a tool:

• High Availability & Clustering: Redis supports a replication architecture comprising a master and one or more replicas, where data is equally safe, and failover is possible. The Redis Cluster automatically splits the data across different nodes thereby allowing a big dataset to be scalable.
• Pub/Sub (Publish/Subscribe): Redis can be a message broker, thus, clients can physically publish the messages to a particular channel, and others can subscribe to it and read the messages. This feature is excellent for the implementation of real-time conversations and event-driven architectures.
• Lua Scripting: Redis provides the feature of server-side execution of complex, atomic operations using Lua scripts. In this way, the execution of multiple commands as one unit is achieved resulting in the efficiency of request-response communication over the network.
• Transactions: Redis allows basic transactions to be carried out through the MULTI, EXEC, and DISCARD commands. Such transactions guarantee that the operations included in the sequence will be executed atomically.
• Persistence: We can recall that both RDB and AOF give users strong choices for saving data reliably.

The Inevitable Question: Why Do We Need All Keys?

Let’s not jump right into the how? Before we have a brief look at the why? What would be the point of having a list of all keys in a Redis database? There is a long list of such reasons which are often related to debugging, monitoring, or administrative tasks:

• Debugging: If the application is not doing what it is supposed to do, you might want to look at the keys and their data types to grasp the system state.
• Data Migration: You may need to find all the keys that would allow you to transfer data from one Redis instance to another, or even to a different database.
• Monitoring and Analytics: The process of total keys or the key distribution can give a lot of valuable information about the usage of your application’s data.
• Backup and Recovery: One of the methods of backing up a Redis database entirely is by accessing all keys one by one.
• Key Expiration Management: Finding those keys which are going to expire soon or getting rid of old keys.

These legitimate usage scenarios make the ability to locate keys a fundamental part of Redis administration. Nevertheless, in the words of the famous Marvel comics: “With great power, comes great responsibility”. The wrong method can cause catastrophic effects.

The KEYS Command: A Tale of Convenience and Caution

The KEYS command is very likely the simplest method by all means to retrieve a whole list of keys. Its simplicity is, on the one hand, its main advantage and, on the other hand, its most considerable drawback.

How It Works

The KEYS command syntax is just KEYS pattern

The pattern parameter is a filter for the keys expressed in a glob-style pattern. The most frequent pattern is *, which corresponds to all keys in the database. For instance, to fetch all keys, you would issue the command KEYS *. If you wanted to collect all keys starting with user:, you would write KEYS user:*

Essentially, when this command is run, the Redis server executes a blocking operation. This means that Redis will cease to handle all other client commands until the whole dataset has been thoroughly scanned and the keys identified by the server are sent back to the client. This is the same situation as with a single-lane bridge where all traffic has to stop to let a very large vehicle pass.

The Danger of KEYS in Production

The blocking nature of KEYS is its Achilles’ heel. Let’s consider a few scenarios:

• Small Dataset (1000 keys): The KEYS command will likely execute in a few milliseconds. The impact is negligible.
• Medium Dataset (100,000 keys): The command might take a few hundred milliseconds. This could cause a noticeable slowdown in your application, leading to a slight increase in latency for other requests.
• Large Dataset (10 million keys): The KEYS command could take several seconds, or even minutes, to complete. During this entire time, your Redis server is completely unresponsive to all other requests. Your application will experience timeouts, and users will face a complete service outage.

The reason why the official Redis documentation and the entire Redis community are very adamant about not using KEYS in a production environment with a large number of keys is because of the blocking issue. It is a command that should only be run in the development, testing, or by an administrator on a separate, non-critical instance.

The SCAN Command: The Safe and Smart Alternative 

Knowing the restrictions and hazards of the KEYS command, Redis came up with the SCAN command in version 2.8. SCAN is a partial iterator, which enables you to fetch keys without blocking the server. This is a much more intelligent and production-friendly approach. Utilizing alternative approaches like SCAN can provide a safer and more efficient solution for production environments, just as choosing optimized Magento Hosting solution ensures efficiency and scalability for high-performance eCommerce sites.

How It Works

The SCAN command achieves its goal by browsing through a non-blocking keys dictionary. In every call, it returns a small batch of keys and gives a continuation point, called a cursor, which is used to proceed with the scan from where you left off. This operation is repeated until the cursor provided is 0, which is an indication that the whole dataset has been traversed.

The form for SCAN is as follows: SCAN cursor [MATCH pattern] [COUNT count]

Let’s look at each argument in detail:

• CURSOR: The non-negative integer of the iterator’s state. 0 is the cursor value you have to pass to initiate a new iteration. Besides that, Redis returns a new non-zero cursor to be used in the next SCAN call.
• MATCH pattern (optional): A glob-style pattern for keys filtering, the same as with the KEYS command. It is highly recommended to be used for targeted searches.
• COUNT count (optional): This argument is a suggestion for the number of elements to return per call. Redis, however, may return more or fewer items than the count you specify. The default value is 10. The scan becomes faster with a higher COUNT, but more network bandwidth and slightly more CPU per call will be used. On the other hand, a lower COUNT is less resourceful on the server.

Anatomy of a SCAN Iteration

Usually, a SCAN procedure goes through the following steps:

• The cursor is initialized to 0.
• SCAN is invoked using the current cursor inside the loop.
• The keys that came with the output are handled.
• The cursor is updated with the one that came with Redis.
• As long as the cursor is not 0, the loop is repeated.

For example, in a terminal environment, you could illustrate this as follows:

> SCAN 0

1) "17"

2) 1) "key1"

   2) "key2"

   3) "key3"

> SCAN 17

1) "33"

2) 1) "key4"

   2) "key5"

> SCAN 33

1) "0"

2) 1) "key6"

Here, the first SCAN call gives back a new cursor of 17. So, the next call is with 17 as the cursor, and it keeps going like this. The last call is with a cursor of 0, which indicates that the scan has stepped through the entire keyspace.

The Advantages of SCAN

There are many reasons why the SCAN command is better than KEYS. Here are some of them:

• Non-Blocking: Throughout the scan, SCAN doesn’t block the server. The server can still take other client requests in the meantime. So, the system works at its full potential, and the clients are happy.
• Incremental: The command does not process all keys at once, but rather in small batches. This is very important for big datasets, where a full scan can consume too much memory and take a lot of time.
• Resumable: The cursor enables stopping the SCAN operation in the middle and continuing it later at the same point. This feature is applicable to situations with a very long script or when a network connection keeps dropping.
• No state is maintained on the server for the SCAN operation except for the cursor. This means that it is a lightweight and scalable solution.

Other SCAN-like Commands

The SCAN command belongs to the same family as SSCAN, HSCAN, and ZSCAN. These commands allow you to iterate incrementally over different Redis data structures. Their efficient usage is the basis for effective data handling.

• SSCAN: Goes through the elements of a Set one by one.
• HSCAN: Goes through the keys and values of a Hash one by one.
• ZSCAN: Goes through the elements and scores of a Sorted Set one by one.

These all use the same cursor/MATCH/COUNT conventions as SCAN, thus they become easier to use once the basic concept is grasped.

Data types

The advanced type support in REDIS increases the capabilities of this database.

Strings

Strings of characters are perhaps the most commonly used data type. The popularity is due to the fact that you can put JSON, XML, and any other text there, which means that you have almost no restrictions. In addition, REDIS does not analyze the content of the transmitted data in any way, which guarantees its immutability.

For each data type, you will find commands that are designed for that type.

Lists

It is a type that you can compare to an array of strings. Access to the data in the list is done in the order of addition or based on the index number. The lists themselves can be gigantic collections, as they can contain over 4 billion items.

To create a list, add the first element to it. You can do this with the LPUSH or RPUSH command. Both of these commands are responsible for adding one or more items to the list. The difference is that LPUSH adds items to the top of the list, and RPUSH adds items to the end of the list.

Hashes

They are associative arrays. Up until now, added elements had only a key and a value. Associative arrays extend the possibilities by defining attributes for a given key. This structure gives you the impression of working with records in a database. 

You can define a new item that will be a simple mapping of a user from the database. It will have the attribute name and email, and the key will be the identifier preceded by the user_ prefix.

hmset user_1 name “Jon Snow” email [email protected]

Having a user added this way, you may display all the information by the HGETALL command. As a result, you will get a list, where the field is displayed sequentially and then the value.

Sets

Sets are very similar to lists, except that they contain unique and unordered values. It makes it a much more efficient type than lists, and if you plan to have a lot of items that you will modify or search frequently, then you should use collections.

How to check all keys in Redis?

Now that you know the basic concepts of Redis, I can get to the point. As you already know, to get a specific target, you will use the GET command. However, to list all the keys in the Redis database, you need to use another command: KEYS. Just enter KEYS followed by a specific pattern, and Redis will search the database for all keys that match that pattern.

If you want a list of all keys, use the asterisk (*). So the command:

KEYS *

should return all the keys in the database.

You can also use redis-cli to get a list of all keys using the following syntax:

$ redis-cli KEYS \*

Or you can limit the keys returned with a pattern. For example

$ redis-cli KEYS V*

will return all records starting with the letter V

Advanced Considerations and Best Practices

Performance of SCAN

Despite the SCAN being non-blocking, it is not without its limitations. The COUNT argument plays an important role in optimizing performance.

• High COUNT: The overall process can be faster since there are less round trips between the client and the server. However, it can cause a short-term heavy loading of the Redis instance, as each call will provide a larger batch of data. In this way, it would be a good strategy for dedicated Redis instances that are not critical ones.
• Low COUNT: The overall process will be slower because of the increased number of round trips, however, it will be much more gentle on the Redis server’s CPU. This version is safer for a busy production environment where the aim is to minimize any potential impact.

Starting with the default COUNT of 10 is probably the best choice, and afterwards, if your application’s workload and server resources consider it appropriate, you can experiment with higher values.

Guarantees and Guarantees

Firstly, it is crucial to understand what SCAN can guarantee and what it cannot.

• No Duplicates: SCAN does not promise that you won’t come across the same key more than once. This situation can occur if the dictionary is rehashed due to resizing while the scan is in progress. The client-side code should be capable of dealing with duplicates, usually by maintaining a separate set of processed keys.
• No Missing Keys: SCAN tries to return all keys, but rehashing may result in some keys being missed or returned twice if they have been added or removed during the scan. This is a compromise for the non-blocking nature. For the majority of use cases, this is an acceptable trade-off.

When is KEYS Acceptable?

Though we have emphasized the risks of using KEYS, there are several situations where employing it is not a mistake:

• Development Environment: In the case where you are working on your local machine and have a small, controlled dataset.
• Testing and CI/CD: In a Continuous Integration/Continuous Deployment pipeline where the Redis instance is only for the testing purposes and is ephemeral.
• Dedicated, Single-Purpose Instance: If you have a Redis instance that is separate from the live traffic and its only purpose is a one-off administrative task.

In all other cases, especially in the production environment, the only acceptable answer is to use SCAN.

Conclusion: The Path Forward

Trying to get all the keys from a Redis database is a common procedure, but it is essential that it is done in a careful and thoughtful manner. Although the KEYS command may look simple, it is a dangerous tool that can severely affect the performance of the system and lead to downtime of the service in the production environment. Besides that, its blocking attribute makes it not suitable for large datasets.

The SCAN command is the right and the most responsible way to solve such a problem. Its stepwise, non-blocking character guarantees that your Redis server will be responsive and accessible even when you are going through billions of keys. By knowing and using the SCAN command family, you can safely manage and check your Redis data without jeopardizing the application that you are using.

Always keep in mind: a little patience in the Redis world goes a very long way. Pick SCAN instead of KEYS, and your server will be grateful.

FAQS