How to use MongoDB with Sequence class or generator from Keras

When working with large datasets in deep learning, it’s important to efficiently load and preprocess data. Keras provides the Sequence class and generator functionality to handle data input for training, but what if you want to use MongoDB as your data source? In this article, we will explore how to use MongoDB with the Sequence class or generator from Keras.

Using MongoDB with Sequence class

The Sequence class in Keras allows you to create a custom data generator for training and evaluating your model. To use MongoDB with the Sequence class, you can create a custom data generator that queries your MongoDB database for the required data. You can then create a subclass of the Sequence class and override the __getitem__ method to fetch data from MongoDB.

Here’s an example of how you can create a custom Sequence class that uses MongoDB as its data source:

    
    from keras.utils import Sequence
    from pymongo import MongoClient

    class MongoDBSequence(Sequence):
        def __init__(self, batch_size, mongo_uri, db_name, collection_name):
            self.batch_size = batch_size
            self.client = MongoClient(mongo_uri)
            self.db = self.client[db_name]
            self.collection = self.db[collection_name]
            self.num_data = self.collection.count_documents({})

        def __len__(self):
            return int(np.ceil(self.num_data / float(self.batch_size)))

        def __getitem__(self, idx):
            batch_x = []
            batch_y = []
            start_idx = idx * self.batch_size
            end_idx = min((idx + 1) * self.batch_size, self.num_data)

            for i in range(start_idx, end_idx):
                data = self.collection.find_one({'_id': i})
                batch_x.append(data['image'])
                batch_y.append(data['label'])

            return np.array(batch_x), np.array(batch_y)
    
    

Using MongoDB with generator

If you prefer to use the generator functionality in Keras, you can also create a custom generator that fetches data from MongoDB. Your custom generator should yield batches of data during training and evaluating.

Here’s an example of how you can create a custom generator that uses MongoDB as its data source:

    
    def mongo_data_generator(batch_size, mongo_uri, db_name, collection_name):
        client = MongoClient(mongo_uri)
        db = client[db_name]
        collection = db[collection_name]
        num_data = collection.count_documents({})

        while True:
            start_idx = 0
            end_idx = batch_size
            while start_idx < num_data:
                batch_x = []
                batch_y = []

                for i in range(start_idx, end_idx):
                    data = collection.find_one({'_id': i})
                    batch_x.append(data['image'])
                    batch_y.append(data['label'])

                yield np.array(batch_x), np.array(batch_y)

                start_idx = end_idx
                end_idx = min(start_idx + batch_size, num_data)
    
    

Conclusion

In this article, we have shown how you can use MongoDB with the Sequence class or generator from Keras. By creating a custom data generator or generator function, you can efficiently load and preprocess data from MongoDB for training and evaluating deep learning models.