Laravel Performance
This episode is for members only

Sign up to access "Laravel Performance" right now.

Get started
Already a member? Sign in to continue
Playing
13. Chunking with Eloquent

Episodes

0%
Your progress
  • Total: 1h 9m
  • Played: 0m
  • Remaining: 1h 9m
Join or sign in to track your progress

Transcript

00:00
If you need to iterate over a huge amount of records for any reason, you're going to want to make use of chunking. And we're going to take a look at this
00:07
with a huge amount of posts in the database. So I've gone ahead and updated the blog seeder with 20 users that each have 10,000 posts. Probably unlikely, but we want a huge
00:19
amount of data here that we can test this out with. OK, so let's imagine that we're building out a console command in here. So let's use Artisan to make a command and say we want to create out a export posts command that we can run through Artisan.
00:36
So let's head over to that export posts command under app console and commands. And let's go ahead and just grab the name of this. In fact, we can change this around. So let's say posts and export.
00:48
Let's grab the name of that so we can use it in the command line. And at the moment, of course, this just does nothing. Let's just run it anyway. So we've got it in our console memory.
00:57
And in here, what we want to do is grab every single post from the database that's 200,000. And we want to iterate over them and do something. So how might we do this usually?
01:08
Well, we would probably go ahead and start either assigning and we would go ahead and say post get and we could either do a for each loop over these. So just put in the namespace for post real quick. We might say for each post as post and then we might do something in here.
01:24
So just as an example, let's go ahead and say this info and we'll just grab out the post ID just so we can see it in the console to test this out. Now I'm going to go ahead and run this. And what's actually happening is now Eloquent is fetching all of them records
01:39
because we're just using get here and it's putting them into memory. So you can see, first of all, this is taking a while to run. And what's actually happened here, although we don't see an error in the console itself, is this is run out of memory.
01:53
We don't see any output here at all. Nothing has happened and it's taken 10 seconds to run. So obviously this isn't working. Another alternative to a for each loop is to go ahead and use each because we get
02:04
a Laravel collection back when we use post. So we can go ahead and do this and we could say this info and post ID. And for each of these iterations, we'll get in here a post like so. OK, so let's run this.
02:20
We get exactly the same thing because effectively behind the scenes, this is just doing a for each loop over each of these records and nothing's happening again. This is going to take about 10 seconds. It's going to run out of memory and we don't see any output whatsoever.
02:34
So really, we cannot do this with this amount of records and even with a much smaller amount of records. It's probably not a good idea either because you're going to run into huge memory issues, particularly if you're running a command like this very frequently.
02:47
So how do we resolve this? What we can actually do is make use of chunking. So we're going to go ahead and get rid of all of this. We can actually get rid of the assignment
02:55
here because we don't really need this anymore. And instead of just getting all of these records, which is going to give us back 200,000, we're going to go ahead and use this chunk method.
03:05
What this will allow us to do is provide a value in here. And Laravel will fetch a thousand records, do something with or give you the opportunity to do something with these records, then it will clear them out of memory and it will give us the next 1000.
03:21
Again, we can export each of them posts. It will clear that out of memory. And again, it will grab the next 1000 until it has gone through every single post in the database or whatever data
03:32
you're working with. So let's look at how we actually use this. We define a closure here as the second argument. And in here we don't get a post itself. We get a collection of posts because we're grabbing the first 1000,
03:46
not iterating over each post. We're chunking this and getting a collection for each of these. So if we just call this post and just die dump on posts, what we actually see here when we run this is we get a collection.
03:58
We had up or reduce this amount quickly. We get a collection of posts, so let's just use get class in here just to actually show you that we get back a collection. OK, so now what we can do is because we're only grabbing a small amount here,
04:19
we can then do something like posts each because each is a method on a collection. Or if you really wanted to or needed to, you could do a for each loop in here. It's going to work in exactly the same way. So now what we can do is inside of here,
04:34
it's right over each of them posts and we now get a post for each of these. And then we can say this info and post ID. Or in this case, we would do something more useful inside of here. OK, let's go ahead and run this and see what happens.
04:50
So there we go. What's happening now is for every thousand we're chunking through these. We are outputting the ID of the post or doing something with it. And then it's just clearing this out of memory until we get to the point where we
05:03
have gone through all of these batches of posts and done something with them. So you can see here that we've gone all the way down to two hundred thousand pretty effortlessly. And of course, more importantly, without running out of memory.

Episode summary

In this episode, we dive into handling really large datasets with Eloquent in Laravel using chunking, which is essential if you ever need to deal with tons of records without running into memory issues.

We start by simulating a huge dataset: 200,000 posts distributed over 20 users. The demo project has already been seeded with this data so we can really put Eloquent to the test!

First, we try the usual ways to iterate through the records — grabbing all posts and looping over them with simple foreach or the each method. Both approaches show their downsides immediately, as grabbing all 200,000 posts at once quickly exhausts the available memory and doesn’t give us any output.

To solve this, we switch gears and use Eloquent’s chunk() method. With chunking, Eloquent fetches a manageable batch of results (like 1,000 at a time), does your processing, clears that batch from memory, and then moves on to the next. This way, we steadily and efficiently work through all the records in the database, without maxing out the server’s memory.

We walk through how to rewrite the command to use chunk(), iterating over each batch with each() or a plain foreach loop, and demonstrate how memory stays under control while successfully processing all records.

By the end, you’ll see how chunking makes handling massive datasets in Laravel not just possible, but smooth and safe!

Episode discussion

No comments, yet. Be the first!