Build a Livewire CSV Importer
This episode is for members only

Sign up to access "Build a Livewire CSV Importer" right now.

Get started
Already a member? Sign in to continue
Playing
17. Batching CSV imports

Episodes

0%
Your progress
  • Total: 2h 20m
  • Played: 0m
  • Remaining: 2h 20m
Join or sign in to track your progress

Transcript

00:00
Okay, so the goal now is to build up a batch of individual jobs, which we're going to call something like import CSV and
00:09
This will go and get pushed onto the queue and worked through much like we've already spoken about So we're gonna go straight in and create out the import CSV job So let's go over to the command line and say PHP artisan make job and we're gonna call this import CSV We're not actually gonna fill this out with anything useful at the moment, but we will actually see this now running in horizon
00:33
so let's go over to that import CSV job and Let's take a look. So if we come down here, we've got a constructor We've got a handle method. One of the things that we need to do to make this batchable is to add in another trait So we're gonna go and I'm just gonna do this on a separate line. We're gonna say use
00:54
Batchable like so and then that's it. So this will just tell Laravel that this is a batchable job So everything in here is absolutely fine. What we're gonna do just for now is Log out if we can get this, right? Out this info. We don't need double slashes and we're just gonna say
01:14
Run, okay, and we're gonna come over to our Laravel log and just clear this out. So it's nice and fresh So by the end of this when we do hit that import button We should have the batch created this should work through each of these Batches of records and we should see over in the Laravel log
01:31
This dumped out ten times and then what we can do Of course is just fill in this handle method with what we need to do to actually insert these into the database So now what we need to work out is from these chunks How do we get this into a batch or into an array?
01:46
So the way that bus batching works is we use the bus bus facade and we go ahead and call you guessed it the batch Method now into this we pass an array So what we're eventually going to end up with is something like this new import CSV Like so new import CSV ten times with ten records each into these
02:07
So we need to actually pass the records in here that we want to batch So we need to basically build this up dynamically The reason we of course we need to build this up dynamically is we have no idea how many chunks there are we have no Idea how many total record records there are so we need to build this array up
02:25
Based on the amount of chunks that we actually have So let's go ahead and just comment this out for now and let's do that now. So we're gonna go ahead and create a Batches variable it doesn't really matter what it's called for now because we're gonna pass it directly in And we're gonna go ahead and collect up what we get back from these chunks or this chunk iterator
02:45
So I'm gonna get rid of this chunks variable and I'm gonna pop that straight into there And I'm actually gonna pull this down just it's a little bit easier to read like so and then we're gonna go ahead and map through these so we're gonna map through and Let's figure out what we're gonna get for each of these
03:01
Well, we're gonna get a chunk which in our case is gonna contain ten records because we're chunking in this by ten Of course when we get to the end if we had a hundred and one records We would end up with eleven chunks of ten and the last one would be Just have one record in there. So it really depends on how many records there are
03:19
Now what we're gonna do is return a new Import CSV job and we're gonna pass that chunk through to here in some way We're not actually gonna do that now, but that's pretty much what's gonna happen So let's die dump on this array that we get back what we actually need to do here is
03:38
In fact, we don't necessarily need to cast this back to an array We could just pass it directly in but just for the sake of showing you this Let's go ahead and cast this back to an array. So let's go through the whole process and Upload a file here
03:53
Let's choose that customers file again. Just Go ahead and import these I think actually what's happening here is we didn't add defer back on here So this is a little bit jumpy. Let's really quickly fix that up now. So let's go to the CSV importer and let's go over to
04:10
This here and let's add a defer on to that So if you found that's been jumpy, we removed the defer earlier so we could demonstrate this but let's put that back now so let's go through this whole thing again really quickly and Let's import the ID first name
04:26
Last name and email. Let's hit import and there we go So we now have something that resembles what we want to see passed into that bus batch method So now we can just go ahead and pass it in. So it really depends on what you want to do here You could leave this variable around. I think I'll do that because it kind of looks a little bit better
04:45
let's go ahead and say bus batch and let's say batches and Then with our bus batch to actually send this off we're gonna then call dispatch So that's what we need to do We've got empty jobs in here, of course, because the import CSV is only logging something at the moment
05:02
So we don't really care that it's actually doing anything There are no arguments to pass through to this at the moment But that's just building up ten of them and they're gonna be run on here now if we come over to our
05:13
Con our terminal and we just exit out of horizon just temporarily while we run this We'll then see in here kind of ready to go. So let's go ahead and import this click upload a file choose customers go ahead and choose each of these and Then hit import. There we go. Now. Let's just take a look at this
05:34
So we're gonna come over to the database and go over to job batches. We've got a unique ID for this No names, we've not specified anything total job 10 We kind of got that because there are 10 batches of 10 making a hundred records we've got 10 pending and
05:47
It's not cancelled and it gives us a created at and of course finished that is null because it's not finished So that is the batch the actual jobs themselves are inside of Redis So if we just come over to horizon and give this a refresh We've got 10 jobs in the past hour if we just head over to our batches. We've got this batches
06:06
The information here that we can actually look at we've also got 10 pending jobs as well So these are waiting to happen because we haven't run PHP arts and horizon Let's go ahead and do that now and remember keep an eye on our log to see if these get logged out so I'm gonna go ahead and really quickly run this and then head over to here and
06:26
you can see that these should be running through let's just keep an eye on these there we go and Pending jobs are gone So we've got 10 completed jobs now and sure enough if we head over to our Lara for log We have a 10 lines of this log that have been run. So that's the entire batch
06:45
Remember, we're iterating over these chunks. So we're not going to get a memory issue we're creating a batch out of these each of these import CSV jobs are going to be run after each other and We can easily get the progress back from this either Directly from the batch or in our case, we're going to update the import model that we created earlier
07:04
Remember we created that just down here. So there we go That is collecting up all of these jobs that need to be dispatched and running them on our queue So now really what we need to do is go ahead and fill in the import CSV job To actually do what we need it to do

Episode summary

In this episode, we dive into batching CSV imports using Laravel's job batching features. We start by creating a new job called ImportCSV and making it batchable with the Batchable trait. At this point, we're not stuffing the handle method with any real logic yet—just some basic logging so we know the batch is running.

From there, we talk about chunking the CSV records into batches (like groups of 10), dynamically building up an array of jobs depending on how many chunks we have. So, for a 100-row import, we'll get 10 jobs (with 10 records each), and if there's a remainder, the last job will just have those extra records.

We go through the process of wiring up the batch dispatch with the Bus facade, showing how each chunk becomes a new ImportCSV job. There's some live coding as we upload a file, check the database for batch metadata, and look at the jobs queued up in Redis and displayed in Laravel Horizon.

Finally, we kick off the worker and watch the logs update in real time as the queued jobs get processed. The episode wraps up with the batch running as expected, ready for us to finish up the actual import logic inside the job itself next time.

Episode discussion

No comments, yet. Be the first!