Build a Livewire CSV Importer/Exporter
This episode is for members only

Sign up to access "Build a Livewire CSV Importer/Exporter" right now.

Get started
Already a member? Sign in to continue
Playing
09. Queuing the export

Episodes

0%
Your progress
  • Total: 2h 18m
  • Played: 0m
  • Remaining: 2h 18m
Join or sign in to track your progress

Transcript

00:00
So when it comes to queuing your exports and imports, you can pretty much use any method you want, but I'm gonna go ahead and use Redis alongside of Laravel Horizon so I can monitor this.
00:09
And then when we get into production, eventually we can configure this with a certain amount of processes and we just have a lot more control over our queue.
00:17
So let's go ahead and get this installed. Prerequisite to this is you'll need Redis installed. And because of that, over in your environment file, under your queue connection,
00:27
you're gonna need to change that over to Redis. So once that's all good, let's go ahead and get this installed and I'll show you how to set it up
00:34
if you've not worked with it before. So let's do a composer require on Laravel Horizon. Once that's finished, we'll run the Horizon installation process,
00:43
which will create out everything we need, including the configuration file on the service provider. And to be honest, we don't really need to do too much. Eventually you're gonna need to come over
00:53
to config and Horizon, and you'll need to configure this to set up your environment. So let's just go down here
01:02
and look at our queue worker configuration. So we've got our defaults here, our environments for production, for local, et cetera, et cetera.
01:14
We don't need to worry about that too much for now because we're just gonna be testing this locally. So we're gonna go over to our local app slash Horizon and that's gonna give us our Horizon dashboard.
01:28
Now it's inactive at the moment because we haven't started Horizon. So we're gonna say PHP Artisan Horizon just on its own, and that's gonna start up Horizon for us.
01:38
So now that is active. Any jobs that we pushed this default queue, we should see here. We'll be able to check the pending jobs.
01:47
We'll be able to check the completed jobs. And more importantly, we'll also be able to see our failed jobs in case anything goes wrong.
01:55
Now, luckily for us, the Laravel XL package pretty much just has queuing already built into it. We don't need to create out a separate job
02:06
and then push this specific store method into here. So the way that this works is, when we queue this, it will be chunked up into chunks of data
02:15
and we can customize that and I'll show you how to do that as well. So how do we do this? How do we put this into our queue?
02:21
Well, we, instead of using the store method, use the queue method. The first argument through to here is the exporter. It works in exactly the same way.
02:32
The second is the file. The third is where we want this to be stored. And the fourth is again, the CSV. What we can then do,
02:40
depending on your strategy, you might have a separate queue for your exports. So you could say something like on queue exports. I'm just gonna push this to the default queue,
02:50
but once you've got this configured, you can create an entirely separate queue for this. Now that is pretty much it. So let's go ahead and try this out.
02:58
I'm gonna go ahead and get rid of the two transaction files that we've already exported here. And let's go over and monitor this over in Horizon as we export this.
03:09
So let's go ahead and just choose just page one for now. Let's go ahead and start the export and let's head over to Horizon and see what's happening. So we've got pending jobs, completed jobs,
03:21
and you can see that sure enough, because we only had 10 records in there, that was pretty straightforward. Now, all of these jobs that have been created,
03:29
we don't need to worry too much about these because they're just internal to the Laravel Excel package. But as long as this hasn't failed, then you're good to go. And what this should have done is created this out.
03:40
Now, that's fine to queue that, but the benefit of queuing obviously comes in when we have more and more records. So if we were to now go over here and just click export
03:52
and we wanted to export 100,000 records, now we see the benefit of queuing because this is just going to work away in the background. Let's go ahead and click start export
04:03
on this 100,000 records. And I'm just going to close the modal off. Eventually we're going to go and redirect the user over. Let's go over to pending jobs.
04:12
And you can see that all of these pending jobs add or append query to sheet are being rolled in. So this is chunking this now by a default amount and it's building up this export for us.
04:26
Of course, it hasn't created the export at the moment because it's so large. It's just going to take quite a while to go through here. And eventually we're going to end up with this file in here.
04:36
So just while we are waiting for that, let's go over to our transaction export and let's take a look at how we can specify the chunk amount. Once again, because this is a very interface driven package,
04:49
we would use the with custom chunk size interface and let's go ahead and pull out the method stuff that we need to implement here, which is chunk size. And if we find this, let's find out where that's added that.
05:06
We just need to return any chunk size that we want. So you can just customize that based on what you feel is best, it's not entirely necessary.
05:14
Okay, let's check in on the progress of our export and yeah, sure enough, it still looks like this is going on. So I'm just going to wait for this to finish and I'll skip ahead and then we'll see the final result
05:26
of our 100,000 values being exported. Just while we're waiting for our export to finish, I'm going to talk about this exportable trait that I pulled in a little bit earlier.
05:37
I forgot to explain what this actually does. If we take a look at this, you can see that inside we've got a bunch of methods. Now, by applying this particular trait
05:48
to our transaction export, what this means we can do is wherever we're doing this, we can actually change this up to not use the Excel facade. So we could do something like this instead.
05:59
So we could take this, let's just copy and paste this actually just to keep the code that we've already got. So we could do something like this,
06:07
then we could say something like queue. So let's just open up that exportable trait again, look at queue and you can see that sure enough, that gives us the ability to specify the file path,
06:19
the disk, the writer type. So we can say queue, export file, file system and the CSV type just in here. Let's grab that.
06:34
So if you prefer that syntax, then sure enough, you can go ahead and do that. I'm not actually gonna do this for now, but that is why I added in the exportable trait over here
06:48
just in case we need to. Okay, let's carry on waiting for our export to finish and then we'll wrap things up. Okay, so we've just arrived at the end of this export.
06:58
It took a huge amount of time. We'll look at scaling this a little bit later, but if we head over to our editor, you can see that we have this new record created just here.
07:08
I'm gonna attempt to open this up in my editor. It's a huge file because of course, we have 100,000 records in here, but it did the job. It did export every single one of them behind the scenes.
07:21
And of course, a little bit later, as we saw from the introduction, we will also create out a specific export page as well. So we can see the progress of all of our exports.

Episode summary

In this episode, we focus on how to queue up exports (and imports) in Laravel, especially when using the Laravel Excel package. We start off by discussing different options for queue drivers, and then dive into setting up Redis along with Laravel Horizon to give us a nice dashboard for monitoring our queued jobs.

We walk through installing and configuring Redis as our queue connection, update our .env accordingly, and then get Laravel Horizon set up via Composer. Once that's done, we touch on configuring Horizon for different environments, but since we're working locally, we stick to the default setup for now.

From there, we explore how queuing works with Laravel Excel. The best part? It already supports queuing out of the box! Instead of using store to save your export, you simply use the queue method, and the whole process works in the background. If you're working with large datasets, you can also customize how the data is chunked for more efficient processing.

We do a test run-exporting a small number of records first, then ramp things up by exporting a whopping 100,000 records to really show off the benefits of queuing. While that's chugging away, we also look at Laravel Excel's Exportable trait—a handy way to streamline your export logic further.

Finally, after the big export is done, we confirm everything worked smoothly and preview the massive exported file. There's a quick teaser about how we'll surface export progress to users in a future episode, but for now, you've seen how to offload big exports to the queue and monitor them in Horizon.

Episode discussion

No comments, yet. Be the first!