Separate database for Laravel queues
Playing
01. Separate database for Laravel queues

Episodes

0%
Your progress
  • Total: 7m
  • Played: 0m
  • Remaining: 7m
Join or sign in to track your progress

Transcript

00:00
In Laravel, we have a really good selection of queue drivers based on our needs but I
00:04
always find myself going back to just using the database for queuing. But what I like to do is separate out my queue database from my normal database. Of course, the disadvantage of using a database for queues is that, you know, you're writing to your database at the same time and it can be a little bit slower.
00:23
But if you use a separate database, not only is it easier to manage, you have, you know, nice and tidy tucked away somewhere, you'll also probably see some kind of speed increase as well. It's not a massive deal. It's usually, in this case, just to kind of tidy it away from your standard database schema. So what we're going to be doing in this snippet
00:42
is just looking at moving our queue table and our failed queue table away from our main database. So to start with, I do have a database just here and I just have a fresh Laravel project installed. That is pretty much it at the moment. So the first thing that I'm going to do is just switch over my default settings here. It doesn't matter if you're using MySQL. This is going to
01:05
work for you in exactly the same way. And we'll just immediately come over and migrate these changes. So let's do that now. Okay, great. So they should be in there and we have our standard table. We're just doing whatever we want. Now in terms of a separate database connection for our queue, let's first of all head over to our config database section. So we know here that we
01:30
have and we're using a Postgres driver just here. Now if you're using MySQL, this is the one that you want to duplicate. If you're using Postgres, this is the one you want to duplicate. It really depends on what you're doing. Now what I'm going to do is duplicate this down and I'm going to go ahead and switch this over to a kind of queue database connection if that makes sense. So in
01:53
this case I just call this queue but you can call it whatever you want. So what I'm going to do here is switch over all of these config details to prepend queue. So we're going to have the queue database port, queue database database, queue database username, queue database password, and I'm also going to set the connection as well. So qdb connection and by default I'm going to set
02:16
that to Postgres just in case in production we need to switch over the database that we're using for this. So now what we want to do is just hop over to our env file and basically go ahead and just duplicate these down. So we're just going to before each line here prepend queue. So we have a completely separate queue database. So what this is going to be in my case is the same connection
02:43
but I'm just going to have fresher queue. So a completely separate database just for queues. So the first thing I want to do is go ahead and create this. So if I just come back here and create a database called fresher queue like so. Now what we want to do just before we go on is just come over to config queue where we were before. Now down here it will give you the database
03:05
option. It will choose which driver but we need to specify which connection because by default the connection that it uses is going to be the connection that you are using for your normal database. So in our case we know that the connection now is called queue because that's what we named it just here and that is pretty much it. So you can configure anything else you
03:25
want in here as well. Entirely up to you. So now that we have done this we can go and generate our queue table and then really importantly we need to change this over. So let's go and say php artisan queue table and what we'll also do while we're here once that is finished is generate our failed table as well. Always useful to have your failed table in there and if we come over
03:51
here to our database migrations these are the migrations that we need to change over. Now in this case if we would just migrate these as normal so if we just come over here and say php artisan migrate of course what's going to happen is that's not going to come into our queue table it's going to go into our normal table here. So let's go ahead and roll back these changes
04:15
and we will come over to our migration just before using the schema builder we create this we can define out the connection that we want to use. We want to use the queue connection. Really importantly don't forget to do this for your down migration just in case you want to roll this back. Of course if you do an up with connection on queue and a down with a normal
04:36
connection it's not going to be able to find that table. So we can do the same thing for our failed job so let's place this in here and we will do the same thing of course for dropping our table on our down migration. So let's go and migrate these changes and that will now have placed these not into here over in our queue just here. So now that we have created that separate database we've
04:59
switched over our configuration so we can actually connect to it and we've changed over really importantly over in our queue config the connection that this now uses we are pretty much done. So what we can do is just test this out by maybe just creating a job dispatching it and see that the data ends up in that table. Before we go anywhere though of course one last thing to do
05:21
we want to make sure that our queue driver is actually set to database because we're going to be testing this out in just a second with a job and then dispatching. So as long as we have our queue driver set to database we can create a job dispatching it should end up in that table. So let's go and make a job in here and we'll just call this test job that makes sense and let's come
05:42
over to app and jobs open this up make sure this implement should queue and inside of here why don't we just log something and we'll see that in our Laravel log so job worked. Let's come over to our roots and we'll just dispatch this from here just so we don't have to spend too much time testing this so a new test job and let's just pull the namespace in for this let's come over here give
06:05
that a refresh and what we should end up with is that job in that new table. So if we come over to the command line and run php artisan queue listen we should see that processed and if we just come over to our Laravel log we see job worked. Now let's make it fail just to make sure that we're putting things inside that failed table so in here maybe we could just get rid of that so that
06:30
namespace can't be found let's come over and give that a refresh over to our table this is that job in here we want to go ahead and wait for this and in this case we've got too many attempts so let's cancel this off and say tries equals five that makes sense so now that that job has failed we should see this inside of our failed jobs table and we actually don't now the reason for this if
06:54
we just come over to config queue we actually have a separate section down here for our failed config now we know that the connection that we want to use for this is queue which is that one we set up earlier and we know that the table is called failed jobs because we just left the migration as it was before so now let's just come over to here just make sure we don't have any jobs in here
07:15
let's go ahead and run this again so we see that job in there and let's go ahead and make sure our queue worker fails that over to here we now see that inside of failed jobs so there we go if you do use database queues and you want to separate these out into a separate table a separate database in fact then that is exactly how you do it

Episode summary

In this episode, we walk through how to set up Laravel queues to use a separate database from your application's main database. If you've used the database driver for Laravel queues, you probably know it drops queuing tables right in with everything else, which isn't always ideal, especially if you want to keep things tidy or improve performance a bit.

You'll see step-by-step how to duplicate your database configuration, set up new environment variables for your queue database, and update your config/queue.php so that job and failed job tables end up in a totally separate database. There's a bit of migration tweaking as well — I'll show you the line you need to specify which connection your tables belong on (so migrations end up in the right place).

We create a test job together, run it, and check that both queued and failed jobs land in the new database. Plus, we'll briefly troubleshoot a common pitfall with failed jobs not being logged in the right place (hint: it's another config setting you might miss!). By the end, you'll have Laravel's database queues housecleaned into their own space — perfect if you're looking to keep your app database drama-free!

Episode discussion

No comments, yet. Be the first!