Matt Stauffer:
All right, welcome back everybody to the Laravel podcast season six. And we are back with another special at conference, you know, in person thing. And I've got some friends here together to record with me. I'm Matt Stauffer, your host. And can you guys introduce yourselves real quick?

Taylor Otwell:
Hey everybody, I'm Taylor Otwell, creator of Laravel.

Jess Archer:
Hi, I'm Jess Archer and I work at Laravel.

Matt Stauffer:
And we are in your city, like we're in your spot here. So what is your connection? Because we're at Laracon Australia right now. So tell us a little bit about your connection to Brisbane, where we are and everything.

Jess Archer:
Sure. So I've lived in Brisbane probably since I was maybe one or two years old. So my whole life, I live about 24 minutes, I think, up the road. So yeah, it's really, really exciting to have Laracon in Brisbane. I said at the start of my talk, but it's I've had so many of these surreal moments like seeing Taylor just like walking on streets that I've like been on all these times seeing like Marcel post photos on Twitter of like Woollah and Gabber and all these places I kind of regularly see. It's this like world's colliding thing. It's really surreal.

Matt Stauffer:
And when we, when we told people, Hey, we're going to be going to Brisbane, you know, we met multiple people in the US who were from Brisbane and they're like, don't worry about Brisbane. Go fly to Sydney and stuff like that. This is a gorgeous city. It is incredible. We had no idea. Yeah. So good on you, guess for planning in a great place. So yesterday you guys did an announcement about Laravel Nightwatch, which is a new product that's coming out soon and sort of like the second big announcement after Laravel Cloud. Everyone who has not seen it, probably want to go watch the announcement video, which we'll put in the show notes before you listen to this, because otherwise this Q &A won't make a ton of sense. But what we did is after the announcement video, as always, people have so many questions. So I asked people to pull together some of their biggest questions on Twitter and on Blue Sky and on our suggest.gg. And so we're going to walk through some of them and I'm just gonna ask the two of you and the two of you being Taylor, obviously founder, CEO, big man and you're also involved in all the products and then Jess as the product lead, is that what you would call her? The development lead for the Nightwatch project?

Jess Archer:
Yeah, I think Engineering Team Lead is the official title.

Matt Stauffer:
Yeah. OK. So both of you have been actively involved at different levels. And so we'll just have you guys pass the mic back and forth a little bit. So let's get started, because this is actually not my question, but it was something I'm most interested in, is how many months developers, time, basically, how long did it take to build Nightwatch? Because it really feels like we started seeing those little trickles of the villain emoji on Twitter, and then all of sudden it was here. So is this true that you all built this just faster than anything I've ever seen of this quality has ever been built? What's the timeline like?

Jess Archer:
So we have obviously history in this space with Laravel Pulse. So we've done a little bit of work in the observability space. I think it was December when, Taylor, you first mentioned to me the idea of a Pulse Pro sort of thing. What if we could go all out and use really specialized infrastructure?

Could like how could we go deeper with this? think that was like December when you sort of started planting the seed. But I was having a look at the code repositories and we had this kind of like playground repository where we did like just all these proof of concepts of just running queries with all these different database providers and the first commit to that was April 18th this year.

Matt Stauffer:
Oh my gosh.

Jess Archer:
Which kind of blows my mind and then the actual like repository like the dashboard that we demoed that repository was created on July 8th.

Matt Stauffer:
Of 2024?

Jess Archer:
Of 2024. That's when we like, like started work on the, like the real version of it.

Matt Stauffer:
Yeah. So that's four months.

Jess Archer:
Yeah. We like, you know, copied some queries over, but like the first dashboard was built in pure Blade with like database queries directly in views to just test stuff. So the real thing, yeah, we kind of effectively built from scratch from July.

Matt Stauffer:
That's amazing.

Jess Archer:
Which blows my mind. It's full credit to our amazing team.

Matt Stauffer:
And what is the general size of the team you're working with on this project? Is it the whole development team? Is it a subset of the team?

Jess Archer:
So it's mostly the Australia Pacific team, as we like to call ourselves. So mostly Australia, but we also have someone from Japan.

Matt Stauffer:
Has it been easier to kind of like do collaboration than it might be across the whole team because you all are relatively in the same time zones and everything?

Jess Archer:
Yeah, I think we tend to group most of our kind of projects by time zone. That's awesome. I didn't realize that. So yeah, Taylor's kind of the overseeing project manager or product manager of Nightwatch. Yeah, the Australia Pacific team has been building it out.

Taylor Otwell:
OK, so Taylor, I asked you this with Cloud, but I want to ask you this again on this one. It used to be you had an idea, you build the idea, and when you need, you kind of task out a piece of one of the projects out to somebody else. What is it like for you? And is it any different than it was with Cloud, where it's your brain, but I'm assuming you didn't write any or many at least lines of code on this?

Taylor Otwell:
Yeah, I haven't written any lines of code on Pulse or Nightwatch, actually. Both on Pulse and Nightwatch, I gave literally a few paragraphs, I think, of an idea, the kernel of an idea. It was like, hey, this might be a cool idea. If you all want to run with it and see what you can do, go for it. Let's just see what happens, basically. And people can see how Pulse turned out or you can go play with it on GitHub since it's available today. It turned out really cool, I think. And then basically the same thing with Nightwatch, very much just a kernel of an idea. Hey, what if we could just double click on all of this data and go a level deeper? Is that possible? What would that look like? And thankfully, all of the teams at Laravel actually are just really good at being self-directed and running with ideas. And when we started growing the team this year, especially with a lot of our sort of first round, maybe first and second round of hiring, you might say, we actually were pretty committed to trying to hire more on the senior side of the development spectrum. Because we knew as a remote team, spread out all around the world, it was even more important to have people that could just take kind of basic guidance and really run with it, you know, without having to be micromanaged or like trained. And I think as we continue to hire, of course, we'll hire across, you know, maybe more on the junior side of the spectrum once we get our feet under us. But we knew it was important while we were first spiking out these ideas to have really great people on these teams that primarily were senior. And I think that's what allowed us to have, or what allowed me to give them like a basic idea and just sort of like trust them to run with it and create something really awesome. And I think it turned out pretty amazing.

Matt Stauffer:
That's so cool. Somebody asked me yesterday, he said, I've heard before that like a senior at Titan might not be the same as a senior somewhere else. I said, is that a lot about technical ability? I said, some of it is, you have to be an expert the whole way through the stack in Laravel. But one of the big pieces about it is self-direction. Can you, on a small team of people, because we don't have 100 person teams where you're given a list of tasks and just do it. We have two person teams or maybe four person teams where it's like, I need to be able to trust that you get a little bit of direction and you can empathize and you can understand. And it takes it really far. And I feel like that's really something you focused on is not just technical seniority, but also independence and self-management and everything. Does that sound kind of online with it?

Taylor Otwell:
Yeah and just the ability to think through problems on your own, guess. And I guess a good example might be being able to sort of own the product a little bit. Maybe there might not be a linear ticket for each specific thing you need to do necessarily. For example, let's take Cloud, for example. If you delete a database in Cloud, the developers on the team are smart enough to think about, what about the backups that we might be storing for that database? Even if there's not a ticket created for delete the backups for deleted, you know what I mean? So like just the ability to like be a little bit self-directed and own your work, your area of the product, you know, I think that makes a pretty big difference in terms of like seniority.

Matt Stauffer:
Yeah. Well, in some of these answers, you guys have kind of made some references to Pulse and saying, well, you know, we can see that particular thing in Telescope or whatever else, but what does it look like to dive into it?
So that kind of leads me to something that you both have kind of made references to already, which is the question of like, where does Nightwatch sit together with the rest of kind of like the observability stack? So we've got Horizon, which is somewhat related, but especially Telescope and Pulse are in there. Do you think of it as connecting to or replacing, or are they all going to kind of exist? Like, how do they all fit together?

Taylor Otwell:
It definitely does not replace Horizon. So Horizon has a dashboard which might make people think like, this is sort of a observability tool. There's actually a lot more to Horizon than that. It's really more of like a queue load balancing tool.

Like it balances processes against your queues based on how much like wait time they have. So if it sees one queue is busy, it will allocate more processes to that queue and you know, maybe to another queue if another queue gets busy. So it's not just a dashboard, it's more of like a load balancer. So, Nightwatch does not do that, it does not load balance your queues, even though it has a similar screen to Horizon where you can see like your process jobs, your failed jobs and stuff. But it's purely just observing. Yeah, it's observing, it's not balancing. So you could totally still use Horizon with Nightwatch. I think that would make sense in some cases.

Telescope also appears somewhat similar to Nightwatch, like on first glance, but was always intended to be for local development only. It doesn't really surface the same kind of insights that Nightwatch might provide, and in some cases, it actually stores more data than Nightwatch. So like, Telescope stores your entire request body, all of the headers, you know, for every single request coming into your application. Because it's intended to be a local debugging tool, you want to dig into all that stuff.

I don't think super feasible or maybe not even desirable to store the incoming request body for every single request in Nightwatch. That's just not the kind of tool I think you would typically see in production. So yeah, and then another, ya know, building like a tool like a Nightwatch requires the use of tech that is just sort of like cumbersome or maybe not feasible to set up locally. Like it's not just a simple Laravel app. There's, you know,

Clickhouse database behind the scenes, there's Kafka behind the scenes, there's Lambda behind the scenes.

Matt Stauffer:
You're not setting up in your local.

Taylor Otwell:
Right, yeah.

Matt Stauffer:
Okay, so obviously this is a drastically simplified answer, but at least from a Telescope perspective, Horizon manages your queues, and yes, it has a dashboard, but it's a queue manager. Telescope is great for local, but no point to this entire process, as anyone ever said, you should put Telescope on production. And...

We asked a question. There's a question that's coming later, but let's just do it right now. I mean, if somebody wanted to use Nightwatch for local development, is that possible?

Taylor Otwell:
It is definitely possible. There are some screens that would not make sense to use locally because they're really geared for providing insights across like a sort of a multi-user application that's at scale or under traffic. Some of the screens might make sense locally or I could see why people might use them like the exception screen or the log screen.

But again, it's not, we did not build Nightwatch to be a local tool.

Matt Stauffer:
It's not the intent.

Taylor Otwell:
No, it's not the intent. It would be like, you know, people could do it, but it definitely was not the intent.

Matt Stauffer:
Okay. And the one we haven't covered yet is Pulse. And I think there's another question that kind of ties in there, which is some folks were asking about building custom dashboards for Nightwatch. And for those not familiar, Pulse definitely has a lot more of like the, yeah, we've got a couple of pre-baked kind of dashboard components, but it's really a, almost like a dashboard building metrics viewing kind of like toolkit, which I don't see Nightwatch as being. So can you talk a little bit about customization stories between Pulse and between Nightwatch and how you think they fit?

Jess Archer:
So with Pulse, we decided to use Livewire to have this kind of no-build customization step. Because we use Livewire, we knew that people could put anything they wanted on these dashboards. So it kind of was never a goal to make it do that, but we just kind of was like, well, it can do that, so this is pretty cool.

The main goal was to make it so that you could kind of choose which cards you wanted to prioritize. I kind of always had this idea that people might put it on a screen in their office if they're still working in an office to kind of get this live updates of what's going on. And maybe they want to put different cards up the top or make other ones bigger and all that sort of stuff. And we liked the idea that you could kind of collect or store different metrics in there. So maybe you wanted to create a card about sales or something along those lines. So it's kind of like, little bits in there that you can kind of hook into to make that possible.

Nightwatch on the other hand is really focused on performance monitoring, exception tracking and logging. We have some cool ideas around like custom measurements. So if you have something that is kind of outside of Laravel, so it's not like, know, mail or an outgoing request or a query, but it still takes some time, you could wrap it in a like a measurement kind of closure and you'll be able to see it in the timeline.

So we have some ideas around that. In terms of dashboard customization, at the moment, no, but I don't see any reason why at some point we might not, like we might have the ability to choose which cards are most important to you. Because different applications, like some applications might not have users, so like having users cards, maybe that doesn't make sense. Or they're not using queues, so they don't want to have that kind of taking up the priority on the top of the screen. So yeah, I think at some point we might be looking at something like that, but.

Matt Stauffer:
Probably not for ship.

Jess Archer:
Not for ship, no.

Matt Stauffer:
One of the things I thought was really cool from the demo was that you kept saying, well, if there's an exception, I think that the exception is more important than where it would be elsewhere. And so I'm going to put the exception up at the top for you because I think that you care. And it showed me that you all have an intentionality of the opinionatedness that you're bringing to this, of this is what you should care about, that a lot of us, when we're building applications, we don't feel like the privilege to do. It's sort of like, this has to be here all the time, and this has to be here all the time.

And we have to ask the question of like, well, this one's important sometimes, but not other times, but we couldn't possibly just move the thing. And it was just such a great thing. I'm just like, yeah, when there's an exception, I probably care about that more than anything else and I can drill into it or whatever. And so, but in order to do that, I feel like you have to have a whole opinionated mindset of what we should care about. And you said as Laravel developers who have run ops for smaller and larger projects before, these are the things we care about and we're bringing an opinion about what you should look at at what moment. And you can obviously you can kind of click through different things, but I feel like there's a level of preference and opinion and this is the way to do it that is opposite a lot of the questions we've been seeing around customization or using more kind of like, you know, people are talking about open telemetry and other things.

I know we don't talk about that much here, but it just seems like you all have an opinion of what a tool like this should look like. And you're willing to just say, this is what it's going to be like. Trust us. You're going to enjoy this experience. And I watched the videos like, yes, I will. Is that true? Do you all have been intentional about being that kind of very intentionally intentional?

Jess Archer:
Yeah, I think we kind of know that we want to... We have been very intentional. We knew we're collecting a lot of data. Our initial kind of proof of concept, we just put everything we had on every screen just to see like, you know, if we had all of this data, what would it look like? And then it was a case of like trying to pare it down and put the right information at the right time. So we worked very closely with a designer as well who could help us kind of break out of some of those like developer things that we naturally wanna do. So in my early version, every successful thing was green. And so the red and orange things, you could see them, but you had this of Christmas tree.

And one of the very first things the designer suggested was make the successful stuff, that's great. For this type of application, you wanna have the errors and the performance problems and have those stand out. And similar with exceptions, like there is an unhandled exception. It's probably the thing you want to come to. Whereas my developer brain is like, this is the page on, this is the place where they go on the page. And so like, it's, you know, it's this extra condition to like put it at the top like that. That feels weird. That's a different condition. It's a, another scenario. I've got to kind of test. There's more edge cases, but once we kind of like played with it and saw how that was working, I was like, this just feels right now. Like, this is what kind of takes software that's designed by primarily developers to having some more considered design and thought going into it. And I think you can see that throughout the whole product that there's been a lot of attention to detail and considering all of those.

Matt Stauffer:
Yes, it's very clear. And when a lot of people were saying it looks so good, think part of it was certainly that it looks beautiful and it's got great animation states and everything like that. But a lot of it was user experience. It was like, things are where they should be. And it's going to help me be better at thinking about APM or thinking about errors or thinking about logs because it's telling me what to look at.
And I think I love a tool that doesn't just give me what I want to ask for, but it tells me, here's what you should be caring about in the first place. And I become better at doing that thing because the tool is kind of like educating me as I go. Very, very excited about this. So let me real quick, we've kind of worked through. OK, so...

Speaking of how beautiful everything is, I want to talk really quickly about the tech stack. And everybody's asking usually two different aspects of it. One of them is what's the backend tech stack? How are you handling all this data and everything like that? And the next one is the frontend tech stack. It's so super responsive and the animations are so not what we commonly see and they're so smooth. And Sam Salikoff, a friend of mine who's just super into UI stuff, spent like, he sent like 30 messages talking about how certain graphs are animated and stuff like that. So can you talk a little bit about like what the frontend and backend tech stacks were for this?

Jess Archer:
Sure. So we, as I said, we built Pulse with Livewire, which really made sense for that product. But for the type of UI and all the data we had on the page for Nightwatch, we decided to go with inertia just so that we had full JavaScript client side abilities to kind of control how all that was going to work. so we picked inertia and then we decided to go with React, which is kind of a little bit of a deviation from like Laravel kind of stuff. I'd heard that Cloud was doing React and we'd just hired someone that was really keen on React. I didn't have a huge strong opinion. Whatever the team is happy with, whatever kind of makes sense, I'll figure it out. So so Inertia and React is the front end stack. We made pretty heavy use of like ShadZion components that are obviously very heavily customized from the defaults, but I think that was like a big kind of win of getting so much progress so quickly, is like all the little details like the tool tips and popovers and all that sort of stuff. Those work really well out of the box. They're very well considered in terms of like putting things in portals so you don't have to deal with like Z index issues and all those sorts of things. Having a designer on our team who's also a very talented front-end engineer made a lot of sense because we could come in as more backend focused and get all the data on the page and get it close enough to what was in Figma. And then he just comes through and does these little polish passes. And Figma always has the static design. So when he comes through, he doesn't just get it pixel perfect, but he also goes and adds these thoughtful animations and transitions and all these sorts of things using all sorts of techniques that I haven't been able to keep up to date with all the cool stuff you can do with CSS and JavaScript animation. So I'm really glad to have someone who
like that's basically his domain. Because I would love to spend the time focusing on like that design stuff, but then it means that it's harder to also focus on the more backend stuff. So segueing, ya know, gracefully to backend. The backend is obviously Laravel is powering the whole dashboard. Of course we're using a Postgres database for the kind of the application.

The main application things, and then we're using ClickHouse as the database for all of our analytical data, so the bulk amount of all the metrics and everything is in ClickHouse. ClickHouse is really interesting. So did a talk at Laracon US all about ClickHouse. It was kind of almost like a behind the scenes of Nightwatch before we released, like before we announced Nightwatch. We'll link in the show notes for anybody who wants to watch it. And in that talk, kind of talk about, like introduce the concept of analytical databases, because they were new to me.

I didn't even know words like OLAP and OLTP what are these? Once I learned OLAP, that I could Google this word and it would show me all the types of databases that are actually the ones that I need. That was really, really handy. So one of my goals with that talk is just let people know that OLAP, let them know that term and to search for it if they have this type of problem. It's not a silver bullet. That's why we use two different types of databases. Each have their strengths.

And then as Taylor mentioned, we've got in terms of getting the data in, there's lambdas and kafkas and all this of like this ingestion pipeline to bring data from all these different applications and get it into a ClickHouse database.

Matt Stauffer:
An incredible amount of data.

So a lot of, this is a tiny piece, but a lot of people have been asking questions about SQLite versus MySQL versus Postgres lately. Is there something particular that motivated you all to choose Postgres for this one for your primary non-massive data store?

Jess Archer:
I feel like I should have some really cool technical answer for this, but I've used MySQL and Postgres and SQLite pretty evenly over the years, I feel like. So I don't really have super strong opinions unless it's really use case specific. If there was some GIS stuff,I feel like PostGIS and Postgres would be the thing I'd choose. But the thing we're using a transactional database for in Nightwatch is pretty standard, and we could probably use any of the three with no issues.

Matt Stauffer:
Got it.

Jess Archer:
So we chose Postgres because we heard cloud was doing some cool stuff with Postgres. so we figured we probably want to put this on Cloud. So let's just build it with Postgres. Yeah, I mean, with Eloquent and Query Builder and all that, we can switch it and it wouldn't even make a difference to us because we're not doing the crazy stuff we do with databases is in ClickHouse rather than in a transactional database.

Matt Stauffer:
And that actually is a great transition to one of our smaller questions, is, is Nightwatch running on Cloud?

Jess Archer:
So not currently. It's mostly running locally. And I think we've got a demo environment on Forge at the moment, because that was just the quickest thing for us to spin up. But I think the plan is to definitely put the dashboard and everything on Cloud. It makes a ton of sense for us to do that. So yeah, now that we've got the demo out of the way, we can actually start going and doing that cool stuff.

Matt Stauffer:
All right. I'm going to give you a second to just breathe.

We just have a few questions left and unfortunately they're all kind of nitty-gritty questions so sorry that we're just kind of dive into these but the first one of our remaining questions is can we select a region for data to reside in for compliance reasons?

Taylor Otwell:
It's something we're definitely exploring. I don't know... I can't speak to what will definitely be available on launch, but we've heard enough feedback to know that this is something we need to be thinking about and exploring, so that's something we're gonna do. And that's kind of where we're at right now.

Matt Stauffer:
Okay, great. The next one is, were there any backend challenges related to scale that you encountered building Nightwatch that you hadn't seen before?

Jess Archer:
So, it's kind of like related to this. When Jack Ellis brought out like further analytics and all those sorts of things, and I just remember thinking like, that seems like the hardest problem in the world because every request that his customers get, he gets a request. So one of his customers gets like denial of service attacked. That's like, and that's just a drip in the bucket of all the other customers. So like the idea of like this like mass kind of collection of data, I always found terrifying. I'm like, that sounds terrible. Never want to do that.

So then we built Pulse and I kind of got a little bit more comfortable with the idea of that. But with using MySQL and Postgres for Pulse, we knew there were limitations with data. So the very first step we did was all this proof of concept work. Tim and I ended up reading academic papers on these aggregation algorithms and to do these kind of rolling aggregations over time. We were just trying to figure out how we were going to deal with all of this. And that's when we eventually stumbled upon analytical databases and column store. And we played with so many different ones, but we ended up settling on ClickHouse. It's open source. It's amazing. The team there are really great and really helpful. It kind of feels like they're almost like the Laravel of databases. It's kind of a bit funny to say, but they have amazing documentation. They've got a really cool community.

So yeah, I kind of felt like similar vibes, whereas some of the bigger players, they kind of felt a bit more kind of corporate.

Matt Stauffer:
OK. Well, and Taylor, you've mentioned a couple of times that this new world of Laravel is allowing you to build things that you couldn't build before. And when I'm hearing Jess talk about, Jess and Tim are brilliant human beings, right? So those two people just diving for weeks to months into academic papers just for one piece of one product, I feel like that's something that would have been really hard for you to just justify the time on.

Is this, am I reading it right? This is part of the story of this larger team is allowing you to do things that you never could have done before.

Taylor Otwell:
Yeah, and it's the larger team. It's the assistance from Accel financially to allow us to tackle riskier problems that require more staff. That, you know, maybe we wouldn't have taken that risk, you know, before her, at least I wouldn't have if it was just me by myself bootstrapping the company.

So you know it's a combination of those factors. I think this letting us take bigger swings at bigger problems that are a little bit riskier and more involved than things we've done before but also much more beneficial to the end users because the products are you know awesome, know things like Cloud and Nightwatch are just way better than anything we built before in terms of user experience and polish and What they'll offer to customers. So yeah, you know, it's exciting.

Matt Stauffer:
Yeah, it is very exciting and I told you all this one-on-one, but Keith Damiani had come to me a year ago or something and said, we should build an APM or some kind of thing. And I looked in into it I just said, that is so much work. I can't figure out what it would look like for us to dedicate a team, take them off client work for so long to do that at the level of quality that I want us to put out at Titan. I didn't totally write it off, but I kind of just said, it's a brilliant idea and I just don't know how to do it. And so when I heard that y'all were doing it, was like, first of all, great, do a great job. But still, I had no idea the level of depth and excellence and quality you were gonna be able to pull off in such a short time. And I am just mind blown because first of all, I would have been mind blown in the first place, but second of all because we did enough research back then, I just got an initial sense of just how ambitious and how big of a load this is for this. So I just wanna say to both of y'all, this is absolutely incredible work overall, let alone in four freaking months. This is amazing, so.

If anybody needed affirmation that this new kind of direction of Laravel is producing something we couldn't have before, I feel like this is it. Like we're there.

Taylor Otwell:
Yep. Yeah, we would definitely not have built this with our small nine, 10 person team earlier this year.

Matt Stauffer:
Okay, well we have two remaining questions and then I'm let you all go. Second to last is, will Nightwatch have any integrations with GitHub issues or PRs or any other things where basically we are able to set up workflows with our exception tracking? Because you all mentioned kind of this is exception tracking, this is APM, this is log. So from the exception tracking point, are we gonna have integrations so that maybe Nightwatch auto opens issues or auto closes issues in response to certain statuses or whatever in various apps? Is that something planned for the future?

Jess Archer:
Definitely, we've done a lot of thought there. The thing with Nightwatch is that there's so many areas we could go like really, really deep on and I mean, we had to choose a few areas to go deeper on and then know that later on we can come back and revisit things. So I feel like we went pretty deep into the APM side of things. But we wanted to have this story where everything ties through, having exceptions and logging in there. But I can see for sure we're going to come back to exceptions and flesh that out a little bit more and have ways to create issues, whether in Nightwatch or in external tools. And we also have some other, yeah, just all sorts of interesting ideas.

Some that you would expect, but also some that you might not expect. So yeah.

Matt Stauffer:
OK. And in my talk for anybody who didn't see it, I said every single new Laravel app, no matter what it is, should have bug tracking turned on. And it'd be really nice to have logs. And I didn't even get to APM, because the number of times we've tried various APM tools and said, no, it's just not worth it for 99 out of 100 projects. So in my talk, said, you should have bug tracking, because it's pretty well-established problem. It would be really nice to have logs, but we just kind of like don't have an answer there and I didn't even touch an APM. So the idea that we are going to have a like first party best practices tool that brings APM and logging to become just like this very easy thing. You just plug it in and turn it on and you get those. Having tried to do that for over a decade in Laravel apps and never finding a good solution, I'm extremely, extremely excited about that. A, and then B, the fact that it's gonna tie together with the exceptions and Jess and I talked about this a little bit about that even if you can get all your exceptions in one tool and all your logs in one tool, being able to correlate that this particular log that happened at the same time is also connected to that request when there might be a delay or maybe there's queued items, tracking the different pieces together when they're not in the same system is nearly impossible. And so having one system that has all the information together in one place but also understands as developers, we want to track it through the request or through the user or through the job or whatever.

There's places where there's no direct corollaries unless you have specifically built the tool to correlate them. And so just seeing in this initial demo, I'm able to say this request from this user kicked off this job that then sent off this mail and they're all just kind of traced together. I'm just, I'm geeking out right now. Like I cannot wait to try this out.

Jess Archer:
That was one of the first things we really kind of focused on was how the data would all be connected together. So these like concepts of trace IDs that would be attached to everything. So in our very, very earliest prototypes, you could kind of just follow this tree, click things from any angle, and you'd always be able to see everything in context no matter what angle you were attacking it from, whether you attacking it from a customer support, I have a user that's got a problem, or I know there's a problem, or maybe I don't know there's a problem and I want to find out if there are problems, and whether I want to drill in at the request perspective, or the jobs, or drill down into queries.

All the different angles, no matter which way you go in, you can still find the context of the other things that are related to it. And yeah, from there, we kind of then started really fleshing out the APM side of things. But yeah, that kind of interlinking of everything was like day one kind of in there.

Matt Stauffer:
Did you have to make any changes to the framework to support that?

Jess Archer:
No. we have, there's some things we'd like to improve for like how we can capture certain bits of data, but we figured out ways that we can still achieve what we need to, because we want to try and support, I'm not going to commit to any version numbers here, but we do want to try and allow people to install this on maybe a couple of versions back. But there's definitely some things that we know we can do better if we change some small things, like even small examples of things that if we stored the high resolution time instead of micro time, I don't know what's going to be involved if that's even feasible to change. Taylor's like, probably like what you're talking about. But yeah, because for our queries, pretty much every metric we capture down to the microsecond rather than the millisecond because a lot of database queries are sub-millisecond. So if we only captured millisecond resolution, it would be hard to build that timeline with any accuracy because it might look like two things are happening at the same time.

Matt Stauffer:
When we know they're not.

Jess Archer:
So yeah, everything goes down to microsecond precision. But there's cases where having the high resolution time might make more sense. So yeah.

Matt Stauffer:
OK. Well, then it's actually a great segue to our last question, which is we're tracking all these tiny little things down to microsecond, millisecond precision, but we're adding a piece of software that wasn't running beforehand to our applications. Is there going to be a performance cost of adding Nightwatch to one of our tools where it wasn't there before?

Jess Archer:
I mean, with any monitoring and observability tool, there has to be a small bit of overhead. It can't observe it without hooking in in some form. The integration points are very, very lightweight. And we've designed it so that you'll run a local agent on the service that hosts your application. Potentially for things like serverless, there'll be some slightly different scenarios, but it's all going to kind of, yeah, we'll figure out how all that sort of stuff works. But the main thing is that during the request lifecycle, when we're collecting these metrics, at the end of the request, after the response has been sent, that metrics payload can be sent to the local agent over a local TCP socket.

So it doesn't have to make any outgoing database requests, any outgoing HTTP requests. It all just gets sent to like within the same machine, effectively like an internal local host. And then that agent can then batch them up and send them off.

Matt Stauffer:
Ina way that doesn't touch the requests at all.

Jess Archer:
Yeah. So there's obviously like, you know, fractions of fractions of fractions of seconds to listen to an event and run some code that captures some, you know, read some properties off that event and puts it somewhere.

Matt Stauffer:
But nothing worse than any existing exception tracking service or anything like that.

Jess Archer:
Exactly, yeah. If anything, probably more performance.

Matt Stauffer:
Because it's not doing its own HTTP calls, which most of them have to do because they don't have their own agents.

Jess Archer:
Yes.

Matt Stauffer:
Okay. Very cool. Well then my actual last question is, is there anything that we didn't cover here that you felt like you didn't have a chance to say or Taylor, if you've been asked any questions on Twitter, or you as well, Jess, is there anything that's come up that you say, you know, I'd really like everybody to hear this? Do you feel like we got everything covered today?

Jess Archer:
No, I mean, there's definitely, I think Taylor said this kind of in his talk that we're at the moment kind of laying the foundations of what Nightwatch is. We have so many ideas of how we could make this thing even better. And it's been really like almost frustrating in some ways to like not be able to go and explore some of these further away ideas. So yeah, we've got a lot of stuff that we would like to explore. And we think this is going to just become like, if you're amazed by it now, wait till you see it in the future.

Matt Stauffer:
You have seen how excited I am right now. So you were like, my god, okay, cool. Very cool, very excited. Thank you both so much for taking some time away. I know there's a lot of stuff going on at the conference right now and thank you for all your work. I'm on the list, I'm on the signup list. The moment I saw there was a signup list, I got in as fast as I could. So I'm hoping I'm gonna get that early access. And yeah, thank you guys for your time.

Jess Archer:
Thank you, Matt.

Taylor Otwell:
Thanks, see you everybody.

Matt Stauffer:
See y'all later.