The Pragmatic Engineer
00:49:30 · Gergely Orosz, Federico Simionato
[00:49:30] S0: And and what are some of the the products things that you're looking forward to? [00:49:35] S2: Yeah. We recently released what I believe, from from my tests at least, is the best AI transcription feature available on any note taking product. It basically allows you to upload a video, audio, or image file and hit the button and, like, basically one click transcribe it, So you get some... A bunch of text as output. It's pretty fast, super accurate. Like, I was honestly surprised myself by by how accurate it is. And one thing that that is coming soon is that we are trying to build this into a standalone tool, so outside of Evernote, which is kind of unusual, I guess, but but we are so proud of it that we want to use it as a as a as a tool to show people that Evernote is cutting edge again in a sense. So that's one thing that that we're doing. [00:50:30] S0: Okay. Thank you. And switching to a different topic, although somewhat related. So we talked about how you built Evernote, how you acquired it, how how you how you've balanced architecture and and features. But within Bending Spoons, how do you typically get projects done? And, you know, like, what I'm interested in is how typical is Evernote versus not. And maybe we could talk about an example of of a of a project that you got done recently inside the company. And do you even have a typical kind of almost like template or runbook or, like, here's for us what a well executed project looks like?
00:51:04 · Luca Ferrari
[00:51:04] S1: Well, I guess it helps to describe how we're organized. So we have different business units, Evernote being one. And these business units have dedicated management teams and resources, software engineers, data analysts, scientists, product designers, so on and so forth, depending on needs specific of that product and business. And we encourage them to operate largely autonomously. And then we have a shared platform, including several platform teams. And what the platform does is it supports and serves the different business units with tools, services that are relevant, generally speaking, to most of them. Some of this is as basic as accounting, finance, and forth. And some of it is as advanced and technically exciting as all sorts of data storage and processing tools and layers, machine learning models to forecast a particular user cohort behavior and value, AB testing systems, automations for marketing. We spend tens of millions of dollars in marketing each year, and it's almost entirely run by machines. So within this framework, the principle we try to follow is to create small teams of exceptional talent density, and give them a very vast mandate in what to do. So we trust people to set intelligent objectives and try their very best to achieve them. And this really is not limited to engineering. It's a cultural trait at the company. And so it works at every scale you look at. For example, if you look at Evernote as a unit, as a business unit, the whole team knows they don't have to follow any mandated direction, they can make their own decisions on their mistakes, on their failures. But also in Evernote, the smaller squads taking care of the various missions or objectives also will have a broad mandate on how to pursue them. We will not impose any particular technological stack on them. We will not impose any particular type of process on them. There are guidelines, there are lessons learned. And people at the platform level, our experts at the platform level, always available to coach, help, support. Ultimately, we trust people to to take full responsibility for for their actions.
00:54:04 · Gergely Orosz
[00:54:04] S0: Awesome. And in terms of engineering approaches, which ones do you use? I mean, we mentioned CICD, and we mentioned that you're aiming to not have on call. But for example, what's your approach to releasing QA testing these areas?
00:54:20 · Francesco Mancone
[00:54:20] S3: I'd say, following the lines of what Luca already said, it's important for us to not be prescriptive in the tools and the methods that we want to follow across the different business units or across the different teams. Tools are tools that are meant to help to achieve a certain objective. Processes have the same objective, let's say. Some teams are more used to follow certain processes than others. We give the team the freedom to choose whatever they believe is the most effective approach. We give them principles. We help them shape what is the responsibility of the team. And then, of course, we work together to understand what technology, tool or process can be the most suitable one to work effectively. Generally speaking, let's say for QA, for instance, we try to give that responsibility, to share that responsibility among all the different software engineers and also, of course, QA testers. But in Bending Spoons, I'd say software engineers really care about the product and the end result of it. So doing white boxing QA is something that they take very seriously. They dedicate a lot of time. Of course, it's also about building the right developer experience. Otherwise, it becomes very cumbersome. We know very well. But on top of that, then, of course, black box QA instead is done by the QA testers. We make sure that everything that can be tested by software engineers before is properly tested And we found that approach to give a lot of, let's say, to make the engineers more responsible for the end result and in turn more productive and also happy about what's happened next because more rare to ship bugs into production if you built the right developer experience and also gave shared responsibility to everyone about testing their code base.
00:56:40 · Gergely Orosz, Francesco Mancone
[00:56:40] S0: And how big are you on automated testing in terms of like, you unit integration end to end, some of these things? You didn't mention manual testing, which obviously is important, but like, or is it more per app, per product, whatever makes sense for the maturity of the product? [00:56:56] S3: Yeah, that's the point. I think it depends a lot on the maturity on the product and also what's the real estate, for instance, for that product. If the core part is about the UI or if it's about some critical features. I'd say we develop a lot of end to end tests for sure. We try to reach the right level of coverage. We try also to be pragmatic, meaning understanding what are the use case worth building the right set of tests that can be generalized enough to be effective more than manual testing, for instance, or to be sure that those tests can be general enough to capture behaviors and not generate false positives. I believe also in terms of observability and monitoring, in addition to QA, something that we have spent a lot of effort is to be sure to tailor metrics or what we test and how in such a way that we're sure that we have the highest coverage, but we're also very careful to not trigger false positives. We have seen multiple times in the past, we improved a lot in time, of course, but that has been some of the things that we've seen improving a lot the life of engineers after, finding themselves fighting with false positive voltage all the day.
00:56:56 · Francesco Mancone, Gergely Orosz
[00:56:56] S3: Yeah, that's the point. I think it depends a lot on the maturity on the product and also what's the real estate, for instance, for that product. If the core part is about the UI or if it's about some critical features. I'd say we develop a lot of end to end tests for sure. We try to reach the right level of coverage. We try also to be pragmatic, meaning understanding what are the use case worth building the right set of tests that can be generalized enough to be effective more than manual testing, for instance, or to be sure that those tests can be general enough to capture behaviors and not generate false positives. I believe also in terms of observability and monitoring, in addition to QA, something that we have spent a lot of effort is to be sure to tailor metrics or what we test and how in such a way that we're sure that we have the highest coverage, but we're also very careful to not trigger false positives. We have seen multiple times in the past, we improved a lot in time, of course, but that has been some of the things that we've seen improving a lot the life of engineers after, finding themselves fighting with false positive voltage all the day. [00:58:34] S0: Am I guessing correctly? Because you mentioned you have about 100 products and you mentioned you have internal teams for some of these shared things. You must have platform teams, right? Yes. And what kind of platform teams do you have? What do those platform engineering teams that is?