Show Notes
In this episode, the hosts discuss the Capability Maturity Model (CMM) and its implications for software development organizations. They explore why CMM was chosen for this episode and its connection to previous topics such as the Team Software Process. The conversation delves into the maturity levels defined by CMMI, from ‘incomplete’ to ‘optimizing,’ and explores whether the lack of a ‘why’ behind processes affects the model’s utility. The discussion detours into how modern tools like Large Language Models (LLMs) and Copilot can impact software development, highlighting both their benefits and limitations. It ends with reflections on continuous improvement and how organizations can leverage CMM and LLMs for better outcomes.
References
- Capability Maturity Model for Software, Version 1.1
- Capability Maturity Model® Integration (CMMI), Version 1.1
- The Capability Im-Maturity Model (CIMM)
Transcript
Mon-Chaio: Hello everyone. After a couple weeks where Andy and I have been on different schedules, and hence you had vacation casts from one of the each of us, we are back together and discussing today the CMM or capability maturity model, I believe is what that stands for. And the idea for this episode came out of.
Our previous episodes talking about certain models to help you think about the world and to use in diagnosis, and I can’t [00:01:00] remember Andy, but why did we land on CMM? Was the last model we talked about derived from CMM? Is that what, uh, what led us down this path?
Andy: Not quite derived from it, but related to it, we were talking about the, uh, team software process, the TSP and the TSP is part of the. Suite of things from the SEI, which is from CMU. I’m gonna see how many acronyms I can get in
Mon-Chaio: Uh huh. Keep going.
Andy: Um, and the S-E-I-C-M-M or CMMI as it is now, uh, which used to be the S-W-C-M-M.
Mon-Chaio: You’re, you’re getting, you’re doing well.
Andy: yeah. Thank you. It’s a suite of processes, uh, that go together with the PSP, the TSP and the CMM about how Watts Humphrey thought that software development should be done, and the CMM came out of, uh, working with the [00:02:00] DOD there. I got
Mon-Chaio: Mm-hmm.
Andy: Uh, to, to give them guidance about what good organizations would look like.
Mon-Chaio: Right.
Andy: And specifically we got to this because we were looking at the TSP and TSP was explicitly at the team level. And we found a, a thing describing that the CMM was at the organizational level. TSP was the team level.
And so we were a little underwhelmed by TSP. We kind of liked it for its measures. We liked it for its, um. For its, uh, symptoms that you might find in the measures that they had, but it didn’t seem to have any system. It, it, it, it had its process, but it didn’t have a system that explained, well, if you changed this, what would you expect to
Mon-Chaio: Mm-hmm.
Andy: And we thought, well, CMM being at the more organizational level, maybe it will have a system.
Mon-Chaio: Right. Okay. That’s exactly how we arrived here. Um, nice use of acronyms, by the way. Uh, I think the last one, [00:03:00] DOD is the one that. Informs us about why there are so many acronyms above it. uh, CMU is Carnegie Mellon University. That’s where this model was developed, and I believe the first version was published in 1991, uh, so quite a while ago.
Um, and it lives on, I think, um, and so therefore I think we can take away that there may be some value to this. Um, and as you mentioned, Andy, it was originally written for the DOD to. Assess maturity models and actually to guide them, I believe in selecting vendors to say, Hey, if I’m looking for a vendor, a government vendor, can I fit them into this model and decide where they are in the maturity chain and use that as a differentiating basis for selection of vendors.
Um,
Andy: use it as a way of helping vendors, uh, be better, I think is another idea behind it.
Mon-Chaio: Right. And actually that’s the next thing, which is when we think about the CMM today, [00:04:00] obviously it’s gone beyond, um, or to your point, the evolution of CMMI. It’s gone beyond just for the DOD in selecting vendors. It’s used by consultants and other people to, um, diagnose, right? That’s one of the key words for this season, to diagnose effectiveness of software organizations.
So. CMM and CMMI are really big. But maybe the first thing we can do is just walk through the high level, their proposal of the levels of maturity within an organization. Does that sound like probably the right place to start?
Andy: Yeah, and I think it will give everyone, uh, some grounding, something to hold onto for when we’re talking about what this is and our assessment of it, why we might have that.
So just going through it really quickly, it’s got. Depending on which version you look at. The one I’m looking at right now is the V 1.1 of [00:05:00] CMMI.
What they have is six levels. And historically it had been five levels.
Uh, and what they have is the zeroth level. Like any good asimov of people, you always have to start from the zeroth law, uh, and that’s incomplete. What they would call that is that you’ve got basically no process. Then their first level of CMMI is performed. You have a process, you perform it. The, the second level is managed.
You manage what it is you’re doing. Uh, the third is defined, which is that you actually know what you’re doing. Sounds kind of strange that you’d be going in that order, but, I think what they’re getting at is that you get more and more attuned to what it is you’re actually doing as you go up in the levels. Then. Level four is they are quantitatively managed. So this is where they start spending a lot of time looking at metrics, looking at measures, knowing [00:06:00] exactly the number of bugs going out, and the number of lines of code and the effort that went in, and all of those things. From the TSP discussion, I bet a lot of that is geared towards really getting to that quantitatively measure managed level.
And level five is optimizing you are paying attention and you are improving as a matter of course. I think the idea is that up until that point, most likely optimizing or improving is a thing that you do in special circumstances,
Mon-Chaio: Right.
Andy: whereas at level five, improving is a thing that you do just as a thing that you do.
Mon-Chaio: And you mentioned that this is the version 1.1 of CMMI about a decade in from the original CMM, the original levels from the CMM. they had five levels, as you mentioned. They were missing the incomplete, um, and they aligned pretty well with the CMMI levels, right?
I think, um, number five was optimization. Um, that [00:07:00] was very clear, uh, for both levels. I did like that they called level two repeatable, though, because it made more sense in my mind than to call it managed.
Andy: Yeah.
Mon-Chaio: I think look like now we’re getting a little bit all over the place. Why don’t we, why don’t we, um, align them a little bit?
So there’s, uh, number zero, which we called incomplete. Um, number one. Um, the original called initial, um, and the new model called performed. Uh, essentially the same thing. Uh, you are doing a process versus not doing a process. I think I like performed a lot better ’cause initial means nothing, right? So, um.
Number two, the original called Repeatable, the new model called managed. I like the repeatable nomenclature because I think the idea is that you have something and you can repeat it for multiple projects.
Andy: Yeah. Yeah, I think that makes a bit more sense to me as well.
Mon-Chaio: Uh, and I think the idea is [00:08:00] you don’t really know how it works or how it’s functioning, but you know, it delivers something. At various stages of the process. And so, uh, and then you can use that to say, okay, well we need to run these five things and um, think one will deliver one thing and then think two will consume that.
And eventually you’ll get some software at the end. Right. Um, and then level three for each is defined, which, um, is fine. And as you were mentioning, this is around the concept that you can now know kind of what’s going on in thing one a little bit instead of input into thing one. And there’s an output, you’re like, oh, input into thing one.
And the output is because there’s some thing happening within thing one, and I can look into it and say, oh, well that thing is or is not happening, right? Um, level four. Quantitatively managed versus managed. Um, I like the fact that quantitatively is in the new models, uh, level description because that is, as you were mentioning, what this level is all [00:09:00] about.
The difference between this level and the previous level is. You can see into the box, but not only can you see into it, now you’re actually measuring what’s going on in the box, right? You’re not just saying, is this thing happening? You’re not happening. You’re saying, well, is it happening? Well, is it going poorly?
Is it trending poorly? Um, so I like that quantitative in there. And then number five, optimizing. Um, you set it, well, just continuous optimization versus one-off.
Andy: Yeah. And, and for all of this, the, the point of these levels in some ways is for a person to know whether or not you should be able to deliver a particular software project. If it’s gonna be a highly complicated thing like the DOD might ask someone to do. They’re, they’re looking for places that are probably a level four or higher, um, maybe level three.
If you’re just making a web app, you can probably get away with, uh, between levels one and three. And, [00:10:00] and they even say that. They’re like, look, not everyone needs to be level five. So I guess mcha, the question is, does this help us? Does this whole model help us in this diagnosis sphere? Does it add anything that we didn’t already have?
Mon-Chaio: It is interesting that you mentioned diagnosis severe. Um, I was gonna answer differently, but obviously we should be thinking about it that way. Andy, I mentioned at the beginning, that’s kind of the theme word for this season. I. I think in some ways it does and in some ways it doesn’t. Um, I think if I’m reading some of these articles correctly, they themselves will say that this is kind of a way of figuring out what the maturity levels are, but the details within it, like the individual processes or why the individual processes work are kind of left.
I feel like as an exercise to the reader, right? It’s a, uh, it’s a [00:11:00] framework to say, do you align? But if you don’t align, there’s no way to figure out, well, where do you go and why?
Andy: Yes. I, I, I agree with you. So my reading of it as I was going through it, ’cause I was reading and I was just like. I, I felt empty as I was reading through, and I try, I I, I scanned a lot of this 703 page BDF that I have. I didn’t read closely every part of it, but as I was reading it, I came up with CMM doesn’t have for, for all of its thing about quantitatively measured.
I didn’t find much about like quantitative measures that they prescribe that you should do. There’s probably a few that I might have skipped, but, but it, it wasn’t heavy on it, not like TSP. So it doesn’t give us much on that side of the, diagnosis spheres, those four spheres of knowledge that we talked about, it does give you [00:12:00] kind of a little bit on the symptoms side.
Along with CMM, there’s an entire diagnostic thing, a kind of like a survey you can do to figure out where people stand. So it does give kind of, not, not quite the measures that we were thinking of. So I guess it is a way a measure this survey, but it’s, it’s symptoms. They would say, oh, well if you find this happening, you should do this.
If you find this happening, you should do that. Which then gives The other thing that I found a lot in it. Is that as you go through the levels, it becomes almost a catalog of activities that you should do.
So it gives you a lot of these solutions. It gives you, you can do this and you can do that, and you can do this other thing. But I didn’t get much out of it In terms of the why, like the thing when we were talking about TSP Mancha that I remember you saying like, I don’t, I don’t know. It doesn’t, it doesn’t give me any why I should do anything.
It doesn’t gimme any why [00:13:00] 1.0 is the right measure in that
Mon-Chaio: Mm-hmm.
Andy: And I felt like CMM was a similar thing. It gave me a lot about symptoms I can look for. It gave me a lot about, uh, solutions that I can apply. It gave me a lot about, well, slightly less than TSP about, um, standards that I can do, I guess standards in more qualitative sense.
Absolutely. Are there in what they consider, you’ve got that five levels and they have what you should find at each level, but the why you do any of these things or why, uh. What will happen if I choose this particular practice? Because the whole way that you move up through these levels, which is what they consider, mostly you should be doing, is you adopt new practices.
But I would want to adopt a practice based on I saw something happening and I have a reasonable guess or a reasonable understanding that rationally this is happening because of these uh, reasons. And if I take on this practice. Those reasons will be [00:14:00] altered in this way, which will cause that outcome to be different.
Mon-Chaio: Mm-hmm.
Andy: And they don’t, they don’t give me much of a model for that.
Mon-Chaio: That’s exactly right. I mean, they’re missing that top s that system about how the system relates to each other, such that when you change one thing, this means that it changes the other thing and affects it in this way. And I think a lot of these, I guess I would say older models to think about software development.
Um. It feels to me tend to be like that. It’s more, if you do these six things, you will be good as if every software organization in the world produces the same software. And so you all operate in the same way. And I wonder if that’s because, and I don’t know this to be true, but there’s still this.
Prevailing, um, thought process as well as terminology, comparing building [00:15:00] software to things like building houses or vehicles or whatever. Those have very defined processes, which you just cookie cutter one to the next, right? It’s not like one general contractor builds houses differently from a different general contractor, and that’s just not the state of software today is it?
Andy: Maybe that’s what they’re saying is that we should be much more standard like you. You are not a special butterfly in the software you make, that you should just be doing the same process as what others do.
Mon-Chaio: So it may be, but I will challenge that
Andy: where we are.
Mon-Chaio: I’m not gonna challenge you. I will challenge them if that’s what they’re saying, as well as something that they write in their original paper, which I found pretty interesting. So there is a section in the original paper about the ability of a mature organization to more accurately predict the performance of [00:16:00] your organization in delivering software. They have a pretty little chart that says, well, you know, if, uh, if you’re in level one, your outcomes are gonna be very varied and you’ll, you know, only hit your dates maybe closer to sort of the tail end of the bell curve.
And when you’re at level five, your bell curve is gonna be very tight, right? You kind of know what you’re gonna deliver. The last paragraph in that section says. The improvements in predicting a project’s results represented in figure 2.4 assume that the software project’s outcomes become more predictable as noise often in the form of rework is removed from the software process.
Unprecedented systems complicate the picture. Since new technologies and applications lower the process capability by increasing variability. Even in the case of unprecedented systems, the management and engineering practices characteristic of more mature organizations help identify and address problems earlier in the development cycle than they would have detected in [00:17:00] less mature organizations.
You know, and then they go on to say, early detection defects is good, or whatever. But it’s that first part of it, right? It’s goes back to what we were talking about with the wicked problems was, is what I’m thinking about. And so I think this fits well for Tang problems.
Andy: Yeah. Yep. Absolutely. And actually, I was gonna say, as you were reading that, the first, the thought that came to my head was, well, software development is basically an activity. This is my. My hypothesis, and I’m taking it. Mac Fleet from Alistair Coburn software, uh, development is basically an activity of discovery, learning as a group, uh, interaction, decision making and trade-offs in judgment.
It is not a factory thing where we can drive out, uh, like the, that kind of [00:18:00] variation on, oh, well you just write this line of code because I will probably. Never write that line of code again. And in fact, we have methods where I never have to write that line of code again. So I’ll never have to revisit, uh, the, that exact same decision again. And so, in my mind, a lot of what we do in software organizations or software heavy organizations is always new Now. Maybe there’s edge cases where it’s like, ah, we just churn out a website for clients. But even there, it’s not the same thing again and again, because each time we’re having to interact with a different client who has never made these choices before.
Mon-Chaio: Yeah, that’s a good way to put it. And I think that underlies a lot of the misunderstanding between folks that are in software and folks that are adjacent [00:19:00] and outside of software. And I’m not even talking about, oh, CEOs of companies that are software adjacent sometimes that even has, is the distinction between the product, organization of a company.
The software development organization of a company, right? This understanding of just because I’m doing another page on the application doesn’t make it the same thing. Sometimes it is, but very often it isn’t. And I think that the inability to understand that, I think is a huge gulf, um, between the people that do and the people that don’t.
Andy: Yeah. And even if it was 10% of the time, we’re doing the exact same thing. Something that has been done before by this organization. It’s probably not been done that by that exact person or that full group. And then you have the entire question of, well does the training to make it so that everyone now knows exactly how to do that thing the exact same way every time.[00:20:00]
For that 10% of cases where this might come up, does it actually outweigh the overheads of imposing that training and imposing the structure to make sure that you can do it, and the learning overheads of, of the people to realize that they’ve picked up on one of these things that has been done before and, and so on and so on.
Mon-Chaio: I think that’s an interesting point, Andy. It actually makes me think about maturity of organizations and this new LLM based coding co-pilot type of situation,
Andy: Oh, really? What has been sparked in your Maybe this is more interesting than where I was about to take us, but let’s see. Let’s see this goes.
Mon-Chaio: When we talk about things that are repeatable and known, I think that’s what LLMs are really good at. They’re good at taking things that have been done before and repeating them with minor variations and spewing that back out.
Andy: Yeah.
Mon-Chaio: and I [00:21:00] think for less capable organizations, that makes them a giant accelerator.
Now, that’s not to say that they’re not an accelerator for advanced organizations too. I think they absolutely are, but for less capable organizations who are thinking that everything they do is simply a repeat of something they’ve done before. These LLM Copilots are a huge accelerator, but. For organizations that understand that there’s nuance between everything they do, they’re less of an accelerator.
And so then the question is, when you take the output of that LLM, how capable are your people of identifying this is the portion that’s relevant because it’s unchanging, and then this is the portion I have to look into and improve because it is new.
Andy: Yeah. Can, they? Uh, separate the wheat from the chaff, the wheat being the interesting new ideas, and the chaff being the, it’s just run of the mill [00:22:00] boilerplate, uh, repeated, uh, an error case that’s always needs to get handled.
Mon-Chaio: Mm-hmm. I will say though, that throughout my career I’ve always repeated the, it’s not the amount of code you type, right? Like the bottleneck in your system isn’t how fast you can type out code on the keyboard. But I will say that with LLM introduction, um, there is some concept of how fast can you spew out 300 lines of code.
Andy: I don’t think it changes it. And I’ll give you the metric that I think doesn’t change, but the LLMs can help with. Based on what you just brought
up. I think the true metric of software effectiveness, software development effectiveness is how many validated decisions can you make per day?
Mon-Chaio: Interesting.
Andy: And [00:23:00] LLMs help in that, in those things that are repeated where it’s like, I’ve got an error and I. Pasted into the LLM, just like I would’ve done on Stack Overflow, and it tells me, oh, on your Docker image, you need to install this library because they’ve stripped out things. Okay, I now try that, and I validated that decision, so I can do that faster with the LLM, but I cannot validate a decision faster.
For something that’s about the unique properties of exactly what we’re trying to do and how that varies based on our business rules. The LLM isn’t gonna make me faster on that. So if you take that as our, as our measure of what is an effective software organization is how, how many validated, um, decisions can you make, uh, the LLMs help you?
In some cases, which you could say is like how much code can you output? [00:24:00] But it’s really, it’s that they’re helping you find validated ideas. Uh, but on the things that you need the full validation cycle to do, they’re not gonna speed you up.
Mon-Chaio: Right, or, or they’ll speed you up. Im materially in of the boiler probably part that we talked about.
Andy: Yeah. The little things here and there. Getting, getting you past little roadblocks. Uh, but to me that’s also why, uh, things like test driven development actually do speed up organizations because what they say is that. What it does is it says that that code that you’ve written, I have validated that it’s correct because I’ve wrote the test upfront, confirmed it with myself, confirmed it with my teammates, confirmed it, possibly with my product manager if we’re doing this really well.
I validated that decision before I even bothered to write the code and, and then I, I then it validates that the code does what that decision was, so. [00:25:00] That, that to me, that’s why TDD helps in those things. And what the, why it’s connected to that measure.
Mon-Chaio: Interesting.
Andy: The, I I bring this up partly because I’ve been doing interviews recently, uh, interviewing people, and I had someone first time really try to lean on an LLM to, to solve the problem.
Mon-Chaio: Oh.
Andy: And I, I thought it was fascinating to watch. Uh, spoiler. They didn’t do very well.
Mon-Chaio: Hmm.
Andy: But what I was watching was that they were missing to put it in these, in these terms, they were missing validation of their decisions. They were letting the LLM just produce things and produce things and produce things, but they never took it back to say like, does it do what I need?
They were trying to do that just by reading the code, and they’d read the code and they’d say, no, do this now. No. Read the code again. No, do this now, but. As I watched it, I was like, and I told them eventually, I said, I don’t think this is getting you any [00:26:00] closer to a working solution.
Mon-Chaio: Mm-hmm.
Andy: Um, and, and that was the thing I thought was really interesting was the LLM had no connection to what, what a validated idea was, but it could produce plenty of reasonable looking solutions.
Mon-Chaio: To maybe the wrong problem, possibly if you don’t have the context. Yeah. Um, that’s really fascinating. I mean, I think, um. I’m gonna be doing a bunch, well, not me, but my org is gonna be doing a bunch of, um, interviewing too. And I think, uh, you just, uh, sparked that idea that maybe I should track and see. How well the LLM assisted interviewers do versus the non LLM assisted.
Um, spoiler alert, we are not gonna be doing leak code style questions and so, uh, the boilerplate stuff that the LLM is good for is already eliminated. But taking this back to CMM and maybe only tangentially, but we do want to connect the loop at some point. One criticism we talked about with CMM. Is [00:27:00] this idea that it doesn’t have a model that it goes by, that it just has a bunch of, uh, solutions that it says, Hey, there’s a bucket of solutions here.
This is level three. Here’s a bucket of solutions here. This is level four. Um, and we said, well, that, that makes it really difficult to understand the whys, even if you don’t understand the why. Then you make, it makes it difficult to predict, well, uh, this solution didn’t quite work. Now why and what part of the solution can I change to make it work better, uh, to get to the next maturity level?
Because it simply says, just do this, um, it works. And I feel like that is exactly what Vibe coding is with LLMs, right? You’re the LLM. The model for the LLM is stuff that’s been done before. In some ways it’s like a CMM model. It says, oh, if you want to do a loop, here are the 17,000 ways I know how to do a loop.
Um, you should pick one of these ways if you wanna do X, if you want to do Y. And what happens is it spits things out [00:28:00] and you look at it and you say, well, this thing isn’t quite working, but. You don’t know why. If you’re a reasonable developer, maybe you do know why, right? But the LLM doesn’t know why. And so it just continually spits out other pieces of things that have worked in the past until hopefully you get to a solution.
And so you see a lot of folks who are going down this vibe, coding path who aren’t engineers. And I mean, a lot of them have never written code before. A lot
Andy: Mm-hmm.
Mon-Chaio: of them have never gone to a Linux command line before, and they’re just using LLMs to say, oh, it says type this in the command line. I type that, here’s the error message.
I pasted it in, it said, type this. And some of them are successful in terms of getting what they think is good product out, and maybe it is good for their use case. But then you see a lot of ’em reaching out for help. Like, here’s where I am. And actual engineers are looking at them like, how did you get to this state?[00:29:00]
Andy: Yeah.
Mon-Chaio: They can’t dig themselves out because they don’t have the context to be able to look at something and understand why it’s wrong. They just know that it is wrong.
Andy: Yeah, it’s an interesting one. I’m gonna leave us with one thing on that and then I’ll take us to something. Uh, it’s an interesting thing to think about on that, which is that they’re getting to the point where they can validate a decision, but there’s a hundred other decisions that they’ve made that the question is, have those been validated? Maybe that’s the thing that’s going on is that there’s, there’s a discovery that there’s a whole bunch of decisions that don’t ever need to be validated up to that point until that other decision is validated. is kinda like the product market fit
Mon-Chaio: Uhhuh.
Andy: and you can, you can ignore validating a bunch of those decisions until you’ve done this one kind of overriding one, and then what happens after that point.
And that’s, I think what you’re saying is that they hit this point where. Of the [00:30:00] other decisions that haven’t been validated now suddenly matter.
Mon-Chaio: And do they know how to pop back into the stack into figuring that out?
Andy: Yeah. Hmm hmm. All right. Back to CMM
Mon-Chaio: Mm-hmm.
Andy: which is, I’m gonna say we’re kind of giving the whole thing short shrift, not entirely, but somewhat because it does. It does. It has things that. I think it could claim our models.
Mon-Chaio: Okay.
Andy: In the document as they describe it, they have these, uh, process areas and in each one of the process areas, they provide a diagram describing interactions between the parts interactions among the engineering process areas.
The one that I have on my screen right
Mon-Chaio: Mm-hmm.
Andy: and what it says is that, uh, for instance, requirements management. Feeds requirements into the engineering process area, [00:31:00] and you have requirements development, which feeds product and uh, product component requirements back into the requirements management, and it has these other interactions and what they’re trying to get to.
There is a bit of a model of how do all of these parts interact to produce the whole. But it just feels like it falls flat to me. And I think I was thinking about this, I was trying to figure out like, why does this feel like it falls flat? What is missing? And I think one is that what they’ve actually got here is just, well, here, here’s some data flows.
Mon-Chaio: Mm-hmm.
Andy: Um, that basically it’s a data flow diagram.
You’ve got this activity and this data flows from here to here, but it doesn’t tell you anything about the influence it has on that other activity. It doesn’t tell you anything about the impact, the, the effect that it has. [00:32:00] And Mancho, do you know, you know, when we read a bunch of these more scientific papers that are like social science research and things, and they, they’ll have these diagrams where they’ll have boxes and arrows, and then they’ll put numbers on those arrows, or sometimes they’ll just have a plus or a
Mon-Chaio: Oh yeah, absolutely.
Andy: Those are kind of like the, that this world of how you say, well, this thing influences these other things and it increases it or decreases it. It’s like you have these different measures of these different things that can be going on and this increases or decreases that, and this increases or decreases this other thing.
And that’s how it influences the world around it. Each one of those components, and that’s what’s missing here, is we have. Named areas and data that flows between them as you might get in, like, uh, uh, something in biology of like the a TP
Mon-Chaio: Mm-hmm.
Andy: It’s like, oh, well this, this molecule comes in here, it gets modified by this enzyme, and then it’s into this [00:33:00] molecule that goes into these two places and these reactions happen.
It’s kind of that without the magnitudes applied and without kind of. Other things. So I think if they wanted a model around this, what they would have is all of their measures that are probably in TSP and other things. And then they would have, uh, their activities and they would have, uh, qualitative or quantitatively backed data, uh, on
what are the effect sizes and effect directions from each of these activities on each one of those measures, and that would start giving us a bit of a model, because then you’d also have like, well, if you do this activity, it will decrease your ability to do this activity because of this mediating with this mediated variable and all of those kinds of things.
It would start giving us a why. Why do these things happen?
Mon-Chaio: [00:34:00] Yeah. Oh, a hundred percent agree. I love that level of thinking and. I think you’re absolutely right. And when we talk about quantitative measures, it doesn’t have to be for every requirements word adds 1.3%, uh, effectiveness to the product. It can be like in those social sciences papers just to say, look, this is a mediating factor, right?
Versus another effect that’s stronger or weaker. And so, you know, oh, well, um, if I do this, this will affect this, but it doesn’t directly affect it. Versus, oh, there is a direct effect on this and it is large or small. Um, and in some ways that’s how anybody who wants to develop a systems model works, isn’t it?
At least I think it is. You kind of do your research and you do your audits about what’s out there, and you figure out, oh, here’s what we’re doing and here’s the discrete processes and here’s how they kind of fit together. Then you [00:35:00] have to think about, well now that I have all of that stuff and I can describe the way the system works today, how can I describe the way that the system evolves from that and the, the steps that the system will go through in order to evolve and why , certain things change.
So the bones are here, right? And to your point, they can just layer on the next additional part. Not to say that’s easy. That may be the most difficult part of systems model
Andy: very likely. It’s the most difficult part. I mean, it just. Evidently it’s the most difficult part. We, they’ve come up with all of these symptoms and these solutions and these measures and we have hundreds of these things, different processes and all of that. And what we don’t really have very strongly is, well, how do all these things interact And how much of an impact will me telling my team to adopt test driven development actually have on on our [00:36:00] ability to ship on time?
Mon-Chaio: Right.
Andy: Something like that.
Mon-Chaio: Well, and that’s also what you know for non-technical folks, that’s what they’re looking for because often they see that in their other disciplines, rightly or wrongly, about how those other disciplines measure. They have those measurements, uh, for their sales team, for example. They know that every five clients that they call, they get.
A certain output. And so they look for that from engineering organizations. But I think because of the state of engineering research, a lot of this comes out as to like, we don’t know, but we know this is good. Or uh, even worse, right? That we talk about a lot. Google does this,
Andy: Yeah.
Mon-Chaio: so we’re doing it. And what’s really funny, the last thought I actually have, and maybe you think you have a final thought and we can wrap this up. Level five in their maturity model is continuous improvement. But I find it difficult to [00:37:00] truly, continuously improve. If you don’t understand the whys, it feels like the only thing you can do is sort of reduce the portions of the quantitative metrics that you know about. My code coverage is 60%. I need to get that up to 75%. That’s
Andy: code, our code reviews take two days. We need to get that down to one day now.
Mon-Chaio: right, but in between the systems, to your point, the arrows and the magnitudes, you can’t say. I need to eliminate code reviews at this point in time for my software project. This model gives you no option to do that as part of its continuous improvement process. I love the idea that they, that they say continuous improvement is the height of a mature organization, because I think that that in itself means that the majority of organizations aren’t mature enough.
But I just don’t think they give the tools for us to be able to do that well in this model.
Andy: Cool. Yeah. Oh, I agree with you on that.
Mon-Chaio: All right. Well, this was, [00:38:00] uh, a really fun thing to talk about. Andy. I’m glad you thought about this as we were talking about TSP. Because like you, I’d seen the acronym CMM all over the place and just never actually read about it. And it was a good opportunity to really dig in and say, you know, what was the level of thinking here and how did it drive all of these other, uh, spawns, um, that, uh, that populate the software engineering world today? Well, I hope all of you listeners have enjoyed our conversation as we talked. Quite a bit about CMM and then kind of veered off and talked about LLMs and copilot and all of those types of things. Um, but this is kind of what we do. We wander about, we try to find the interesting parts of the conversation. Did we miss anything? I mean, the paper that Andy was referencing was what, 700 pages?
Andy: Yeah.
Mon-Chaio: And I think the [00:39:00] original CMM paper was. About 50 pages. All of these things
Andy: What is it that big of a difference?
Mon-Chaio: is a huge difference. Yes. So maybe you have dug deep into CMM, maybe you using it to assess the maturity of organizations. Maybe you work in the government, maybe you’re a contractor.
And you think that we’ve missed something because obviously we can’t cover 50 or 700 pages in a single episode. Write us, let us know what do you think about CMM or our tangential topic? What do you think about vibe coding and LLMs? Maybe we should just do a whole series on that, Andy. I feel like occupy half of a season there.
So write us hosts@thettlpodcast.com. That’s hosts with an S. We always love to hear what you think. Also for folks that heard Andy’s vacation cast, andy will [00:40:00] be at least temporarily leaving the podcast. Who knows what the future holds. If he comes back as a master woodworker and decides not to want to podcast about software engineering, technical organizational topics anymore, also write to us and let us know how would you like this podcast to evolve?
Would you like to be the next co-host? Temporary or not of the TTL podcast. Would you like more case studies? Would you like more vibe podcasting, where we just look at the latest LinkedIn post and talk about what we think about it? Also, let us know hosts@thettlpodcast.com. Uh, listeners like you all are why we are successful and, uh, we wanna make sure that what you want we can help deliver.
All right folks. Until next time, be kind and stay curious. [00:41:00]
Leave a Reply