Transcript

Hello. My name is Lauria. I work here. I’m going to introduce Shriram. So Shriram Krishnamurthi is a professor of computer science at Brown University. He’s done research in a number of areas.

Okay. Stuff that he’s worked on that you might’ve heard of is like Racket and the book, How to Design Programs and an outreach effort known as Bootstrap that does computer science and math. And I’d like to tell you that he is a campus personality at Brown University, and he has exposed thousands of programmers to functional programming. And for that, we are all very grateful. And today he’s going to tell us some cool stuff about how he teaches people at Brown and elsewhere.

Thank you.

It was not one of those.

Yeah, no. It could have been worse. I was worried where that was headed. It sounded disturbingly academic. Let me put this on for the benefit of recording.

Okay. Thank you everybody for coming. I would like to first make sure I understand by the show of hands, how many of you identify as a sort of daily working engineers, like you code or you manage systems or something for a living? Okay, good. Some alumni in the room. Thank you for coming. I mean, they came back, that’s pretty disturbing. How many of you are teachers, educators of some sort? Oh, wonderful. Awesome. My people are in the room. Great. Okay, good.

So let’s start with the engineers. So here’s the situation. So everyone’s trying to build out some kind of computer science curriculum or the other. So somebody walks into the room and they say, “I want to teach computer science. You’re the expert, what should I do?” Some of you almost certainly have been asked this question and if not, I can assure you, you will be asked this question.

So just to get you a little started, this kind of question shows up on the Brown Computer Science Department Facebook list as well. So here’s a question. Somebody says, “I was wondering if anyone has any advice on coding activities. I’ve taken CS-4,” which is a terminal course. It’s for non-majors, but it’s for STEM non-majors. So like the engineering and science majors. “So does anyone have any advice?” And you get lots of different advice. Somebody suggests Scratch. Somebody suggests Logo. I had a lot of fun with Logo and Turtle Graphics. Someone else actually really hates Logo and says, “I was very frustrated and things only started to make sense once I got to C++,” et cetera.

Okay. All right. I mean, Brown, what are you going to do? We’ve got to make due with what we get. So any additions, any suggestions, anyone in the room? Engineers, anyone? Throw something out. Come on.

Haskell.

Haskell. Okay, good choice. Anything else?

JavaScript.

JavaScript. Okay. Good choice. Any others? Come on, come on.

OCaml.

Huh? OCaml. [inaudible 00:02:51]. Okay, good. You earned your dinner. Okay, next. That’s it?

JavaScript.

All right. Now, we’re done. Marks over. Any others?

ReasonML.

Huh?

ReasonML.

Ooh, ReasonML. We’re upping the ante here. Anything else?

Assembly.

Assembly. Okay. Anything else?

Perl.

Perl.

Okay. All right. So we could go on for a while, and a lot of you are going to have different … And of course, if I asked you you’ll have all sorts of … Well, I assume you’ll have all sorts of justifications for your answers as well; why this over that? Why that over this? One hopes. Now, let me ask you a different question. So this is again for the engineers in the room. Somebody walks into your office and they say, “I want to make some software. You’re the expert. What should I do?”

So of course this is Jane Street so OCaml.

[inaudible 00:03:51].

Oh. How about that? So notice that you actually asked a question before providing an answer. What a concept. So before we talk about solutions, in every other engineering domain, what we do is we ask, well, first of all, what is your goal? What are you trying to achieve? And then you might ask, what are your constraints? What are the things that might prevent you from using some solutions and some other solutions? And then we start to talk about the methods and eventually put all these together and come up with solutions.

Now of course, when it comes to education, it’s kind of backward. We start with methods; OCaml versus Perl versus everything else. So what I’d like to do is I’d like to think about curriculum design a little more from the perspective of how I think about other kinds of problem solving, like when I’m building a system or administering a system or running a system or any of these other things.

So we’ve only talked about methods, so now let’s go back up the hierarchy a little bit. Let’s talk about constraints. What kinds of constraints might be there on a curriculum? Well, anyone want to throw out some constraints?

Time.

What else?

Standards.

Ooh, standards. Okay. Teacher. Yes. Okay. Excellent.

Training and teaching staff.

Training and teaching staff. Oh, so you do know some questions. Very good. Other questions?

Equipment available.

Equipment available. Here’s a few that you could come up with. First of all, you might ask what age are the students. In that particular case, we knew that they were fifth graders. You might ask, what have they seen before? If these kids have already done two years of Scratch, what is the point in recommending Scratch? I mean, it’s possible that Wall Street is hiring hardcore Scratch programmers. And so a third year of Scratch would be a good idea, but perhaps not. Is this a required class or an elective? How long is the class? How often does it meet? Is it even a class? Maybe it’s like an afterschool thing of some sort. And how many students are in the classroom? The thing that works with two students doesn’t necessarily work with 30 students and vice versa.

You all would have come up with this list and several more. Let me suggest a few more things that perhaps not all of you would necessarily have thought of. Is there internet access? And you’re thinking like, wait, which century are we in? But of course the teachers in the room are looking, thinking, yeah, that’s not a bad question to ask. Of course, are there any Googlers in the room? I don’t think Google even knows about the existence of non-internet places, judging from the software on my Android. But it turns out that not every classroom is fully equipped with internet all the time.

Is the teacher new to programming? Now, I don’t know about you, but somebody who has never programmed before suggesting they go and teach Java may or may not be a feasible solution. Can the students type? You’re all expert typists. I’ve seen some of you type. Some of you’re typing away in the middle of my classes, even. But not everybody is an expert typist. And if you can’t type, all of a sudden you have some very real constraints about what you can possibly program. And again, the teachers are nodding and the technologists are kind of staring at me like, what?

Are there students with disabilities? Has anyone in this room used Scratch? Can somebody tell me what Scratch is? What is Scratch?

It’s a drag and drop block programming language.

Wonderful. It’s a block-based programming language. Everything is done through drag and drop. Has anyone done Scratch with a child, maybe their kid or some other kid? Have you done it with your eyes closed? No, you should go home this evening and try it with your eyes closed to simulate what it’s like to be a blind programmer who has to deal with Scratch. Or somebody who has maybe motor impairment, trying to drag a block exactly into that tiny little notch where it suddenly wakes up and says, “Yes, you are exactly in the right place.” So, if you’re in a room with those kinds of issues, that might affect the set of constraints that you have.

Do the students have regular computer access at home because that’s going to determine whether you can give them homework or not. And, even more shockingly to some of you, do the students have regular computer access in schools. Not every student sits in front of a computer, thank goodness, 24 hours a day in their school. Some of them might only get an hour a week in a computer lab. How does your curriculum, that is completely driven around them being in front of a machine at every moment of the day, how is that curriculum going to work for them? So this is just a sampling of the kinds of constraints you might want to think about before you propose a solution.

So there’s methods. We know plenty of methods. We’re talking about constraints. Now, let’s do this. Let’s talk about goals. Here’s what I’m going to do, I want each of you to buddy up with somebody else. So turn to your neighbor. If you don’t know who they are, introduce yourself and spend a few minutes talking about goals for computing curriculum. Go.

We’re back here. All right. Who wants to throw out some goals? Throw out some goals. Goals, anyone? You were all talking, so you must’ve hopefully talked about something relevant. Yes.

An entry level activity that is the Blink for the whole world, but it’s so wild.

Okay. Wonderful. Next. There was a hand up over here.

Yeah. [inaudible 00:09:16] relates to real life so that they can see [crosstalk 00:09:18].

Relate to real life. Okay.

Be able to ask better tech questions when they need help.

Oh, that’s interesting. That’s very interesting. Be able to ask better tech questions if they need help. Other things.

Know what’s easy versus what’s difficult to program.

To know what’s easy versus difficult to program.

Building analytical reasoning abilities.

Oh, building reasoning abilities. That’s great. Wonderful.

Get the experience of using a precise system that does exactly the terrible thing you told it to do.

Oh, wonderful. I love that. So being able to use a precise system that does exactly the terrible thing you told it to do. Other things? Trading systems for example. Yes.

Some people believe there’s knowledge transfer that you take that, what you learn in computing and [crosstalk 00:09:56]

Ooh, knowledge transfer. That’s a really interesting one.

… composition and move it to math.

Wonderful knowledge transfer. We’ll get back to that one.

Not saying that I agreed with that, I’m just saying.

Oh, we’ve got a lot to talk about then. Oh, yes.

Cultivate their confidence to create something.

Confidence to create things.

A job pipeline.

A job pipeline. So prepare them for jobs. Okay, great. I’m sure several of you have several more goals. These are all wonderful goals. So what I’m going to tell you about is some of the goals … So as Kshitij said at the beginning, we’ve been running a program called Bootstrap, and I’m going to tell you what three of our major goals are. We have what I consider minor goals, like student engagement and things like that. We can simply take that as pretty evident for us. I mean, nobody goes in saying, “I want to make my class as boring as possible.” So we can take those as a given.

But I’m going to tell you about three goals that drive everything that we do. And they’re not in any particular order, and it seems to be hard to read on some of these screens; equity, rigor, and scale. So rigor is pretty evident. I’m not interested in teaching a non rigorous curriculum. And you might think, well, who does? You should spend some time looking at computing curricula then. For lots of interesting reasons that we’ll come to later in this talk, there has been arguably a lowering down of rigor in many computing curricula. And I’m not willing to compromise on that. Yes.

What do you mean by rigor?

I’m not going to get into that here because we will spend all evening on that. I will take this as a Justice Potter kind of thing. And if you’re still not convinced by the end of it, we can talk about it later on. Is that fair? Yeah. Okay, good. Scale, well, here’s the thing. I, in my day job, I work at Brown and I don’t have to worry about scale. Some committee takes care of admitting 1,000 out of 35,000 students or something like that and I teach to those thousand and I don’t care about what happens to anybody else. I’m not paid to worry about it.

In my non day job, which is Bootstrap, I care profoundly because we can’t come up with nonscalable methods and expect them to apply in large sets of circumstances. We have to care about scale. And that means sometimes compromising on … The kinds of things I do at Brown are not necessarily the kinds of things I can do in a scalable setting, and that’s something I’m very consciously aware of. But I would like to take everything I do at Brown and move it down, eventually. And then equity is critical because I at least believe, we in Bootstrap believe that we have a representation problem in computer science, and there’s no good reason for us not to be working to make it better. So we care about what our consequences are for equitable learning.

Okay. So, let’s talk about three different methods we’ve studied carefully about how to take all these ideas and put them out inside a curriculum. So there’s three avenues that are very commonly used. One is required courses, the second is elective courses, and third is afterschool programs. Let me discuss all three of these through the lens of our goals of equity, rigor, and scale.

So required course; make computer science a high school graduation requirement. There are some states where this starting to happen. It’s being put in place. There’s a lot of fanfare. States are really excited about it. Let’s talk about what the consequences of doing that might be. Well, the problem with required courses is, they don’t scale. And here’s why not? If you’re going to have teachers, you need to worry about certification. Now, again, the tech people in the room are like, “Certification, I’m not certified.” Well, good for you. But teachers it turns out have to be. And you might say, “Well, we could work around that.” But you have to find teachers. The problem is, anytime you get a qualified teacher, people like Jane Street go and hire them and pay them twice as much as what they get paid otherwise.

Okay. Now, this is not to say there are not qualified teachers. I can see some of them in this room. I know some of them really, really well. Eric Allatta is over here from Academy for Software Engineering. And Eric is astounding. The school is lucky to have a person like him. But there aren’t that many teachers who are willing to make that sacrifice. So getting teachers who can teach really high quality curricula, not that easy, especially when you have an economy for computing people the way we currently do.

Okay. Then there’s this minor problem of paying people for doing this. If you just take the number of schools we have in the country, about 25,000 schools, you multiply it out. Take two to three computer science teachers for different grade levels. Even just say one primary, one middle, one high. You’ve got to pay each of them. Well, you got to pay them something. It’s not going to be industry salaries, but you still got to pay them. You got to worry about benefits. We’re talking literally billions of dollars a year. And finally, there’s not enough time in the day and space in the curriculum to squeeze computer science into the curriculum all that easily.

So what ends up happening is, when you make it required, things start to fall away. Scale falls away. Some schools can afford to pull this off. A lot of schools cannot afford to pull this off. They can’t find teachers. They may work in an area where it’s difficult to attract computer science teachers. All sorts of things start to fall away.

The other problem is, once you start making things required, rigor has to kind of take a hit. That’s just an inevitable consequence of trying to make sure that everybody has to get through this course. If everyone has to pass this course to graduate from school, rigor might well take a hit.

So there’s a pretty obvious solution. Don’t make it required, Shriram. Let’s make it an elective course. So students are going to opt in to take an elective computer science course. Let’s analyze that solution. The problem with elective courses, they’re not equitable. We already know this. We already know from the data coming out of schools that the statistics of who takes optional, elective, computer science classes is extremely heavily biased in favor of White and Asian males. And once you make it elective, and once that’s who’s already taking the course, that’s going to make it even more likely that more people who take it are going to be those and not anybody else.

So that’s already a problem and you’d say, “Well, that’s okay. We can work around that. I’m sure we can work around that.” And maybe we can. But let’s think a little bit about what happens from the point of view of setting up a graduation pipeline. What schools focus on rightly is the courses that are required. And what are the required courses, typically literacy, math and maybe some science courses. So students have a curriculum like this and there’s a hole in there for an elective, and all of them are going to take the computer science course. That’s wonderful in theory. What happens now is a student maybe fails Algebra 1.

The moment they fail Algebra 1, and if you don’t know what the failure rates in Algebra 1 are, you should look them up and not … Well, the failure rates are a little disturbing. So they fail Algebra 1 and boom, all sorts of things that depend on it are shifting down. And if they happen to fail a second class, boom, things shift even more. And as a result, what you end up with is for students who are failing required courses, there’s just not enough room in the curriculum for anything else. So what happens is, it’s the students who … These are students who may have actually benefited from a computer science course. They may have related better to a computer science course than they did to Algebra 1, but they’re just never going to get to that room in the curricula. And there’s also, sadly, a so-called achievement gap, which I don’t quite like that phrase, but that’s the technical term.

Basically race and class, for various historical and social reasons, end up having a disparate effect on these kinds of pass rates. And so you’ll find a ton of graphs like this if you go look on the web. And so you’re also now further reducing equity by trying to make it an elective course.

So the elective problem is, first of all, equity drops out and rigor ends up sometimes taking a hit as well because it’s an elective. It’s okay. Trying to teach something. It’s all right. Some schools do an amazing job with electives. Some schools do a terrible job with electives. How do I know? Because I see them coming downstream into my intro class that Kshitij TA’d and actually more than Kshitij TA’d. Multiple of you, I think TA’d.

So we see the effect downstream. There’s a huge disparity between elective classes. Okay, good. So here’s our solution; afterschool programs. All right. Here’s what we’re going to do as an exercise. Here are our goals. You’re going to play Shriram. Afterschool, tell me about it. What do you think? Solution?

No.

No. Why not?

No rigor.

No rigor? Okay. Maybe rigor mortis sometimes but not rigor.

Well, because you don’t really have a lot of [inaudible 00:18:43] in the staffing.

Okay.

Not a lot of time to train them [inaudible 00:18:48].

Because?

Because it’s not all kids. Some kids can’t afford to go to them. Sometimes the kids have to work, babysit their siblings.

Thank you. Yes.

Can’t you work on some of that?

So Bootstrap,we started as an afterschool program. We were one of the most successful afterschool programs. Here’s a problem, when you do this through volunteers and some of the people who are in this, they might even be people in this room who did Bootstrap volunteering. We got amazing volunteers. We got Apple and Google and all these wonderful companies sent employees to volunteer. Well, it’s exhausting and volunteers leave after a while. They’ve gotten their hit, they leave. Most afterschool programs tend to have a returning rate for volunteers; 25, 30% is pretty good. Bootstrap was at about 70%. So we were actually doing really, really well.

The problem is much bigger than that. In terms of equity, as you said, a student who has to earn for their family or the student who has to go home and be the babysitter for a younger sibling so that their parent can go get a second job, that student has already been eliminated from the computer science afterschool program. Our experience was, on average, students would make it to about 50% of the sessions. I don’t know what other people’s afterschool, but this was across the country about 50%.

Well, anybody here who’s had any STEM training knows that the problem with STEM is all the stuff is incremental. So if you miss a class, you’re playing catch up. It was our experience in every single afterschool offering that by the time we got to the end of 10 weeks, literally every single student had their own customized program because they were all on different paces. This was almost like some sort of combinatorics game. It’s like can choose two for the two Boolean values, and you have all of the possibilities there. So you can’t build a rigorous program. And I don’t know how to solve those problems.

And ultimately, even if you can solve the problem and you can make sure every single student is going to come every single day, you still have the problem that they’re exhausted at the end of the school day. So I don’t know. Equity is hard, rigor is hard, maybe scale works. Even scale is a little questionable. If you are in a rural area and you don’t have a group of volunteers from Google and Apple, well, where are you going to get people to come and do this? I don’t know.

Okay. All right. Is everyone completely depressed? Good. Okay. Let me now tell you a way out of this problem. And I’ll start by presenting to you an inequation. So the inequation is the following. We’re starting with the premise that everybody should learn CS. We’re just taking that as a premise. We’re not going to argue about it. If you want to argue about that, I’m happy to discuss that later on, but we’ll take that as a premise. Everyone should learn CS. That is not the same as saying everybody should take a computer science class.

Okay. So here is option four, integrated computing. After all, as you said at the beginning, if computing is in fact a universally applicable skill, it’s the new mathematics, it impacts every single subject we’re going to study. And I genuinely believe it does. I’ve been designing curricula at Brown since 2003 on that premise. Big data courses. We started our first big data offering in about 2003 before the data were big. I believe this, genuinely believe this. If that’s the case, why bring everybody to computer science? Why not take computing to them?

Imagine if you went to an econ course, you were told, “I’m sorry, you’re not allowed to write. This is the econ course. It’s not the English course.” That’ll be insane. We allow these tools, these universal tools to pervade curricula universally. So let me tell you one particular favorite subject, and this is the part where you and I are probably going to have a good discussion, which is algebra.

Okay. So let’s talk about algebra. First of all, why does algebra matter? Well, first of all, most critically, it’s the gateway subject. If you look at the literature on math education, algebra is the point at which students start to drop out of math. It’s the point at which students … The diction that students use changes from math is hard to, watch for it, I can’t do math. Why is algebra difficult? Someone, why is algebra difficult?

Requires abstract thinking.

What kind of abstract thinking? Requires abstract thinking. What does that mean?

You have to look at a set of symbols and imagine them having concrete values without actually knowing [crosstalk 00:23:28].

Ah, so you have to imagine that those Xs are actually going to be concrete values without knowing what the concrete values are. That’s pretty good. Anyone else want to throw something in? Yeah.

You’re usually not taught super concrete algorithms and [inaudible 00:23:41] arithmetic.

Ooh, algorithms. That’s interesting. I’ll get back to that. Yeah.

I studied this a long time and I think the problem is not algebra, the problem is arithmetic. And kids who have trouble with fractions can’t get through algebra. It’s a very high rate. And it’s not that you’re required to be able to handle fractions in order do algebra, but kids get turned off to mathematics, not in ninth grade or eighth grade when they take algebra, they get turned off in fourth and fifth grade when they [crosstalk 00:24:20].

That may be, but there is also a great deal of research showing that this is the turning point. Because what happens is until this point, as Kenny said, until this point, math is a procedural discipline. There is an algorithm. You may not like it. You may not have internalized it for fractions, but everything is solve for X and you know what you have to do. There’s an algorithm. You move this thing over there. You put this thing down here. You put slashes. You cancel things out. Don’t cancel the Ds and D by DX, but everything else you can cancel. You know the algorithm. This is the first time there’s no algorithm. There’s nothing to solve for. F of X equals X squared, solve for what? There’s nothing to solve for there.

It’s the first time where you have to deal with abstract objects. And what’s worse is you have to deal with abstract properties of abstract objects. Is this function injective? That’s an abstract property of an abstract object. So we care about algebra partly because it’s a gateway course. If you can’t do algebra, you sure as heck are not going to turn into a civil engineer or a chemical engineer, forget software engineer. You’re not going to do any of these things. So it’s the gateway course for all of STEM. You’re also unlikely to be a very successful accountant or work for a trading firm or anything else. So we think it’s a crucial pinch point. It’s the dominator in the dependency graph.

Next thing is, it’s very highly tied to income and equity, sort of for the reasons I already talked about. The third thing is it’s a graduation requirement. It’s already being tested for. It’s a high stakes subject.

Okay. Now, that’s going to come back and be an issue in a moment, but let’s go with that. So here’s the thing about integrating with algebra. First of all, every kid already has to take algebra. So if your goal is to get to every kid, if you can get yourself into the algebra class, you’ve got every student automatically. Second, you can leverage a huge existing network. You don’t have to hire anybody new. There are teachers, there’s already courses. There are whole organizations like in our National Council for Teachers and Mathematics Teachers, NCTM. There’s all sorts of organizations for professional support of these teachers that already exist and are deployed on the ground so you can leverage this entire network.

Third, did I mention that you can get to every single kid? And fourth, the content is already well-suited to rigor. Algebra teachers don’t go in saying, “Today, I shall think about how to entertain my students first and foremost, and nevermind whether they learn anything or not.” They’re already teaching a rigorous discipline, and if we can work with them, that would be really great.

All right. So it turns out there is a free lunch. Get computing into algebra classes. Problem solved. Well, as you can tell from the time, it’s not. Here is what is difficult about integration, and it’s what you were talking about earlier. It is the problem of transfer. What does transfer mean? Transfer means you learn a skill in one context and you have to successfully apply it in a different context. That’s a term out of education. It means that’s what transfer is. And a lot of people have studied transfer for a very long time. And the literature on transfer generally identifies three things that you need to be able to do to achieve transfer. The first one is you need explicit instruction. The second one is you need deep structural similarities between the two things you’re trying to transfer between.

There are studies in physics for example, where students cannot transfer the idea. You teach them about dropping a ball through a cylindrical pipe down about this height, and they master the subject. Then you ask them, “What happens if you drop it from a tower?” And they have no idea. Transfer is hard. Transfer is really, really hard. And you’d be surprised by how fragile it is.

And the third one is you need a process for performing tasks. Without a process, you’re not going to get any transfer. So there’s a whole colloquium and a half I could give you about what we do about this. I’m going to give you two small examples of this. But let’s talk about this deep structural issue. Because you’re going to be asked to help a math teacher, and you’re going to say, “That’s great. I heard a talk. And there was this guy from Brown who said something about transfer, and he said math classes are the way to go.” And so you’re going to teach programming in a math class and it’s all going to be awesome.

Okay. So let’s take a look at how programming stacks up against math. Coding as you know doesn’t have functions. Now, those of you who are JavaScript programmers must but like, “No, no, no, there’s a keyword. It’s even spelled function.” It turns out there this minor inconvenience where there’s … Math teacher’s like this thing called the vertical line test. They’re absurdly attached to it. And your functions that don’t satisfy the vertical line test are actually teaching students the worst possible thing. You’re going to use the same word for a completely different concept. That’s not going to confuse anyone, right? Sometimes your functions don’t even produce output. That’s not going to confuse anyone.

Turns out you don’t have variables because you can write that, but you can also write that and that’s not a concept that exists in any algebra textbook, or you can also run into trouble with numbers. I’m going to show you something. What is that? 1/2.

Zero.

Yeah, zero. Exactly. You should see what happens when you present this at a math teacher conference. Because they have enough trouble. We just heard about the difficulty of fractions. They’re already having trouble mastering fractions and now we’re going to tell them, “No, no, no. You need to understand. It depends on how many bits your hardware has.”

This stuff, equivalent? Maybe, maybe not. Overflow, underflow, that’s a really useful concept for a student to deal with while they’re also having trouble with arithmetic and algebra. Or this stuff? Is it the same? Is it not? Depends. Sometimes you add positive numbers, you get a negative result. This is all stuff that inhibits transfer. So for those of you who are a technology people in the audience, technology does matter. Of course it matters, but it matters in preventing bad things from happening. If you are going to make recommendations, think hard about the low level issues of transfer as well. So we work with functional programming, with sane number systems. And by working with functional programming and sane number systems, we avoid this set of issues.

But there’s more to it. There’s also, as I said, a process for performing tasks. So what I’m going to show you now is an extremely rigorous graph. Pay close attention, please.

Okay. So what I’m going to plot on the x-axis is lines of code and on the y-axis, frustration. Here’s what happens, you hand out a programming problem to a student. Now, of course, those of you who are like hackstars in the room, you won’t recognize yourself, but you’ll have to trust me on this when I say this is what happens. When you first hand out a problem, it’s actually a really intimidating experience for a student because they’ve got a problem statement here. They’ve got a blank page in front of them. A blank IDE screen, page, whatever you prefer. And somehow this thing has to be filled up with some magical incantations that are going to end up doing what this thing asked them to do. So they start at a very high level of frustration and then they start making progress and things get better, things get better. So there’s this blank page syndrome that you have to overcome.

And then of course they’re making progress and things are going well. And that’s great, frustration is relatively low. But student programmers are like 3D printers. They work by additive manufacturing. When things don’t work, they add more code. And that’s what ends up happening. And that’s the point when they finally walk into your office and in their good extreme articulation give you a very precise analysis. They say, “My program doesn’t work.” And I could give you a whole talk about what to do about this problem. I’m not going to give you a whole talk about it. Simply asking you to recognize this phenomenon exists. And one of the central problems we have in teaching programming is how to mitigate this problem.

Okay. Now, this is the state of program design. Let’s talk about the state of design in other disciplines. Turns out, of course, we’re not the only discipline that does design. Let’s think about architecture for a moment. You have buildings, but buildings aren’t just … Well, I mean, actually Frank Lloyd Wright may not be the best example to take. A bit of a hacker himself. But buildings come with a whole bunch of different things that make up a building. There are lots of different schemas. There are lots of different viewpoints, lots of different representations, and they have lots of phases that go into the design of a building.

If you take some other thing like electrical engineering. Again, you have lots of different representations. They have lots of different affordances, lots of different phases before you eventually assemble a final product. There is a well understood tradition of design, which is multistage, multi representational and multi viewpoint. A math teacher actually recognizes this issue because one of the things that math teachers struggle to do in algebra classes is get their students to understand that all of these different things are actually the same thing. Actually, if you look at new generation science standards, this is an aspirational goal to get students to understand that these are not four different phenomena, this is one phenomenon that’s being expressed in different ways.

So what we have been doing is trying to put programming in the same kind of context. We actually make students write programs, and this is the point where all of you hackers can laugh out loud, through worksheets. We give students worksheets and the worksheets give them a structured process for designing programs. And we make them work on contracts, make them work on purpose statements, refine that into concrete examples. Because that’s what you do when you solve a math problem. You don’t start by doing a proof from scratch, you start by working through examples and building your intuition from the concrete to the abstract. And once you’ve got your examples, you abstract over your examples to actually produce working code. And those of you are looking carefully at the screen might recognize these are people working in Racket. Yes, they’re working in parentheses. It turns out only computer science teachers have a problem with parentheses. Everyone else is quite okay with it.

And we have names for these different steps in computer science; we call them type specifications, we call them test cases and then we call them code. But of course everything is code. This is just the part that executes. Well, it turns out math teachers have names for these things too. These are called domain and range specifications, input output tables, and symbolic form. And math teachers are trying to make students understand the connections between all of these. We’re sitting on a gold mine. We make these connections concrete. If you get any of these connections wrong, your program will stop and tell you, you got it wrong. How awesome is that? We want to reinforce the connectivity of these. It’s completely intuitive if you work through it in a precise, methodical way, not the way we usually program.

So multiple representations of code correspond to multiple representations of functions. The net result of this is we take that curve and we smooth it down. By doing this multi-stage process, the technical term an educator would use is scaffolding. At each point, you give them a scaffold that helps them build up to the next step and you reduce the frustration of every one of these steps. So in Bootstrap, middle and high school students with no prior computing experience, they build a video game of their own design. So they start–the first five to 10 minutes is spent designing their own video game. They’ve never programmed before, by the end of it, at the end of 20 hours, they have a working game. And in the process, they learn both computing and algebra. And when I say, learn, I don’t say they learn because we taught it, I say they learn because we measure it. We are researchers. We evaluate. We’ve got controlled studies that show actual measurable learning gains against controls.

This is not an academic talk so I’m not going to throw P values at you, but if you’re interested, come and ask me about that separately. So there’s a dependency graph here. The dependency graph says language does matter. Language does matter because the wrong language will screw you up. So you have to worry about language details, but it’s not just about the language. It’s just as much about the curriculum and about the pedagogy, and all three of these are closely interdependent.

Just to give you one example of that, you notice that in the program, we wrote test cases in the source code. If I have to now get my students to go learn JUnit to be able to write two examples, it ain’t going to happen. You’ve now destroyed that link between the math part and the computing part. So there’s a lot of details here, not the purpose of this talk to talk about this. Here’s some obligatory photographs. Here’s some students.

One of the things we’re very proud about is at the end of the course, the students have a launch party because it’s a video game, you got to have a launch party. But not only do they have a launch party, they do code reviews. Yeah, exactly. They do code reviews. The students go up there and have to answer questions about their code in front of a live audience. You also see the girl up in the top right is doing a math review. She writes math formulas on the board and answers questions about the math formula. Code reviews, math reviews, same thing. Deep structural similarities. And so the students get this enormous thrill out of not just showing off a game, but having the confidence to go up there and answer questions in front of a live audience about something that they created. And in the process, they’re doing something that is a truly authentic computer science industry experience.

I can’t get my Brown students to do code reviews. We have 14-year-olds doing code reviews. So we’re all over the country and we’ve got good numbers, blah, blah, blah, female student, color. Obviously they’re great. Why are they obviously great? Because we’re in algebra classes. It’s the same diversity as the school itself. Everybody’s taking it. Problem solved.

So last thing I want to talk about, little plot twist, we don’t have to stop with algebra. So what Bootstrap is now doing, we’ve been doing now for the past two years, is working on another curriculum for data science that we’re now also rolling out across the country. So I’m not going to go through all the details. Let me briefly tell you what’s going on there.

New domain, but it’s the same playbook of everything we’ve talked about so far, but our teachers are now drawn from a much more diverse audience. We have stats teachers. We have social studies teachers, technology teachers, many, many more teachers can teach it because it fits in lots of other contexts as well. It’s also a vehicle for teaching basic stats, which everyone will agree is one of the most crucial skills that we need everybody to be learning. Students don’t just produce programs, they produce a report. The program is a small part of the report, but at the end of the day, they ask a question and they produce a report as an answer. In particular, the lesson I’m trying to convey in this curriculum is we’re not trying to supplant other methodologies, we’re not trying to replace them, we’re just providing one more.

Social studies already have lots of good methodologies. They can do a literature review. They can do surveys online. They can go stand at a street corner and do a survey. They’re all valid in different ecologies. There’s now one more tool in your toolbox, which is you take a question and pose it as a program over a dataset. It’s an opportunity to talk about WMD. What’s WMD? Weapons of Math Destruction. Everyone’s read Cathy O’Neil’s book?

Yeah. Okay. So we get to talk. It’s a good opportunity to also talk about the impact of computing on society when that is not always a positive impact. And of course here’s a place that has lots of need for language support. So this curriculum uses a programming language called Pyret. Pyret is basically, it’s a five-year-old programming language that I invented because I was tired of everybody complaining parenthesis. It’s the same damn language, but it does parens and everyone’s happy now.

So we’ve got a lot of cool support in there for dealing with tables and query forms. So we get past this whole, do we teach them loops or do we teach them recursion? Well, neither. We teach them queries. And once they have queries, they can write a whole bunch of interesting programs without having to learn any of these things. And they can erect really interesting questions very, very quickly.

There’s a third one I’m not even going to tell you about which is a physics curriculum. And so we have three different prongs. We have outreach into algebra, outreach into data science and now also outreach into physics. The physics is being done with the American Association of Physics Teachers and the American Modeling Teachers Association. The algebra curriculum is being done with NCTM, which is the National Council of Teachers of Mathematics. Anyone here recognize the name NCTM. All the math teachers in the room do. And I got to say, every time I look at the NCTM Logo, I just can’t help noticing there’s a lambda and a parenthesis.

So what’s the playbook here? Why does Bootstrap succeed? This is the part that sucks to hear. Because this is the part that is unsexy, boring and not fun. And that’s why we succeed because we’re willing to do this and other people are not. We’re teacher-centric. Teachers are not interested in your fancy, clever programming environment that does amazing things. What teachers need, they need professional development, they need support, they need materials like workbooks, and they need support with all the problems that they encounter in an actual classroom. If you’re not willing to invest the time into understanding what those issues are, you’re not very helpful. You’re pushing a technology, you’re not solving a problem.

We have this idea of personalizing curricula. And again, this is a whole talk in itself. There is this problem we have in computer science education where everyone thinks that the moment you put any restrictions on a student at all, you have thwarted their creativity. Oh my God. It is the Wild West. You don’t dare thwart their creativity. We have a paper coming up next week at SIGCSE, where we show that we can accomplish almost all the creativity you want with literally five minutes of effort, five minutes of time. That’s all it takes for students to feel ownership of their project. So you want to talk about that? Talk to me about it later on.

How to teach problem solving this worksheet style. Math teachers don’t have a problem with worksheets. You know who has a problem with worksheets? Computer scientists. Oh my God, paper. And you would not believe how painful this is. Places where our transfer story fails, every single case has been a computer science teacher where we said, “What went wrong? Did you use the worksheets?” “Oh no. I didn’t use the worksheets.” “Why not?” “Well, everything else looked great, but the worksheets clearly didn’t belong in there in the classroom, so I took the worksheets out.” “The worksheets are what gave you the transfer.” “Oh, oops.” The idea of bringing paper … and they’re like, “Can you give us a PDF version?” It’s worse in PDF. It’s better on paper. You can’t tell computer scientists that. But if you’re not a computer scientist, it’s not a problem. It works fine.

And finally, you got to solve problems that people actually have, not the problems that are fun for you to solve. So part of that, what that means is, no administrative detail is too small. If you want to really help a school, you’ve got to actually engage with their problems. When they come to you and say, “I have only one hour of internet a week. What can you do for me?” If you say, “Well, can you go and, I don’t know, buy some internet,” you’re not solving their problem, you’re just trying to push your technology.

So I’ve thrown a lot of different things at you. Let me try to summarize what I’ve done in terms of three converging triangles. Our goal is to achieve computer science education. I’ve thrown like nine different things at you, and I’m going to put them in some structure. The first is our goals. Our goals are equity, rigor and scale. The second is our means. Our means, our pedagogy, language and curriculum. And the third is our avenues. Our avenues are algebra, data science and physics. And we believe that by doing all of this, we will actually get to a better version of computer science than people who are going directly through the door of computer science itself.

Okay. That’s the end of my talk. This is Bootstrap. It’s bootstrapworld.org. If you’re an engineering person and you want to contribute, you can contribute by getting involved, hack on some stuff. If you’re an advocate, go talk to people about it. But now what I’d like to do is to take questions. Questions, comments, transfer. Yes.

Already a fan, full disclosure. Can we do some of this before high school?

Both Bootstrap, algebra and data science are being used extensively in middle schools.

For school that teach algebra in the eighth grade, which as you know sadly is not every school. That doesn’t totally fly. And we engage with the data science community which we love-

Bootstrap data science?

Bootstrap data science, which is amazing.

Yeah.

But I don’t know if it’s middle school in every school.

No, you’re absolutely right. We are dancing around that edge of middle and high.

Yes.

We are now consciously trying to design something called Bootstrap Junior where Bootstrap Junior will be explicitly targeted from day one for middle schools unambiguously so.

That will be extremely helpful.

Okay. We’re also building a block-based environment for it. And the block-based environment in particular, I know, typing issues. You don’t have to use it. Nobody’s forcing you to use it. Well, the other particular detail in the block-based environment, it’s being co-designed with a blind programmer.

No, that’s great.

So it is completely screen reader accessible. It is completely ARIA compliant, completely drilling that detail. We worked on it for a year and a half. It’s going to be another six months to a year before … Oh, anyone wants to get involved, that’s a project to get involved in. And anyone who wants to code, hack some JavaScript, that’s what you want to hack with us.

Okay. Yeah. Kenny, your hand goes up, hand goes down. The moment I mention JavaScript you’re like, “OCaml.” See engineers, there’s your problem. Okay. Sorry. I cannot make fun of Kenny. I can make fun of you, right?

Yes.

I can still make fun of you. Okay, good. So Bootstrap Junior is what you want. Talk to me about why data science is not working. Fine, I’d love to hear more about what we can do to make that more accessible. Okay. Questions. I saw a hand up over there somewhere, or maybe the hand went back down. Yes.

You said 43% of your students who were admitted [inaudible 00:48:24] all of Algebra 1 course.

That’s right. Because we have computer science courses using it too. They screw up everything, man. Telling you. Yeah. Next question. Good catch. It’s because the computer science courses that we don’t have, like 51% women. Any other questions? Yes.

Have you had success in getting parents involved at home that way that the lessons go in home life as well and then-

You know what, that’s a wonderful question. And I would say that the reason we’ve not pushed it is for the equity reason, because we could easily write up a whole bunch of materials. I mean, we’ve tons of material. We could easily write up a whole bunch of materials and say, “Hey, kids, practice this with your parents.” As you perfectly well know, there are parents who don’t have the resources literally to do it. I know they are single parents, they’re working extra hours, whatever, and we don’t want to disadvantage those students. And it’s a very delicate line. As I said, for me, this constant tension I live with is, in Bootstrap, we’re always trying to do lowest common … not lowest common denominator, but a common denominator.

And in my other half of my life, I get to hang out with people like Kenny and John and Kshitij–Brown students who are astounding. And I can do all these great things. I don’t want that dichotomy in my life. I’d like to take down as much of that as possible, but we always have to restrain ourselves. We’re always like excited about, oh, maybe we could do … Well, are you sure we can do it? And I’d love to talk offline about ideas or thoughts about that.

I’m just going to mention about the parents thing. I do some work in Brooklyn, and there was a study, Michael you could probably help me, a Google study from a couple of years ago. They said the interest in computer science is inversely correlated with income. In Brooklyn at least, when we go out to Bedford-Stuyvesant, Brownsville, parents are like coding code and we want computer science, break it off. When we’re in Park Slope or Carroll Gardens, they don’t want to use that. There’s too many places, no interest. I mean, not no interest but the interest, we see that same pattern in Brooklyn when we’re out in more economically disadvantaged communities. The interest in computer science in their parents is high.

Very interesting. Next.

I’m curious about teacher education for mathematics teachers. In the private school environment, many years ago, we integrated [inaudible 00:50:52], it was seventh grade math curriculum like 20 years ago, and it stuck without any processing. That was in the private school environment. So we didn’t have a lot of trouble getting the math teachers interested in making that part of what they taught. It wasn’t as [crosstalk 00:51:07].

Were you part of the Bank Street Efforts or?

I was at Bank Street [crosstalk 00:51:09].

Oh, wow. Awesome.

That curricula, which still exists is at [inaudible 00:51:14].

Yeah, Okay. Sure

And fascinating, but we were kind of doing something like that. Then it was a little bit separate, like it was on one day of the week, but the math teachers took ownership. And we began and wrote it and then they took it. So that was really interesting that it was happening then and totally makes sense. But there’s also been research, and this is somebody at Columbia, on how exposed students studies become math teachers or to various technology tools. Not yours [inaudible 00:51:43].

Yeah.

So is there somebody working there to make sure that people coming out certified as math teachers are as open as those teachers I worked with years ago in the project [crosstalk 00:51:57].

That’s a great question. So we’re actually Bootstrapping now a project to go to, not just in-service, but pre-service teachers and also go to MAT candidates and try to get this into MAT Curricula. So it’s already the case that Utah is doing this. I think both Utah State and BYU, which are the two main producers of teachers in Utah have already been doing Bootstrap with the pre-service teachers. We’re now starting to look at other states. We’re actually in the process of trying to get like a nationwide effort to see if we can get some of that going. So it’s something very actively on our mind.

And there’s a woman, [inaudible 00:52:36] is her first name at Columbia whose dissertation research is on preservice training in technological tools for mathematics educators.

That’s wonderful. If you could just give me that name again after the talk or email me or something, I can’t wait to read more about her stuff. Thank you so much. Other hands. I saw some other hands. Yeah, John.

What are you most worried about?

What am I most worried about? So taking on computing in the context of other curricula is a very dangerous thing. Because I have to convince skeptics, and I’m pointing at her as like, I want skepticism. We don’t grow without skepticism. But I have to convince a skeptic because when you go to a school and you say, “Hey, we’ve got this cool computer science thing.” “Oh, awesome, wonderful.” You say, “Well, you’ve got an intervention in your math.” I was like, “Maah.” Why? Because schools are graded on their math performance. Schools will literally lose funding if their math performance goes down. So we’re going in and saying, “We can make our math scores better.” But what they’re hearing is, you might make my math scores worse. So it’s a very tricky, very delicate thing. And so there’s always the danger that we’ve picked the wrong horse entirely.

If we want our curriculum to get into schools, we should have just gone the safe route which is computer science rather than touching like this really high stakes stuff. But safety’s … Who cares? But that is what I worry about, that we have not done enough to make a hardcore computer science curriculum. This is ironic. We’ve got computer science professors, Kathy and myself who keep doing math curricula instead of a computer science curriculum. So not only am I unqualified, I’m not even doing the thing that I am qualified for. Anyway. New hands. Yes.

So this was a really good job of talking about how you can get equity and scale and rigor at the first stage, but what’s next? How do you keep those things going?

That’s a good question. We actually have follow-up curricula that I didn’t talk about. For example, there’s something called Bootstrap:Reactive that builds on Bootstrap:Algebra and goes deeper into computer science and into data structures. And we are building out other extension curricula. So we are actively thinking about that question and trying to build out more. Something I need to point out is Bootstrap is actually like about four people. Two of whom actually also have full-time jobs as professors and … So we’re kind of strapped for resources and that’s-

Yeah. I’m not just asking about Bootstrap, just in general.

Oh, I don’t know. I can’t solve the general problem. I have enough trouble solving this problem. Sometimes I get called out to talk to people in other countries and I’m slightly jealous because there are unified Nashville curricula like computing at schools in the UK and things like that. We can’t even get cities to agree on a curriculum. We can get neighboring schools to agree on a curriculum in the US. In the short run, I think that’s terrible. In the long run, like with many things about America, I think that’s actually kind of cool because it means all these experiments can bloom and we can try different things in different places and see what works. The problem is, just because we find something that works doesn’t mean anyone will adopt it. That’s the downside of these experiments. So we don’t have any centralization. So it’s almost like I have to unask a question. I saw a hand up over there, I thought. Hand’s gone down. Yes, back there.

This is sort of related, how long has Bootstrap been around, and do you guys have any kind of tracking longitudinal of like what students do you have?

That’s a great question. So two different questions. How long has Bootstrap been around? We started as afterschool in about 2007. We dropped the afterschool in 2012. I mean, all the materials are still amenable and there are some afterschool people who still use it, it’s just not something we wake up in the morning excited about for the reasons I talked about. We’re really focused on in-school, and that’s been about four years at this point. Four to five years. Longitudinal is very difficult because as a teacher, I don’t know if you’re a teacher, but teachers will tell you, “Schools do a good job of protecting student privacy.” What that means is it’s extremely hard to get data, and we as academics take great pains to not track without telling people what we’re doing. We’ve had school districts look at our tracking and tell us that we’re like best of breed.

Pirate, I would like nothing more than to log every single thing you do in my programming environment. We don’t. I could do tons of great research, but we think that’s just the wrong thing to do and we want to be good guys. So it’s difficult to get longitudinal data. We have done some studies. We get a little bit of longitudinal data sometimes, but we could have a whole conversation about why it’s just difficult. That’s kind of a cop-out answer. Basically the short answer is no, but the long answer is for privacy reasons, it’s just hard to get the data. The schools have it, but we can’t always get it and sometimes the schools just don’t have it. Yes.

Do you consider proofs as a avenue for computation?

I’m actually very interested in the question of how to teach proof. We’re doing a lot of research on that question at the collegiate level. And what I want to do is I want to nail it at the collegiate level before I start trying to take it to the high school level. I can tell you later, we’re doing, teach this great course at Brown called Logic for Systems which is … Oh, yeah. Hey, there you go. You TA?

First [crosstalk 00:57:54].

First class. There you go. Thank you. It’s now the second largest class after machine learning. A logic course.

Oh, my [crosstalk 00:58:01].

Yeah, exactly. Rock and roll. So we’re going to figure it out. The other question.

Yes. I want to follow up on the question about pre-service and in-service.

Yeah.

The place where I do most of my work is the straddling of pre-service and in-service. I’ve worked in school districts. I’ve worked in teacher training institutions, and from a strategic point of view, trying to figure out how to penetrate the system is really a very, very difficult problem I’m sure that you’re wrestling with every day. The most difficult of these problems is that we have thinking in gross generalizations. Elementary schools’ teachers do not know a great deal about mathematics, this is why they go into elementary education, because they can avoid mathematics. You only just take one course in mathematics.

Some, maybe yeah.

It seems to me that the whole math problem revolves around students who are between the ages of about 11 and 14, and the people who teach them and what they have to go through to become teachers.

So it’s a complex situation. There’s lots of variables. Look, one of the things that makes algebra complicated, algebra shows up at the same time as hormones show up. What are you going to do about it? So I’m not willing to say there’s a single determinant variable here, but we’re trying to tackle as many variables as we can. So the pre-service angle, the motivation angle, the retention angle, they’re all of interest to us. I certainly don’t know enough and don’t have the confidence to say the problem is…, but I always admire people who can. Yaron.

So this is sort of a useless question, but you’re … The beauty of Bootstrap is it really engages in the kind of horrible details [inaudible 01:00:13]. It thinks about, what is the real administrative and human governmental structures there, and hacks that system and figures out a way to hook into it. But we don’t choose to teach mathematics by integrating it into history, we decided to make mathematics its own discipline. And cubic specialized separated out times. And I’m just a little curious, if you had a freer your hand [inaudible 01:00:35], and you could choose how to structure it, would you make computer science more independent [crosstalk 01:00:42] make mathematics less independent?

I will give you a more modest dream than a completely free hand. My dream is that we get teachers from science, statistics, in IT, whatever, computer science, all of these subjects training together and all of them teaching the same kind of material so that the students see the same thing from four different angles so it finally completely sinks in. And it becomes not just a thing they learned, but an actual weapon they can wield. I would love it, and we love it when schools will send teachers from across disciplines. And there are some schools here that have done that, and we love that when that happens. So it’s a more modest dream, but I think it’s a much more practical dream of trying to get these teachers across the disciplines because here’s the other thing, if a school sends one teacher, that teacher is isolated. The moment a school sends two teachers, two is not twice one. Two is infinitely better than one because the one teacher has zero other people to talk to and that’s a denominator of zero. That’s where infinity comes from.

So the moment they send two teachers that teacher the social stuff, the technical stuff, they say, “I ran into this bug.” And at least the other teacher can say, “I can’t figure it out as well.” And that’s empowering. You’ve had this problem. You go in a computer lab and you walk down the hallways like, “I can’t figure out this OCaml typer.” It’s like, “Don’t worry. Nobody can. [Zavier 01:02:16] can’t either.” That’s empowering. So we love it when we can even get two teachers from a school. But if we can get teachers from across disciplines, then we can really change education. We can embed computing into everything and lift all of the boats. They’ll actually become better mathematicians, better social scientists, better computer scientists all at once. That’s where I want to go. Yeah.

[inaudible 01:02:38] but do you see this as a long-term approach because I could go and I could say, “Hey, I could teach chemistry, calculus and physics, and I could overlay a whole bunch of stuff if I wanted to.”

Yeah.

But really the world that we’re moving towards, computer science is becoming more and more important everyday, I think. Do you see this as a long-term approach or a long-term [crosstalk 01:03:05].

So I don’t view this as an either or. So the question is about, as computing becomes more important, and in some sense I’ll rephrase your question and tell me if I’ve done it in justice in doing so. It’s like, all this integrated stuff is okay in the early days, but eventually CS is going to become its own thing. And shouldn’t we just have CS courses. I think that’s not an either or. Sure we can have CS courses, but CS, if they don’t know how to apply it, then the dream of … The sales pitch is, CS changes every discipline. But if the student doesn’t know how to apply computing in any other discipline, it hasn’t changed anything. If all they know is how to write for loops, they haven’t actually learned how to do anything at all.

So taking away the siloed approach and saying computing is explicitly going to be embedded in these subjects actually enhances both. So if somebody wants to also teach a CS class, go right ahead. But actually I didn’t mention the C word, but I’m actually planning out a calculus curriculum as well. I’ve actually got a calculus curriculum in the works because that’s a subject to me. I was 30 when I finally understood calculus, and I had a math degree.

Okay. When I really understood, like I really, really understood calculus. My aha moment when I was studying temporal logic and I finally said, “Oh shit, they’re the same thing.” And then it all made sense. And now I’d like to go back and take that insight.

Here’s what I learned from the physics curriculum. High school physics is the science of things that have closed-form solutions. Real physics has no closed-from solutions. You go to a physicist and say, “Closed-form …” They’re like, “Hehehe, here, let me tell you something.” So high school physics has been warped into the thing for which you can give it T and it gives you an answer, the deterministic universe. Even things that are deterministic you can’t solve for. ODEs are the fundamental way. You know ODEs, PDEs, whatever. Differential equations are a fundamental modeling technique because as long as we can talk about the current state of the world and its impact on that state to produce the next state, you can describe all the physical systems you want. And there are many physical systems you can only describe that way. Well, that’s a calculus perspective.

Turns out those students in middle school who are writing games in algebra in Bootstrap are actually writing difference equations. This isn’t being recorded, right? Don’t tell anyone. They’re actually writing difference equations. So what is the behavior of a game over time? It’s the integral. So we’ve already got all this stuff in there. It’s all in there. We’re the sleeper cell, we’ll get there eventually. So if we can do things like that, we can change everything. There’s no need to worry about … Computer science courses, they’ll be the boring stuff. This will be where the excitement is. Yeah.

Do the insights that you have gained from studying how to teach these ideas at the high school level have any implications for the teaching computer science at the college level?

I can’t even tell those apart anymore because yes, I think thinking about the question … What I learned from Bootstrap, if there’s one thing I learned from Bootstrap, the thing I learned the most, it’s the T word; transfer. And the more I look at computer science now within computer science … I mean, at the college level I work inside the discipline. I see transfer problems all over the place. For example, when we teach Big O calculations to students, we teach them about algorithms. We teach them Big O. We teach them programming. Can a student actually take their Big O knowledge and apply it to change a program? That is a transfer question. And you know what? I think many of them actually can’t. And they can’t because all the three steps I talked about for transfer, none of those three steps are present in our curricula.

So there’s certainly this very deep meta knowledge, which is the meta knowledge of transfer. And I’m seeing transfer problems all across inside computing curricula as well, and trying to take those skills and apply them here. Absolutely, I think so. Yeah.

When you were showing the three courses, I know you did most of this on algebra.

Yeah.

But the physics one was actually very interesting to me because I feel like you can simulate the things that you’re trying to do. It’s kind of like a lab, but you can adjust parameters and really distinguish [crosstalk 01:07:25].

Absolutely. The downside is, it’s very easy for the computer to lie. The physical world, it turns out doesn’t obey what your computer wants, or it does? And so it’s a really interesting … That’s absolutely true. So in fact, this was Papert’s original view. So I’m actually not the world’s biggest Papert fan. I’m not a constructivist, unfortunately. And part of the problem there is because there’s the model and there’s reality, in algebra, we can make the model exactly correspond with reality. In calculus, we can pretty much do it. In physics…

This is one of the reasons I disliked teaching robotics first. It’s hard enough when you have hard, correct abstractions. Tell a robot to turn 90 degrees and turn back 90 degrees, which way is it facing? God only knows. Now, actually that’s not entirely true. But the moment you want to understand the answer to that question, I need to tell you about probability distributions. Now, it’s a great way to motivate probability distributions if that’s what I want to teach you. But if it’s not what I want to teach you, I don’t have a choice. I still have to teach you about probability distributions. So that’s where the stuff gets really tricky. But your insight is right, but the problem is the practice.

Let’s take one or two more questions.

Go.

What properties or qualities do you reserve for your Bootstrap students that you miss the most when you’re [inaudible 01:08:46] students?

Oh, I’m not going to answer that question in public. No. Okay. Here’s the easy answer, I don’t have any Bootstrap students. The teachers have Bootstrap students, so I can’t answer. No, I’m not going to answer. What a set up, man. Yeah.

Now, if Bootstrap takes off and everyone sees the value of Bootstrap, how do you see the organization growing?

I would like to hire a fourth person. No, it’s actually a little painful. I mean, right now the biggest problem is that Kathy and I and our professors, we have jobs and we do paperwork most of our day. I mentioned this to someone, the biggest thing that made a difference to my life was buying a phone. I spend half my day on the telephone. I hate telephones, but I spend half my day on telephones and Kathy and I shuffle paperwork. Anyone here from New York Department of Ed? I shouldn’t say this here. But that’s what it takes. So it’d be nice if I had the financial resources to hire somebody to deal with departments of education in various places. That’s what we need, administrative help.

Okay. Good. I think that’s a good note. I am going to stick around and answer every question that anybody has. I thank you all for coming. And I’d like to say hello to those of you who I haven’t met before. Thanks so much for coming and thanks for hosting. Yaron.