Exercism Teams... Coming soon?

Exercism
1

Hi everyone.

I’m planning on launching a very lightweight version of Exercism Teams. The idea is that people (friends, colleagues, teacher/students, etc) can create a team to track each others’ progress.

I’d like to understand what features people would consider essential to pay for. I’m thinking the price will be $10/team-member/month, potentially with a PPP option (based on the location of the team-member?).

This is my high-level spec. (In this scenarios, there are three people wanting to use Teams together. Nicole, the teacher. And Aron and Erik her students.)

Team Creation/Invites

I don’t really want the hassle of dealing with matching email addresses and things, as most people sign in via oauth which often leads to an abnormal email address on the account.

  • Nicole can create a Team
  • Nicole can provide a secret team link to anyone they want to join.
  • Aron can use the link to request to join.
  • Nicole approves the request.

Team Management

  • Nicole can remove anyone (except themselves) from their team.
  • Nicole can make anyone else an admin, who then have the same powers as her.

Team Member Settings

  • Aron can leave the team whenever he wants.
  • Aron can choose who can see his unpublished iterations (admins/everyone/no-one). Defaults to “admins”.

Team Dashboard

  • An activity feed shows all activity (exercise status changes) by all members of the team.
  • Anyone can select an exercise and see a page with their team-mates solutions to it

Admin Dashboard

  • Nicole can select an exercise and see the status of all team members (started, published, completed, etc), and see their iterations, even if they’re not published (pending the above setting)

What other features would you like to see? Please clearly mark your features as “essential” or “nice to have”, and tell me what your use-case is. Thanks!

Also, if you’d definitely use this, tell me and it will encourage me to definitely do it! :slight_smile:

6 Likes
2

I think that when you make very large teams it can become expensive, in those cases it is better to have plans by number of people, something like:

Individual plan (per person)
Team plan of 6
Bigguer team plan of 15
Enterprise plan (large companies or Education places)

Those who could benefit a lot from this and could pay for it would be companies, many times they have people on the bench studying technologies and with something like this they could track their progress

I see it more for educational places and companies

1 Like
3

By default, published solutions have comments enabled. Leaders will want to encourage members to engage in discussions about other’s code.

When you did the “gohort”, were there any particular features that were essential/missing?

3 Likes
4

Nice to Have far in the future: Would a long term future possibility be a custom track? I could see an “onboarding track” to get up to speed with a companies custom libraries/tooling or specific domain knowledge relevant to the team being extremely useful.
For context I work at Coder (a cloud development environment provider), so I’m picturing opening a workspace with a button click and solving exercises with access to internal libraries and tooling in your own editor.

2 Likes
5

We would be interested in learning more as we are starting an internal mentorship program and would be nice to have a mentee do a challenge assigned by mentor and then mentor could review without the mentee needing to have it on some “public” profile.

Would also be interested in doing internal competitions on challenges…

1 Like
6

The two situations I’d consider paying for this would be:

  1. As a teacher where I’m running a course and using Exercism for the class exercises
  2. As a manager or tech lead onboarding new employees

I’m not sure if this will match your vision… In both of these situations I’d really care about the ease of associating and disassociating Exercism users as “members.” I’d really like to be able to put start/end dates for membership. Especially an end date so I can stop paying after the class is over, or the onboarding period is over.

I’d still like to be able to see results from that time period for whatever language I was trying to track for grading/assessment, but probably not care as much about any of their other activity before/after

2 Likes
7

A really nice to have feature would be to allow team members to request mentoring, but only among team members (or perhaps only by admins). This way, not only can you track each other’s progress, but you can review others’ code as well. In the teacher-students scenario, that sounds useful.

4 Likes
8

It could also be useful when adopting a new technology (e.g. switching to TypeScript).

1 Like
9

In addition, admins may “assign” mentors from the team for certain exercises or per exercise. This is extremely valuable in situations, where teams are new to giving / getting feedback to / from others about code and “requesting feedback” is not likely.

I also could see a “live coding” or “live mentoring” feature for teams. Something to actively edit and discuss the code as a team and maybe continue separately later. Such Pair or Mob Programming can be achieved using other (external) tooling, but integration and ease of use within the Exercism online editor environment seems very appealing to me.

3 Likes
10

This is a cool idea and I’m always excited for new ways for Exercism to monetize and be sustainable. That said, I think it’s worth focusing more on what problems you’d be solving in exchange for their money; “track each others’ progress” seems a little shallow.

I think there’s broadly 2 groups who would use a paid Exercism team:

  1. Companies (probably tech, but not necessarily)
  2. Schools

I don’t imagine friends would go for it since the free version is already great and it’s worth a little hassle (checking someone’s profile manually) to not have to pay.

Companies

I work for a big-but-not-ginormous tech company. I’m not totally sure if we’d go for something like this at work, but I think it’s possible if positioned correctly. The company encourages “professional development” and we’ve got a paid O’Reilly plan we can access on a limited basis. But, we’re also there to create value for our customers, so actual work always comes first.

We do expect new hires to be proficient in our stack (Ruby, Java, TypeScript, etc), but they can already do that using the free Exercism offerings. In that sense, you’re sort of competing against yourself. I don’t think our managers would get any value from seeing their employees’ track progress. They might look at our public profile occasionally, but all they ultimately care about is our output (not our progress on a specific track).

That said, I think Exercism could really shine with custom private tracks. We’ve got a lot of propriety libraries we use internally that have a learning curve. Today, we solve this with an “Engineering 101” course for new hires (that I teach periodically). They make basic common changes to our codebase, like adding a new API endpoint, adding a new database model, etc. It helps them get a feel for our tools and ecosystem. But, they only do half a day on each topic, so there’s a ton of information that they have to learn later in docs. If we could make custom tracks with stub versions of our actual packages, we could introduce many more concepts and tasks interactively, which would work. I could see a lot of value in that. But, with access to proprietary code also comes much more stringent privacy and security requirements on our end.

Team membership wise, everything is tied to our corporate email addresses (including new, corp-only GitHub accounts). So if the company was paying for seats on an Exercism team, they’d definitely want the email-allowlist vs “any Exercism account can click this link to join”. If you’re targeting companies, SSO is also a very common feature in this segment. Triply so if there’s anything non-public available to accounts in that team (if you go the route of custom tracks mentioned upthread).

Alternatively, you could allow paying customers to self-host the whole platform. That would simplify some things (access controls) and make other things harder (tech support you don’t want to be doing). It would also simplify billing, since you could charge a (high) subscription fee and not worry about the mechanics of per user per month pricing (which has a lot of edge cases). This is what GitHub does.

School

I haven’t been in school recently, but I’d imagine their focus is more around controlling costs and generating graded reports.

A teacher would only want to pay for each student for the duration of the class (a semester, or ~ 4 months) and would need to be able to lower their bill if students drop from the class early. Also, I’m not sure that admins should count against a team’s head count. If you’re teaching 3 classes of 10 students each, I’d expect to pay for 30 students, not 3 teams of 11 members (10 students + 1 admin).

In a use case where a specific set of 3 exercises are this week’s homework, I’d imagine the teacher would want to be able to:

  • generate a report of the whole class to see how far each student got so they can be graded
  • ensure all submissions happened by a deadline (and maybe restrict submissions between a start and end timestamp so students don’t run ahead in the work)
  • check for similarity between student solutions to catch cheating

There are also privacy concerns with comments & kids (at least in the US, laws might be different in other countries). I’d imagine that teachers wouldn’t want it to be possible for internet randos to comment (or maybe even see) student solutions.

Lastly, you’d probably want to change how community solutions work on a team. You might:

  • restrict access to (or hide entirely?) public community solutions before a student has submitted a passing iteration. They could still see them in an incognito window (since they’re public anyway), but adding a little friction to get to the homework answer is probably good.
  • have a “class solutions” section to show community solutions, but only from members of the class
  • Support for featuring solutions to members of the class, so a teacher could showcase their own as an example (or a particularly good student solution)

It’s probably worth getting in touch with some actual teachers to learn more about their needs. Teachers who participate in the Microsoft Teals program might be place to start.

Conclusion

I think a team offering is a really promising direction for Exercism as a whole! It might be worth focusing on a single use case (companies, schools, or else entirely) rather than going for a generic “teams” offering, at least to start. Needs in this space are complex (and often non-obvious) so the more you can learn about your target demographic and cater to their needs, the better a chance at success you’ll have. It will also help on pricing, since companies and schools have very different budgets and willingness to pay for this sort of thing.

Happy to expand or chat more about any of the above as well, if that’s helpful.

2 Likes
11

Thanks everyone. Really enjoying everyone’s thoughts.

@xavdid - brilliant feedback, thanks. I thought I’d colour in a couple of points for you (and everyone else) to know where my thinking comes from:

  1. I’d say once a week I get an email asking if there’s a way to for teachers to track what their students are doing and if they could pay for that. We’re used in schools/unis/bootcamps globally and the main pain point, as far as I can tell, is visibility on progress for teachers.
  2. We had “teams” in Exercism v1 (for free). About 30% of all submissions were to a team at that stage, rather than individuals doing stuff. At the time, most of those were friends.

The idea of focussing on one group first is probably a very wise one, but I’d also sort of like to put something out that I can make in a week (which was basically my proposal above) and then see what features people actively ask for.

My sense is also that any companies >20 devs is probably not going to happen, as they require security, conversations, and other stuff more enterprise things (which I’m not interested in doing atm) and also I sense that most companies that have >20 devs already have some sort of O’Reilly or similar thing in play.

I guess my question is, would you use it for your specific team? Like if it cost $70 for you and your 6 teammates (guessing at your team size - the average is 3-8 I think)?

I’ll tackle your other comments along with everyone else’s once people have all had a chance to add thoughts, and I’ve had time to digest, but I thought that might be useful for future comments from people as an insight into my base position. Thanks again! :slight_smile:

1 Like
12

I’d like to preface this post by saying that below are my opinions as someone who likes Exercism and would see myself as a “teacher” in the Teams product. I know very little about running a business or how to make a product profitable, so people are free to point out or ignore if something I said doesn’t make sense.

At work, I try to level up the knowledge of less experienced developers of my team in areas they want to improve. The way I currently do this is by scheduling sessions with them, where for each session there’s a particular topic we’ll talk about. Sometimes the topics are suggested by me, other times I let them suggest something they want to know more about.

I recommended Exercism to some of them, and I could see myself using Teams.

Pricing

I think 10$/month/user by default might be a bit too much. Such pricing makes sense for big enterprises that are absolutely convinced Exercism will help them, but before we get there, we could have more generous plans.

I currently suggested Exercism to 2 of my teammates. If we made a team, that would cost us 30$/month. At my current company we would have 3 options:

  1. Pay from our own money
  2. Pay a year of the subscription and deduct that from the yearly self-development fund of each team member
  3. Have the company recognize Exercism as part of their global leaning strategy and employees could request licenses for users/teams without without paying anything themselves or have anything taken from any of their budgets.

For option 1, 30$/month, would be a lot. I wouldn’t feel comfortable paying that from my own money, specially because my teammates are still trying out Exercism, and might want to quit at any moment.

Option 2 is more reasonable, but that would put our little team of 3 at 120$/user/year, which is significant investment of our self-development fund we have for the year. So we would really have to think about it.

As a “teacher” of these small teams and someone who would most likely have pay upfront for the whole year, with the size of my team varying, I would be more comfortable with 10$/month (120$/year) but I could have a team single team with up to 10 or 5 elements and I could bring new elements and remove inactive ones more freely.

Which brings us to option 3. The way I see it, the more these small teams of people exist in a company, the easier it is to convince management of the company they should consider using Exercism as a part of their global learning strategy and convince them to pay the big bucks, like 10$/month/user.

Although, if a company is big, I’m wondering if it wouldn’t make more sense for the pricing to be $/team/month, instead of $/user/month. Maybe with pricing per team, the max number of elements could be fixed at a value, and have higher tiers for teams with more max members. As a teacher, I would see myself requesting a license to my company for a team of Exercism Teams that I would manage. Not sure how the flow would work be for a team with pricing per user, assuming option 3. Should each employee request an user for themselves and then the teacher would create a Team where the other users could join?

Features

Features I’d like to see in Teams:

  • A teacher dashboard to track the student’s progress would be great indeed! I’d like to be able to see at a glance which tracks each team member is doing and their progress. But also be able to drill down into the exercises they made in each track.
  • An “activity” page, where all people in the team could get a feed of the most recent events of the team. Someone completing, or request mentoring for an exercise could be an event. I’m thinking something like the “Activity” page of Github, where you can see the most relevant actions people you are following are doing on Github.
  • A team should be able to have multiple teachers, and students should have an easy option to request mentoring just to teachers of the team.
  • Milestones. In companies I’ve worked with, most people have some kind “self development plan”, where personal goals are set for the year. Learning can often be one goal, but more often than not you are required to be specific. “Make a project in Rust” is a more specific objective than “learn Rust” (what does learning rust even mean? when do we know the objective is completed?). I can see Exercism Teams as something that could be part of those self-development plans. But would be great if Exercism itself set some kind of milestones, to allow people to set more specific goals than just “learn via Exercism”. A milestone can be as simple as “do X exercises of track Y”. Potentially more interesting , milestones could also be groupings of exercises related together, like “do all the exercises on the X track related to topic Y”. The idea for these milestones is that students can set specific goals for themselves that they and their managers (and their teachers!) can track. Maybe teachers could also set custom goals with grouping of exercises they see fit? As a social feature, it might be cool to have a set of “community goals” where teachers from different teams could share their custom goals.
  • Remove limits of the Exercism platform. If someone is paying for exercism, seems fair that the limits to the number of mentoring slots and test runner timeouts should be removed or increased a lot.
  • Have access to insider features, like the ChatGPT integration.
1 Like
13

I also think this could be a huge factor for companies to decide to use Exercism Teams, for sure.

14

That’s super useful context! I had no idea a version of teams existed before. It’s also really interesting that you get so much email from teachers directly. That sounds like a good place to start/focus.

Getting something out to start is also reasonable, though you want your first impression to be good. If you start too small, you run the risk of potential users seeing it doesn’t do what they want and never updating that opinion. Definitely a balance :sweat_smile:

As for my team specifically, I think it would be a pretty hard sell without a lot of new functionality. My team actually works across 7 languages, so a tool that help us improve at non-primary ones is right up our alley. But the expectation has been that we learn that on our own as needed, not as big team exercise (on company time).

I’d imagine the conversation goes something like:

  • David: Hey, I’d like to try out a new language-learning tool with the team. It’ll help us improve in our non-primary languages
  • Manager: That sounds neat! But we have a lot of pressing deadlines. I’d be worried about taking time away from those
  • D: That’s fair, but learning is sort of like technical debt - the more we invest now, the faster we can compete those deadlines in the near future
  • M: That’s fair I guess. What sort of time commitment would it be?
  • D: As much or as little as we wanted. I figured ~ an hour a week. Won’t hurt any deadlines now, but will be enough to drive individual improvements
  • M: Sure, I’m down to try that for a few months! How will you measure the success of the project?
  • D: Probably informal surveys to track how much participants benefit from their learning time? The site lets us track individual progress in a language track, but it’s hard to translate that directly into work impact. There’s also a paid version which will let as a manger see our code
  • M: Can I see your code without paying?
  • D: Yes, but it won’t be grouped as nicely and you can only see what we explicitly publish
  • M: That’s probably fine, I’m not concerned with the text of your solutions - I just want to see business impact. You can spend your education budget on this if you’d like, but I don’t need to be directly involved.

From there, I’d probably recruit people to try Exercism, but I’m not sure that we’d form a team. I’d probably put an internal google doc together for a “monthly review” for each person to talk about what language they did, what they learned etc. The Exercism experience itself would be largely self-directed (like it is today). We could manually request mentorship from each other if it came up. Seeing a group activity thread would be neat, but not worth the cost, IMO.

All that said, I’m happy to pitch this to my manager to get their actual feedback. I’m putting a lot of words in their mouth right now. I’d be curious to hear what would make this worth team budget from their perspective.

15

I’ll try to describe the use cases for Exercism at my company we can imagine / we tried out. We are a small company, with 2 apprentices and 5 programmers.

Use case 1: Get the apprentices to learn the languages we use

In school, apprentices learn a set of programming languages and the basic mechanics of programming. Unfortunately they only learn very little about the languages we use daily. So we tried to make them use Exercism to learn the basics.

It didn’t work as required, because they simply ignored the opportunity. They chose a different course, that promised building a “real life project”. They currently don’t make any progress with that course, because, well, they are young and such project is hard work. But they insist, that they want to do “real work”, not “toy problems” for learning.

Without insight into their progress it is impossible to put the required pressure on them. So we designed an “Exercism workspace” (a git repository in our company hosted GitLab + readily prepared Cloud Workspace) for us to use Exercism pool of exercises in our company’s workflow / working environment. I currently am in discussion with management how to enforce the use, as up to now we never enforced learning (“make accountable to progress in learning”?).

Use case 2: Introduce a new programming language

We currently use a frontend and a backend programming language for our projects. And we have frontend and backend specialists. This worked fine for some years, but we now feel the need to make everybody more “fullstack”. And we have bigger projects, that need enhanced skills and a wider variety of languages.

So we encouraged using Exercism for improving language skills and trying out new languages. For the languages we already use, we have our companies working environment. But for new ones we wanted to use the exercism online environment.

Currently I am the only one actively using Exercism. My co-workers use their learning time for trying out new frameworks or courses “with real life projects”, with about the same results as for our apprentices. I currently am in discussion with management how to enforce the learning, as up to now we never enforced learning.

Use case 3: Onboarding co-workers

While selecting a new hire includes some technical skill assessment, the new person ususally has to learn a lot about the languages they state to have learned. And about our working environment and workflows. It took a bigger part of a co-workers working hours per week to guide and track progress here.

To lower that effort, we combined both learnings in our “Exercism workspace” (see above). While doing the easy-levelled Exercism exercises, the learning focus is on our company’s workflow / working environment. And we can actually see the level of skill in the language during the more advance exercises.

I have been thinking about the “custom track” idea stated above also. But that is the hard work part of a training environment. Meaning: That is the part we would want to buy, not make ourselves.

Essentials for our use cases

The missing features of Exercism for our use cases we built with externalising the personal progress and the mentoring in our GitLab:

  • Access any commit (“Run tests” in online editor) of a solution
  • Mentor / discuss using GitLab Merge Requests
  • Assign reviewrs / mentors per exercise
  • Build personal training plans using GitLab issues
  • Train working in our company’s working environment
  • Pair- / Mob-Programming using our company’s working environment

One thing that might be essential for “success” of Exercism Teams: More “technical real life” stories around the exercises. Seeing examples in e.g. open source projects of what is asked to program may help self motivation.

16

I don’t think I would be able to swing 10 per month on a continual basis, but using exercism as a kata tool for internal workshops feels like it could work to me, for a fee per workshop or similar

17

Most of my concrete thoughts/suggestions have already been raised by various replies already, so I’ll just add my voice and experience to the mix.

I’m in the process of spinning up an after-school math+coding program for motivated ~high-school age kids, and I think I would make use of an Exercism Teams-like product, for sure. The features I’d want the most have already been raised, so I’ll just add my vote to the following:

  • Some way to curate the experience for student accounts, have other teacher/students on the Team be given some sort of privileged placement in the mentorship queue
  • Some sort of way for the teacher to easily track progress for a class at a time, and even a public Team leaderboard to gamify the experience a bit
  • The ability for me to implement custom tracks (with some combination of existing problems on the site as well as my own; I’d be very willing to learn whatever the markup system is to implement) would be awesome. The focus of the class is more on algorithms/data structures/problem-solving rather than the programming languages themselves, so I’d love the ability to curate my own subject-matter-focused tracks.
  • My students are high-school age teenagers, not really “kids”, so while blocking them out from the existing social features of the site would be fine, it’s not really that urgent. I’m 100% sure the students are daily on places a thousand times more toxic than Exercism at its worst has ever been.

How much would I be willing to pay for it? The approximate price point (converted to USD) that I’m intending to charge is on the order of <$200/month, and I would certainly intend to bundle any required software like this into the packaged price, not require students to go out and purchase subscriptions on their own. So at the $10/month rate that is being floated here, that represents about 5% of total revenue. That’s high, but not completely out of the question; I’d give it serious consideration at that rate but it’s not a foregone conclusion. If there was some sort of multi-seat discount (because I’d regularly be bringing in new cohorts of 30-50 students every term), or some sort of educational/non-profit discount, that would sweeten the deal even further.

Really hoping these plans come to fruition! I agree that this seems like a promising avenue for Exercism to hit its funding goals finally. This site is really a godsend so anything that keeps it healthy long-term is something I’m very excited for.

3 Likes
18

The main features I’d love to see in a potential Teams product is mostly teacher/student based:

  • Assign exercises with due dates
  • Give feedback on student submissions
  • Autoscore assignments based on how many tasks were passed
  • Basic LMS/CMS integration with Google Classroom, for example
  • Custom assignments with custom notes

If I’m paying $10/user/month (Which would cost me nearly $1000 per year, well over my class budget), I think whitelabeling/theming the site should be possible as well. I teach at a Christian school, so some people find the name “Exercism” a little bit concerning, especially if we were going to pay for it and it ends up on our budget.