Intro

To whoever is curious enough to reach this post, welcome.

I’m trying to write this post so that it is useful to readers (i.e., those who’re interested in the Kernel Mentorship program).

To give you a sense of where I’m coming from, during the application process, I did some self-reflection and came up with this thought:

“With System Programming, I found myself reading textbooks on computer systems on my own - just like reading any other books. It is something I did not do while I was an undergrad. I’m not sure what this is, but I think I’m onto something.”

That’s how I knew I should apply for this program - and see how it goes.

These are the patches I sent upstream. Some have been applied, some are still under review. If you’ve never contributed to the Linux kernel before and somehow feel like having your patches on the public record is cool, I guess you are in the right place.

Accepted and Applied Patches

[PATCH] media: videobuf2: move vb2_querybuf() kernel-doc to header
[PATCH v3] media: imx219: add test patterns
[PATCH v2] media: v4l2-ctrls: default HEVC SPS chroma format to 4:2:0
[PATCH] media: hantro: Harden MPEG-2 control access against NULL
[PATCH v5 1/2] media: hantro: release runtime resources when device_run fails
[PATCH v5 2/2] media: hantro: use DEFINE_RUNTIME_DEV_PM_OPS
[PATCH] media: hantro: disable runtime PM before hardware teardown
[PATCH v1 1/2] KVM: x86: Reject reserved CR8 bits in KVM_SET_SREGS
[PATCH v1 2/2] KVM: selftests: Add CR8 reserved-bit checks to set_sregs_test
[PATCH] KVM: selftests: Fix VM leak in early return

Sent but not yet accepted

[PATCH] KVM: selftests: Fix reg_list and VM leak in get_reg_list test
[PATCH v2] KVM: selftests: Test partial KVM_CLEAR_DIRTY_LOG requests
[PATCH v2] KVM: selftests: Verify KVM_MSR_EXIT_REASON_INVAL on non-canonical FS_BASE write
[PATCH 1/5] KVM: selftests: Introduce a shared base for test memslots
[PATCH 2/5] KVM: selftests: Use the shared memslot base in memslot_perf_test
[PATCH 3/5] KVM: selftests: Use the shared memslot base in memstress
[PATCH 4/5] KVM: selftests: Reserve memslot 1 for the ucall MMIO address
[PATCH 5/5] KVM: selftests: Test protection of the ucall MMIO region

Background & Preparation

It is enormously helpful to have taken an Operating Systems class. I think any standard undergraduate-level course at any college will do. In case you haven’t taken one, there are YouTube lectures from the University of Wisconsin. The instructor is one of the authors of Operating Systems: Three Easy Pieces </u>.

After I was accepted, I read through the book Linux Kernel Programming by Kaiwan N Billimoria. Most likely, beginners like us will not contribute to the “core” kernel right away (like memory management, the scheduler, CPU architecture, etc.). Those areas do have a high barrier to entry and require a deep understanding of the system.

But reading that book helps you get a sense of where everything is and how your subsystem of choice interacts with the core kernel. It took me about 2 weeks to read - I wasn’t sweating every detail.

Another thing to note: the Linux kernel is a moving target, so trying to find “the book” that does the job is an impossible pursuit. I highly suggest that you get comfortable being uncomfortable: you have to learn as you go and read as needed. It never works for me to read the kernel documentation linearly, top to bottom. It only works when I know, even vaguely, what I’m looking for.

Nice quote

That was something I should have figured out earlier (I did try many times to read docs linearly, top to bottom). Eventually, I found that my problem was not unique. And the quote from Andrej Karpathy nicely summarizes what I should have done instead.

Experience (first half) - Media Device Driver (Camera & Video Decoder)

It is natural to feel overwhelmed: where should I start? And there is a constant inner voice saying that you’d better have 5 patches sent. The knee-jerk reaction is to ask, “Where are the TODOs?” I learned later on that this was the wrong question to ask.

Here’s what I discovered: choose a subsystem (or device) that interests you, and patch opportunities will arise naturally. A lot of the patch opportunities I found came from “code inspection”, i.e., reading the source code just to understand how things work and interact in Linux.

At first, I chose to work on Camera & ISP. The first two patches were by no means impressive. They’re here to illustrate my point: your patch opportunities will arise naturally.

[PATCH] media: videobuf2: move vb2_querybuf() kernel-doc to header
The first one is a documentation fix. While reading the source of the media (videobuf2) framework, I found that a function’s documentation had been left stranded and was not showing up on the kernel documentation website.

[PATCH v3] media: imx219: add test patterns
The second one simply came from me reading the IMX219 camera sensor datasheet and comparing it to the driver. That’s when I found some missing test pattern support.

There was no way my Kernel Mentorship experience was going to be just about 5 documentation fixes, but hey, this gave me some peace of mind (at worst, I think I can pull this off).

I soon found the camera work somewhat limited in scope. I spent a week going through the Raspberry PiSP front-end and back-end image capture and processing pipelines. And found a “potential” race condition bug.
[PATCH] media: raspberrypi: rp1-cfe: acquire state_lock in cfe_start_streaming()

However, my own conclusion was that device drivers like these tend to be maintained by the vendors themselves, and often they do have their own trees (true in the case of Raspberry Pi). I found that work less approachable for an outsider like me (totally understandable).

So I moved to the closely related video decoder work. I bought the Rock 5B board that comes with lots of media decoder chips.

Rock 5B board

As usual, I played with the device and read about what I encountered. I found that forcing myself to write things down is really helpful. It helps crystallize my understanding. I get it, we can now find this information at our fingertips (and probably in a more complete form). But that is not the point. The point is to slow down and understand things. Writing is a vehicle for that.

Diagram of stateless video decoder Diagram of stateless video decoder

My attempt to map how the source code interacts My attempt to map how the source code interacts

Example of me trying to quiz myself Example of me trying to quiz myself

I spent about 2 months learning about these video decoder drivers. By that time, the new semester had started and I had about 4-5 more patches in the kernel. One of them was really unexpected:

[PATCH] media: hantro: disable runtime PM before hardware teardown
This patch opportunity arose when I removed the driver module so that I could load my new version. It turned out that removing the module caused a runtime PM issue (system error). I ended up learning a few more things about power management in the Linux kernel.

I thought it could be a good idea to connect my kernel work with my coursework, i.e., hitting two birds with one stone. This semester I’m taking the Virtualization course. Luckily, the kernel has KVM (Kernel-based Virtual Machine) support. To say God loves me is an understatement.

Experience (second half) - KVM

I cannot overstate how much “peace of mind” matters in learning. As you can tell, by this time I had > 5 patches accepted, so that was a relief. Learning after this point felt more “pure” and less like patch hunting.

I should have known this earlier, but pairing school coursework with kernel contributions is a very good idea! I don’t think I spent that much time studying specifically for the exam, and the school programming assignment was like “just another day of learning KVM”. It feels so good. I highly recommend doing this if you can.

Anyway, back to KVM. KVM has a nice set of selftests in tools/testing/selftests/kvm that lets you explore how the pieces of the subsystem interact. The selftest code itself is a form of documentation on how the system should behave. Tracing through the calls and asking myself why is quite educational.

This is no secret: I asked the AI to list the 25 “most educational” KVM selftests, and I just went and studied them. Along the way, again, patch opportunities arose as I thought more about the test coverage. Sometimes they came from reading documentation and realizing a check was missing, like the one for CR8.

[PATCH v1 1/2] KVM: x86: Reject reserved CR8 bits in KVM_SET_SREGS
[PATCH v1 2/2] KVM: selftests: Add CR8 reserved-bit checks to set_sregs_test

As I’m writing this, I’m taking a bit of a pause to finish up the LKMP report and school coursework. I’ve made a mental note that I want to go deeper into KVM (outside of selftests), into parts of the core like VMX, VMCS, exit handling, and other low-level mechanisms.

Closing Thoughts

Kernel Mentorship has been a great experience. I mean, 500-1000 lines of code do not scare me anymore. And I’ve somehow developed an intuition while navigating code: “At what point am I going too deep down the call stack for what I’m trying to understand, and should I stop?”

And also, this is in no way a program with “hand-holding”. I think at the beginning of the program, Shuah said something like (paraphrasing), “In the kernel community, nobody tells anybody what to do.” If you take a step and get a bit lost, the mentors are there to help. But that step has to be initiated by you. One side effect is that I feel like I’ve developed some confidence that I could just go study something I have never seen before and figure it out on my own. That is kind of cool.