coreyja weekly - July 28th Update

Hey Team, how's it going? I hope everyone had a good week!

It's Friday again, so time for another coreyja weekly. This will be number three! Thanks to all the early birds who are subscribed to hear my ramblings each week.

Today I want to talk about schooling, college and the "right way" to learn computer science.

But first, a recap of what I've worked on this week!

On Friday morning, just as last weeks newsletter went out, I was producing a podcast! This is new to me but a ton of fun so far. Me and my wife Brandi, have a client that wanted help creating their digital brand, and had interest in recording their own podcast.

So last Friday we sat down and they recorded their first episode. It was cool to be on the "other side" and see the behind the scenes magic that goes into a podcast!

If you remember back to last weeks newsletter I was working on rmbg, a tool I started to try and remove the background from videos. Well this podcast was the reason I wanted to give that a shot! Our clients don't have a dedicated podcasting setup, because who does when they start out. And so we are working together to try and create a final podcast we can all be proud of. One of the partners was worried about their background, hence working on rmbg.

But back to the podcast! Me and Brandi are having a ton of fun becoming podcast producers. Brandi created an awesome logo for their podcast, and spent some time this week playing around to create a jingle intro and outdo. It's been some new learnings for both of us, and excited to continue and get some more episodes recorded! I'll share the clients we are working with and the podcast when it launches!

rmbg and RVM

Speaking of rmbg.... I think it might be headed for the coreyja project graveyard 🪦

And mostly because I did more research and found out I was building off of old tech! I was using u2net which was released in 2020, a whopping THREE years ago! Which is at least half an eternity in the AI world of recent years.

It seems like the latest and greatest is Robust Video Matting, or RVM, which was funded by ByteDance the company behind TikTok.

This solution is specifically designed for videos! And seems to fix one of the main issues I had with rmbg and u2net.

The big issue I had was since each frame is processed in isolation, there are objects that were sometimes part of the background and sometimes part of the foreground as detected by u2net. For example the head rest on my desk chair while I'm streaming. In one frame it might be part of the background, which would get blurred out by rmbg. But in the very next frame it might be considered part of the foreground, so appears without any blurring. It's not so much an issue that it was in either the foreground or the background. Either would be fine. The issue was that it would change back and forth, which made the video feel very choppy and not professional.

RVM as I understand it, works around this issue by NOT processing each from in isolation. Each frame is processed along side some information that was generated by the last frame. This allows the background removal to be more consistent frame to frame.

Orrr that's the idea with RVM! I haven't actually played with it yet. The models and code are available: https://github.com/PeterL1n/RobustVideoMatting and I tried to get a model working in Replicate without any luck so far... Just running into Python version issues, of course.

I was trying Replicate because I don't have a dedicated GPU that I can use and it seemed like an easy way to basically rent a GPU on demand. I'm going to keep poking at it soon, but so far haven't been able to install everything at the right version to get it working. Which is made more complicated by the fact that I can't test this locally since I don't have a GPU!

I'll likely try to run this on CPU only locally to see what kind of speeds I can get, before I invest more time into getting it running in some cloud environment. If you have any tips of know of a place where I can easily run RVM on a GPU without much setup, please reach out!

The last thing I wanted to write about that I did this week was try out a hackattic challenge!

I love programming challenges!

So finding a new site of them was exciting. They market themselves as "Hard, no nonsense Programming Challenges" so I was sold.

On stream on Wednesday (VOD: https://www.twitch.tv/videos/1882612527) I tried out the lowest point/difficulty challenge, "Help Me Unpack"

The basic idea of the challenge is that you are given a string of characters. These are the output of Base64 encoding a binary blob of data. And there are ints and floats contained in the binary blob that you need to pull out, and return as a JSON object. Nice and easy!

Except true to their name, I ran into some hard parts that threw me for a loop.

The first one had to do with alignment, and I think I have a good answer to it now! The problem statement said that one of the values was a short which is 2 bytes long. I decoded this short without any issues; grab the bytes and have rust interpret them as a u16, nice and simple. The issue was laying in wait though!

That short was only 2 bytes long, but the binary I was given wasn't "packed" as tightly as possible. Instead everything was aligned to 4 bytes! This meant the short that really only took up two bytes was padded with two bytes of zeros after it!

After realized I needed to discard these padding bits I was off and running into problem number two!

And that one stumped me for a bit!

See I was getting a float that was oddly similar to the right answer, but mine was way less precise than the server was telling my the answer was. And that felt off...

Floating point is weird... So was this just a rounding issue? I didn't want to accept that answer for a challenge that was about decoding some binary data. Floating point is weird, but the binary should represent the same floating point no matter which machine you run in on, that's the power of IEEE 754 standardization. After a good bit of back and worth with chat (Thanks Opusbopus for hanging out and helping me debug!) I came to my conclusion.

The solution had a bug.

And not only did I think there was a bug, I was pretty sure I knew what it was!

See I was supposed to be decoding a float or an 32bit floating point number. But the answer the server as giving me had the precision of a 64bit float! AND if I took my f32 in Rust and casted it to a f64, the answers lined up perfectly! Ah ha!

A quick browse through the hackattic FAQ showed me that the site was built in Django, and therefor Python. And Python only has a single float type, which is 64 bits. And the puzzle pieces finally fell into place!

My original f32 answer was right all along! The site was just accidentally using a 64 bit float instead of a 32 bit float like specified. And therefor the answer was way more percise and a 32bit float would allow.

I used my cast to get the 👍 on the challenge and then I sent the creator an email. He responded the next morning and confirmed my investigation and the bug! [Thanks for the quick response Jack!] Success!

While it was a bit frustrating on stream to not understand why my answer wasn't accepted at first, I think this outcome was way better in the end! I was forced to challenge myself on if I really knew how floating point and binary encodings worked. And we found a bug! And not only that, but we were able to determine the exact bug from the outside and that felt pretty cool!

This is a topic that I see come up every so often in different programming circles. And I was reminded of it on Wednesday while working on the binary encoding challenge. This isn't a new topic and I don't think I have a ton to add to the conversation, but its on my mind and I want to share!

Part of my problem with these conversations is that they usually don't start from a place on honest curiosity, or from a place of trying to help others. Instead I see them repeated as ways to gate keep others, and often mock other developers with different skills.

And I think thats all a bunch of crap. From many different angles.

I went to college for computer science. I have a bachelors degree with a triple major in Computer Science, Software Engineering and Mathematics. So I definitely learned in school about little endian and big endian numbers. And about the different standard sized of floats and ints and what not.

And those skills were useful to me this week! Except... Not in my job... Not in any programming I do for any real reason

They were helpful skills in a contrived challenge. That I did in my free time. The "fakest" of all programming.

Are there things I learned in school that I use more regularly? Probably.

But do I think I needed those four years of schooling to do the web dev I do day to day? Hell no, not even a little bit!

Did it make me a better programmer? Maybe, I'd be lying if I said I don't think it did. Knowing about endianness, and binary encodings and whatnot can be useful.

But did I need to learn it first? Before I'd been programming for awhile and ran into a problem with endianness? I don't know, I don't think so.

In my experience the best learnings come from solving a problem you have. Not a homework problem, but something in your life or work that you can't figure out. THIS is when a new learning is most likely to stick, because it was useful. It's easy to forget which direction the bytes flow in a little endian number after reading a text book. But run into a problem where you are reading the bytes in the wrong order. Which prompts you to learn about the differences between big and little endian. I think that's where the real learnings stick!

And there is another angle to this issue that rattled around in my head this week. And that's thinking about abstractions.

Programmers love abstractions! They've allowed us to build off the effort of all the programmers who came before us. Building bigger and better things!

Abstractions have to be built from the ground up. I can't have a desktop app without an operating system. And I can't have an operating system without a CPU. And you can't have a CPU without resistors or transistors.

But that that doesn't mean the abstractions should be taught from the ground up. And I think we sometimes forget that as programmers.

If someone is able to write an application on your operating system, without ever having to interact with the kernel directly, or having to read bits from CPU registers, we wouldn't call that a failure. We'd call that an amazing success!!

The whole point of abstractions is that you don't need to understand everything about the underlying systems. That's why we create abstractions!

So the next time you hear someone learning programming by opening a JS console and typing some characters in, without ever studying computer science, I think we as programmers need to frame that as a success of all the abstractions we've built!

If someone can jump right in and build something useful, all the coders that came before them and laid the ground work through abstractions have done an AMAZING job!

We shouldn't want newcomers to have to know all the pains of the past. Now of course they'll learn about the underlying systems eventually the more they continue to program. They'll find leaks in the abstractions, and be forced to dive into the underlying systems. But by this point they'll be in the right place and time to better learn about and appreciate the systems they are working on top of.