{"version":"https://jsonfeed.org/version/1","title":"anecdotal thoughts","home_page_url":"http://anecdotalthoughts.com/","feed_url":"http://anecdotalthoughts.com//feed.json","description":"anecdotal thoughts","icon":"favicon.ico","favicon":"","expired":false,"author":{"name":"Anas Mohamed","url":"http://anecdotalthoughts.com/"},"items":[{"id":"e308029dd923aaa7e3185a577d2dd5f911fbf928","title":"Profiling List Comprehension \u0026 Generators in Python","summary":"","content_text":" \u0026ldquo;We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.\u0026rdquo;\n\u0026ndash; Donald Knuth\nIf you already have an existing service, for the most part, Python\u0026rsquo;s benefits (easy to write \u0026amp; read, library ecosystem, etc.) out-weigh the benefits of optimizing for that 97% by switching to a compiled language. And really, for most services, Python is rarely the biggest bottleneck. I mean, most APIs needn\u0026rsquo;t be scalable to gazilion requests/sec.\nRegardless though, if your hot-paths include some sort of iteration over huge data, then Python\u0026rsquo;s shortcomings in speed, and its memory hungriness are hideously bare to witness. For many of these cases though, there a two simple best practices that can help alleviate those ills tremendously. Those are defaulting to list comprehension instead of loop + append combo, and returning generators instead of returning lists.\nBoth of these tools can yield a surprising amount of benefits for little to no extra effort. When I first read of these benefits in Effective Python, I thought this would be a nice chance to try out profiling Python code. For this, I used Python\u0026rsquo;s built-in cProfile for execution time and a neat library called memory-profiler for memory usage.\nCheck out this gist for the entire Python script I used.\nComprehend This Speedup # Let\u0026rsquo;s take the following example of a generic loop\ndef exhibitA(n): a = [] for i in range(2**n): if i%2 == 0: a.append(i) This code simply iterates over a range and creates a list containing only even numbers. Now lets run it and profile it.\n524292 function calls in 0.122 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.004 0.004 0.122 0.122 \u0026lt;string\u0026gt;:1(\u0026lt;module\u0026gt;) 1 0.096 0.096 0.118 0.118 comprehension_and_generators.py:5(exhibitA) 1 0.000 0.000 0.122 0.122 {built-in method builtins.exec} 524288 0.022 0.000 0.022 0.000 {method \u0026#39;append\u0026#39; of \u0026#39;list\u0026#39; objects} 1 0.000 0.000 0.000 0.000 {method \u0026#39;disable\u0026#39; of \u0026#39;_lsprof.Profiler\u0026#39; objects} Exhibit A: max=39.58 MiB, min=24.3 MiB, diff=15.27 MiB It takes approximately 0.122s to run and uses about about 15.27 MiB (wow, python is one chonky boi) when you basically remove miscellenous import modules. Alright that\u0026rsquo;s not too good, let\u0026rsquo;s what we can do about that.\nHere is the code re-written to use list comprehension\ndef exhibitA_comprehension(n): a = [i for i in range(2**n) if i%2 == 0] Succinct and to the point.\nAnd here is the profile\n5 function calls in 0.051 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.003 0.003 0.051 0.051 \u0026lt;string\u0026gt;:1(\u0026lt;module\u0026gt;) 1 0.000 0.000 0.048 0.048 comprehension_and_generators.py:11(exhibitA_comprehension) 1 0.048 0.048 0.048 0.048 comprehension_and_generators.py:12(\u0026lt;listcomp\u0026gt;) 1 0.000 0.000 0.051 0.051 {built-in method builtins.exec} 1 0.000 0.000 0.000 0.000 {method \u0026#39;disable\u0026#39; of \u0026#39;_lsprof.Profiler\u0026#39; objects} Exhibit A with list comprehension: max=37.75 MiB, min=24.31 MiB, diff=13.43 MiB Now we\u0026rsquo;re cooking with gas. Our execution time is down to 0.051s from 0.122s, a huge speedup factor of ~2.4x. Function calls are also reduced, by 5 orders of magnitude nevertheless. A nice bonus for debuggability.\nSince we are still creating the same list, our memory usage is essentially the same, at about 13.43 MiB for our function.\nSo, next time you catch yourself writing the same old for loop + append combo, see if it can be written using list comprehension instead. In most cases it can be, and personally I would argue it is also more readable as well, but that\u0026rsquo;s entirely subjective and is neither here nor there.\nYield or Else # Generators are a bit less clear cut in their use cases. Where they really shine is where memory is a bigger constraint than speed. For example when you are ingesting a whole lot of data which needs to be processed in some form. In these scenarios using a generator will avail your program from holding all that data in a in-memory list while they are waiting to be processed.\nHere is the same loop from the initial example, but in this case we need to carry out another filtering process after our list is created\ndef exhibitB(n): a = [] for i in range(2**n): if i%2 == 0: a.append(i) div_by_8_and_2 = len([x for x in a if x%8 == 0]) and here are our results\n524294 function calls in 0.140 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.005 0.005 0.140 0.140 \u0026lt;string\u0026gt;:1(\u0026lt;module\u0026gt;) 1 0.098 0.098 0.135 0.135 comprehension_and_generators.py:14(exhibitB) 1 0.017 0.017 0.017 0.017 comprehension_and_generators.py:20(\u0026lt;listcomp\u0026gt;) 1 0.000 0.000 0.140 0.140 {built-in method builtins.exec} 1 0.000 0.000 0.000 0.000 {built-in method builtins.len} 524288 0.021 0.000 0.021 0.000 {method \u0026#39;append\u0026#39; of \u0026#39;list\u0026#39; objects} 1 0.000 0.000 0.000 0.000 {method \u0026#39;disable\u0026#39; of \u0026#39;_lsprof.Profiler\u0026#39; objects} Exhibit B: max=40.91 MiB, min=24.91 MiB, diff=16.0 MiB Oh lord, that\u0026rsquo;s almost 16 MiB just to store 131,072 ints. Shouldn\u0026rsquo;t that be like 1MiB or so? What\u0026rsquo;s up here?\n*few tabs \u0026amp; getsizeof calls later*\nHuh, why are ints in Python 28 bytes minimum? that seems a bit excessive.\n*even more tabs later*\nWAIT, WHAT DO YOU MEAN EVERY INT IS AN ENTIRE OBJECT IN PYTHON? WHAT? WHY DO YOU NEED A REF COUNT FOR AN INT???\nPhew! alright, that seems like entire rabbit hole that I do not want to jump into right now. I mean, I\u0026rsquo;m sure there is good reasoning for this (better handling of large numbers maybe??), but that\u0026rsquo;s an investigation for another day.\nComing back to generators, let\u0026rsquo;s see how much of that memory we can get back.\nHere is the generator version\ndef _generator(n): for i in range(2**n): if i%2 == 0: yield i def exhibitB_generator(n): it = _generator(n) div_by_8_and_2 = len([x for x in it if x%8 == 0]) and our results\n524295 function calls in 0.124 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.124 0.124 \u0026lt;string\u0026gt;:1(\u0026lt;module\u0026gt;) 524289 0.072 0.000 0.072 0.000 comprehension_and_generators.py:22(_generator) 1 0.001 0.001 0.124 0.124 comprehension_and_generators.py:27(exhibitB_generator) 1 0.051 0.051 0.123 0.123 comprehension_and_generators.py:30(\u0026lt;listcomp\u0026gt;) 1 0.000 0.000 0.124 0.124 {built-in method builtins.exec} 1 0.000 0.000 0.000 0.000 {built-in method builtins.len} 1 0.000 0.000 0.000 0.000 {method \u0026#39;disable\u0026#39; of \u0026#39;_lsprof.Profiler\u0026#39; objects} Exhibit B with generator: max=28.61 MiB, min=24.97 MiB, diff=3.64 MiB From 16 MiB down to 3.64 MiB, that\u0026rsquo;s an astounding reduction of -440% in memory usage for arguably no extra code.\nYou might notice there is also a very slight speedup, from 0.140 to 0.124, but that is actually an artifact of this specific example. Depending on your case you could very easily end up with a noticeable slow down when using generators. Also techincally there is 1 more function call but that just the iterator asking if there is anything left and returning a no.\nLike I mentioned earlier, generators are much more case specific than list comprehension. If you know that your list might get large, or if you can start processing that list one by one, it is a huge boon to your memory usage. But if that\u0026rsquo;s not the case, and you need the entire list first before processing it, or your case is more constrained by speed rather than memory, then generators are not the right tool.\nReally that\u0026rsquo;s just a bunch of words for \u0026ldquo;it depends\u0026rdquo;. But as is often the case, that\u0026rsquo;s the answer for most programming questions.\nRegardless, it\u0026rsquo;s important to know the benefits \u0026amp; drawbacks for each tool you use. That\u0026rsquo;s what will help you optimize for that critical 3%.\nConclusions # List comprehension is signficantly faster than loop + append combo. Returning generators is substantially more memory efficient than return entire lists. Read Effective Python, it\u0026rsquo;s pretty good. That was fun. And I learned a lot about Python after delving a bit into some of the opcode in some tangents. Something I kinda wanna look into a bit more is how \u0026lt;listcomp\u0026gt; is so much more efficient than an append loop. It seems that\u0026rsquo;s thanks to its usage of the LIST_APPEND opcode instead of loading the append method, so that would be a good place to start. But that\u0026rsquo;s for some other time, that\u0026rsquo;s enough Python existentialism for one day.\n","content_html":"\u003cblockquote\u003e\n\u003cp\u003e\u0026ldquo;We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u0026ndash; Donald Knuth\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eIf you already have an existing service, for the most part, Python\u0026rsquo;s benefits (easy to write \u0026amp; read, library ecosystem, etc.) out-weigh the benefits of optimizing for that 97% by switching to a compiled language. And really, for most services, Python is rarely the biggest bottleneck. I mean, most APIs needn\u0026rsquo;t be scalable to gazilion requests/sec.\u003c/p\u003e\n\u003cp\u003eRegardless though, if your hot-paths include some sort of iteration over huge data, then Python\u0026rsquo;s shortcomings in speed, and its memory hungriness are hideously bare to witness. For many of these cases though, there a two simple best practices that can help alleviate those ills tremendously. Those are defaulting to list comprehension instead of loop + append combo, and returning generators instead of returning lists.\u003c/p\u003e\n\u003cp\u003eBoth of these tools can yield a surprising amount of benefits for little to no extra effort. When I first read of these benefits in \u003ca href=\"https://effectivepython.com/\"\u003eEffective Python\u003c/a\u003e, I thought this would be a nice chance to try out profiling Python code. For this, I used Python\u0026rsquo;s built-in \u003ca href=\"https://docs.python.org/3/library/profile.html#module-cProfile\"\u003ecProfile\u003c/a\u003e for execution time and a neat library called \u003ca href=\"https://pypi.org/project/memory-profiler/\"\u003ememory-profiler\u003c/a\u003e for memory usage.\u003cbr\u003e\nCheck out this \u003ca href=\"https://gist.github.com/amohamed11/8bc40153d964c0a7cd8bf13f5f7dae9d\"\u003egist\u003c/a\u003e for the entire Python script I used.\u003c/p\u003e\n\u003ch2 id=\"comprehend-this-speedup\"\u003eComprehend This Speedup \u003ca href=\"#comprehend-this-speedup\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s take the following example of a generic loop\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eexhibitA\u003c/span\u003e(n):\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e a = []\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003efor\u003c/span\u003e i in range(\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e2\u003c/span\u003e**n):\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003eif\u003c/span\u003e i%\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e2\u003c/span\u003e == \u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e0\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e a.append(i)\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis code simply iterates over a range and creates a list containing only even numbers. Now lets run it and profile it.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 524292 function calls in 0.122 seconds\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e Ordered by: standard name\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e ncalls tottime percall cumtime percall filename:lineno(function)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.004 0.004 0.122 0.122 \u0026lt;string\u0026gt;:1(\u0026lt;module\u0026gt;)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.096 0.096 0.118 0.118 comprehension_and_generators.py:5(exhibitA)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.122 0.122 {built-in method builtins.exec}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 524288 0.022 0.000 0.022 0.000 {method \u0026#39;append\u0026#39; of \u0026#39;list\u0026#39; objects}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.000 0.000 {method \u0026#39;disable\u0026#39; of \u0026#39;_lsprof.Profiler\u0026#39; objects}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eExhibit A: max=39.58 MiB, min=24.3 MiB, diff=15.27 MiB\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eIt takes approximately \u003cstrong\u003e0.122s\u003c/strong\u003e to run and uses about about 15.27 MiB (wow, python is one chonky boi) when you basically remove miscellenous import modules.\nAlright that\u0026rsquo;s not too good, let\u0026rsquo;s what we can do about that.\u003c/p\u003e\n\u003cp\u003eHere is the code re-written to use list comprehension\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eexhibitA_comprehension\u003c/span\u003e(n):\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e a = [i \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003efor\u003c/span\u003e i in range(\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e2\u003c/span\u003e**n) \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003eif\u003c/span\u003e i%\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e2\u003c/span\u003e == \u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e0\u003c/span\u003e]\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eSuccinct and to the point.\u003cbr\u003e\nAnd here is the profile\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 5 function calls in 0.051 seconds\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e Ordered by: standard name\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e ncalls tottime percall cumtime percall filename:lineno(function)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.003 0.003 0.051 0.051 \u0026lt;string\u0026gt;:1(\u0026lt;module\u0026gt;)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.048 0.048 comprehension_and_generators.py:11(exhibitA_comprehension)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.048 0.048 0.048 0.048 comprehension_and_generators.py:12(\u0026lt;listcomp\u0026gt;)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.051 0.051 {built-in method builtins.exec}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.000 0.000 {method \u0026#39;disable\u0026#39; of \u0026#39;_lsprof.Profiler\u0026#39; objects}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eExhibit A with list comprehension: max=37.75 MiB, min=24.31 MiB, diff=13.43 MiB\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eNow we\u0026rsquo;re cooking with gas. Our execution time is down to \u003cstrong\u003e0.051s\u003c/strong\u003e from 0.122s, a huge speedup factor of ~2.4x. Function calls are also reduced, by 5 orders of magnitude nevertheless. A nice bonus for debuggability.\u003cbr\u003e\nSince we are still creating the same list, our memory usage is essentially the same, at about 13.43 MiB for our function.\u003c/p\u003e\n\u003cp\u003eSo, next time you catch yourself writing the same old for loop + append combo, see if it can be written using list comprehension instead. In most cases it can be, and personally I would argue it is also more readable as well, but that\u0026rsquo;s entirely subjective and is neither here nor there.\u003c/p\u003e\n\u003ch2 id=\"yield-or-else\"\u003eYield or Else \u003ca href=\"#yield-or-else\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eGenerators are a bit less clear cut in their use cases. Where they really shine is where memory is a bigger constraint than speed. For example when you are ingesting a whole lot of data which needs to be processed in some form. In these scenarios using a generator will avail your program from holding all that data in a in-memory list while they are waiting to be processed.\u003c/p\u003e\n\u003cp\u003eHere is the same loop from the initial example, but in this case we need to carry out another filtering process after our list is created\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eexhibitB\u003c/span\u003e(n):\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e a = []\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003efor\u003c/span\u003e i in range(\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e2\u003c/span\u003e**n):\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003eif\u003c/span\u003e i%\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e2\u003c/span\u003e == \u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e0\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e a.append(i)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e div_by_8_and_2 = len([x \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003efor\u003c/span\u003e x in a \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003eif\u003c/span\u003e x%\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e8\u003c/span\u003e == \u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e0\u003c/span\u003e])\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eand here are our results\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 524294 function calls in 0.140 seconds\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e Ordered by: standard name\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e ncalls tottime percall cumtime percall filename:lineno(function)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.005 0.005 0.140 0.140 \u0026lt;string\u0026gt;:1(\u0026lt;module\u0026gt;)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.098 0.098 0.135 0.135 comprehension_and_generators.py:14(exhibitB)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.017 0.017 0.017 0.017 comprehension_and_generators.py:20(\u0026lt;listcomp\u0026gt;)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.140 0.140 {built-in method builtins.exec}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.000 0.000 {built-in method builtins.len}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 524288 0.021 0.000 0.021 0.000 {method \u0026#39;append\u0026#39; of \u0026#39;list\u0026#39; objects}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.000 0.000 {method \u0026#39;disable\u0026#39; of \u0026#39;_lsprof.Profiler\u0026#39; objects}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eExhibit B: max=40.91 MiB, min=24.91 MiB, diff=16.0 MiB\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eOh lord, that\u0026rsquo;s almost 16 MiB just to store 131,072 ints. Shouldn\u0026rsquo;t that be like 1MiB or so? What\u0026rsquo;s up here?\u003cbr\u003e\n\u003cem\u003e*few tabs \u0026amp; getsizeof calls later*\u003c/em\u003e\u003cbr\u003e\nHuh, why are ints in Python 28 bytes minimum? that seems a bit excessive.\u003cbr\u003e\n\u003cem\u003e*even more tabs later*\u003c/em\u003e\u003cbr\u003e\nWAIT, WHAT DO YOU MEAN EVERY INT IS AN ENTIRE OBJECT IN PYTHON? WHAT? WHY DO YOU NEED A REF COUNT FOR AN INT???\u003c/p\u003e\n\u003cp\u003ePhew! alright, that seems like entire rabbit hole that I do not want to jump into right now.\nI mean, I\u0026rsquo;m sure there is good reasoning for this (better handling of large numbers maybe??), but that\u0026rsquo;s an investigation for another day.\u003c/p\u003e\n\u003cp\u003eComing back to generators, let\u0026rsquo;s see how much of that memory we can get back.\u003cbr\u003e\nHere is the generator version\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#ff0086;font-weight:bold\"\u003e_generator\u003c/span\u003e(n):\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003efor\u003c/span\u003e i in range(\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e2\u003c/span\u003e**n):\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003eif\u003c/span\u003e i%\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e2\u003c/span\u003e == \u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e0\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003eyield\u003c/span\u003e i\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eexhibitB_generator\u003c/span\u003e(n):\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e it = _generator(n)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e div_by_8_and_2 = len([x \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003efor\u003c/span\u003e x in it \u003cspan style=\"color:#fb660a;font-weight:bold\"\u003eif\u003c/span\u003e x%\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e8\u003c/span\u003e == \u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e0\u003c/span\u003e])\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eand our results\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 524295 function calls in 0.124 seconds\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e Ordered by: standard name\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e ncalls tottime percall cumtime percall filename:lineno(function)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.124 0.124 \u0026lt;string\u0026gt;:1(\u0026lt;module\u0026gt;)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 524289 0.072 0.000 0.072 0.000 comprehension_and_generators.py:22(_generator)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.001 0.001 0.124 0.124 comprehension_and_generators.py:27(exhibitB_generator)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.051 0.051 0.123 0.123 comprehension_and_generators.py:30(\u0026lt;listcomp\u0026gt;)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.124 0.124 {built-in method builtins.exec}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.000 0.000 {built-in method builtins.len}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e 1 0.000 0.000 0.000 0.000 {method \u0026#39;disable\u0026#39; of \u0026#39;_lsprof.Profiler\u0026#39; objects}\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eExhibit B with generator: max=28.61 MiB, min=24.97 MiB, diff=3.64 MiB\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eFrom 16 MiB down to \u003cstrong\u003e3.64 MiB\u003c/strong\u003e, that\u0026rsquo;s an astounding reduction of -440% in memory usage for arguably no extra code.\u003cbr\u003e\nYou might notice there is also a very slight speedup, from 0.140 to 0.124, but that is actually an artifact of this specific example. Depending on your case you could very easily end up with a noticeable slow down when using generators. Also techincally there is 1 more function call but that just the iterator asking if there is anything left and returning a no.\u003c/p\u003e\n\u003cp\u003eLike I mentioned earlier, generators are much more case specific than list comprehension. If you know that your list might get large, or if you can start processing that list one by one, it is a huge boon to your memory usage. But if that\u0026rsquo;s not the case, and you need the entire list first before processing it, or your case is more constrained by speed rather than memory, then generators are not the right tool.\u003c/p\u003e\n\u003cp\u003eReally that\u0026rsquo;s just a bunch of words for \u0026ldquo;it depends\u0026rdquo;. But as is often the case, that\u0026rsquo;s the answer for most programming questions.\u003cbr\u003e\nRegardless, it\u0026rsquo;s important to know the benefits \u0026amp; drawbacks for each tool you use. That\u0026rsquo;s what will help you optimize for that critical 3%.\u003c/p\u003e\n\u003ch2 id=\"conclusions\"\u003eConclusions \u003ca href=\"#conclusions\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eList comprehension is signficantly faster than loop + append combo.\u003c/li\u003e\n\u003cli\u003eReturning generators is substantially more memory efficient than return entire lists.\u003c/li\u003e\n\u003cli\u003eRead \u003ca href=\"https://effectivepython.com/\"\u003eEffective Python\u003c/a\u003e, it\u0026rsquo;s pretty good.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThat was fun. And I learned a lot about Python after delving a bit into some of the opcode in some tangents.\nSomething I kinda wanna look into a bit more is how \u003ccode\u003e\u0026lt;listcomp\u0026gt;\u003c/code\u003e is so much more efficient than an append loop. It seems that\u0026rsquo;s thanks to its usage of the \u003ccode\u003eLIST_APPEND\u003c/code\u003e opcode instead of loading the append method, so that would be a good place to start. But that\u0026rsquo;s for some other time, that\u0026rsquo;s enough Python existentialism for one day.\u003c/p\u003e\n","url":"http://anecdotalthoughts.com/posts/profiling-comprehension-generators/","image":"http://anecdotalthoughts.com/photos/","banner_image":"http://anecdotalthoughts.com/photos/","date_published":"11026-11-09T20:1111:00+00:00","date_modified":"11026-11-09T20:1111:00+00:00","author":{"name":"Anas Mohamed","url":"http://anecdotalthoughts.com/"}},{"id":"65df3042fa526da288908acde119aecb6204bbe2","title":"Technical Debt or Why I Hate Shovelling Snow","summary":"","content_text":"I really hate shovelling snow. But regardless, I still have to do it. And often I have to, do it in frostbite-inducing weather, shovel a ridiculously long corner sidewalk, and do it multiple times a week. As such, every year, and without fail, there will be a series of snowy days where I simply do not shovel out of exhaustion, being too busy, or mostly abject laziness. My mind always defaults to the same justifications that I know to be silly, like why shovel now if it\u0026rsquo;s gonna snow tomorrow anyways? Why shovel for both today and tomorrow, instead of shovelling it all at once the day after?\nI\u0026rsquo;m sure anyone who had had to deal with copious amounts of snow is fighting the intense urge to facepalm. Most likely, because they too have fallen for that trap of their own design before.\nYou see, the thing with not shovelling consecutive days of snow is that the interest on your snow debt is horribly efficient at compounding. If you let the snow be, the bottom layer will very quickly begin to harden into an icy rock that really wants to grow up and become a glacier. As such I always end up paying the price of my laziness in double, often having to shovel for 40 mins to an hour (never live in a corner lot, trust me). And recently, after one of those arduous hours where I had to hurl ossified snow off an awfully dull shovel that I probably should\u0026rsquo;ve changed by now, it dawned on me how apt of an analogy snow shovelling is for technical debt.\nWith disregarded \u0026amp; unshovelled snow, just as with accidentally accrued technical debt, the one who always ends up paying the price is yourself - or at least it would be if a career in this industry did not hinge on job-hopping, resulting in git blame being borderline useless - and your team as a whole. When you want to go outside (read: fix bugs, and add features) it becomes a balancing game and a struggle with inches of snow. And when you finally buckle up to shovel properly (read: refactor) it ends up taking X times (where X is a function of time left untouched) the original effort if you had just done it from the get-go. The bottom layer of the snow, now turned rock-like ice, becomes increasingly difficult to remove. Just as those short-sighted architectural design decisions that become unmovable glacials your critical systems have now been built on. Systems that have long ago turned into a big ball of mud, and are now too large, too complicated, and too obtuse to refactor in a reasonable time without breaking everything else that is dependent on it.\n\u0026ldquo;Systems that are permitted to evolve gracefully in a series of small, controlled stages can stay the seismic upheaval that can result from deferring change.\u0026rdquo;\n\u0026ndash; Foote \u0026amp; Yoder - Evolution, Architecture, and Metamorphosis\nJust like how with snow, simply shovelling even a few layers of today\u0026rsquo;s snow will pay off big time when you have to shovel even more snow tomorrow, incremental refactoring will pay off by delaying or all on ending the need for the complete upheaval of your codebase. So, next time you notice that one function is getting a bit too long \u0026amp; obtuse in what it does and is also somehow being called by everything else, try to take the time to shovel a few layers of it here and there, or it too will become as solid as granite and will require a much sharper shovel.\nyeah, I really should\u0026rsquo;ve shovelled yesterday, and I definitely should\u0026rsquo;ve refactored that function that is few hundred lines too long and resulted in 2 hours of debugging.\n","content_html":"\u003cp\u003eI really hate shovelling snow. But regardless, I still have to do it. And often I have to, do it in frostbite-inducing weather, shovel a ridiculously long corner sidewalk, and do it multiple times a week. As such, every year, and without fail, there will be a series of snowy days where I simply do not shovel out of exhaustion, being too busy, or mostly abject laziness. My mind always defaults to the same justifications that I know to be silly, like why shovel now if it\u0026rsquo;s gonna snow tomorrow anyways? Why shovel for both today and tomorrow, instead of shovelling it all at once the day after?\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;m sure anyone who had had to deal with copious amounts of snow is fighting the intense urge to facepalm. Most likely, because they too have fallen for that trap of their own design before.\u003c/p\u003e\n\u003cp\u003eYou see, the thing with not shovelling consecutive days of snow is that the interest on your snow debt is horribly efficient at compounding. If you let the snow be, the bottom layer will very quickly begin to harden into an icy rock that really wants to grow up and become a glacier. As such I always end up paying the price of my laziness in double, often having to shovel for 40 mins to an hour (never live in a corner lot, trust me). And recently, after one of those arduous hours where I had to hurl ossified snow off an awfully dull shovel that I probably should\u0026rsquo;ve changed by now, it dawned on me how apt of an analogy snow shovelling is for technical debt.\u003c/p\u003e\n\u003cp\u003eWith disregarded \u0026amp; unshovelled snow, just as with accidentally accrued technical debt, the one who always ends up paying the price is yourself - or at least it would be if a career in this industry did not hinge on job-hopping, resulting in \u003ccode\u003egit blame\u003c/code\u003e being borderline useless - and your team as a whole. When you want to go outside (read: fix bugs, and add features) it becomes a balancing game and a struggle with inches of snow. And when you finally buckle up to shovel properly (read: refactor) it ends up taking X times (where X is a function of time left untouched) the original effort if you had just done it from the get-go. The bottom layer of the snow, now turned rock-like ice, becomes increasingly difficult to remove. Just as those short-sighted architectural design decisions that become unmovable glacials your critical systems have now been built on. Systems that have long ago turned into a \u003ca href=\"http://www.laputan.org/pub/foote/mud.pdf\"\u003ebig ball of mud\u003c/a\u003e, and are now too large, too complicated, and too obtuse to refactor in a reasonable time without breaking everything else that is dependent on it.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u0026ldquo;Systems that are permitted to evolve gracefully in a series of small, controlled stages can stay the seismic upheaval that can result from deferring change.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u0026ndash; Foote \u0026amp; Yoder - \u003ca href=\"http://www.laputan.org/pub/foote/plop95.pdf\"\u003eEvolution, Architecture, and Metamorphosis\u003c/a\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eJust like how with snow, simply shovelling even a few layers of today\u0026rsquo;s snow will pay off big time when you have to shovel even more snow tomorrow, incremental refactoring will pay off by delaying or all on ending the need for the complete upheaval of your codebase. So, next time you notice that one function is getting a bit too long \u0026amp; obtuse in what it does and is also somehow being called by everything else, try to take the time to shovel a few layers of it here and there, or it too will become as solid as granite and will require a much sharper shovel.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eyeah, I really should\u0026rsquo;ve shovelled yesterday, and I definitely should\u0026rsquo;ve refactored that function that is few hundred lines too long and resulted in 2 hours of debugging.\u003c/em\u003e\u003c/p\u003e\n","url":"http://anecdotalthoughts.com/posts/shovelling-technical-debt/","image":"http://anecdotalthoughts.com/photos/","banner_image":"http://anecdotalthoughts.com/photos/","date_published":"15016-15-09T152:1515:00-07:00","date_modified":"15016-15-09T152:1515:00-07:00","author":{"name":"Anas Mohamed","url":"http://anecdotalthoughts.com/"}},{"id":"a5982aa7eac81476ae323bf87889f800e46e7969","title":"Wealth is not a Virtue","summary":"","content_text":"Hey, remember when as a teenager you were suddenly hit with the crippling realization that you, contrary to what your 8 year old self thought, do not in fact know what you want to do as an adult? How come that 8 year old you knew exactly what you wanted to do, but your teenage self does not? is it the fact that you began to realize that firefighters \u0026amp; math teachers do not get paid well?\nHonestly how come is it that every kid coming-of-age is forced to decide between earning a livable wage or doing what they love \u0026amp; might even be good at? And on that note, since when have universities, the supposed bastion of academia, become merely a pipeline for industries?\nIt seems that in the providence of capitalism, no individual nor institution is safe from monetization. Kids are nothing but future labour, and universities are nothing but skilled worker factories. Your worth is your productivity. And everything is merely fodder to the machinations of the capitalist machine.\nIn university, I remember anyone that did not opt for a degree that is highly likely to be lucrative, were mocked for it. Science \u0026amp; Engineering students making fun of Liberal Art majors is the quintessential background noise of any campus. This is what we have decided as a society. To forsake the silly humanities and devote ourselves purely to economically sensible paths. More and more, every path that does not result in more money in the pockets of megacorp CEOs is seen as the fool\u0026rsquo;s path. Even STEM in its own way is not safe from this creeping monetization of education. The incentives for STEM students \u0026amp; faculty continues to be constantly wrapped by the whims of corporations, till it has become essentially hiring grounds for companies.\nSide note: as a student I found that I learned \u0026amp; practiced critical thinking more in Philosophy and History courses than I have in any Science or Math course. Aristotle\u0026rsquo;s Unmoved mover taught about the power of syllogistic demonstration, while investigating the fall of the Ottoman Empire taught me about the complex navigation required in attempting to alleviate the biased lenses of history. Other conclusions from each course is that infinity is really weird, and nation-state relations is an N-body problem.\nMoney has no intrinsic value. Instead its value is completely based on what you exchange it for. Obvious, I know. Yet somehow, we have glorified the pursuit of wealth, and prioritize working towards its attainment over almost everything else. If you go to school, you do it to find well-paying jobs, not pursue interests. If a job pays enough, we disregard any moral agency we have to justify the pay. This twisted glorification has infected every last vestibule of our society. In the dogged pursuit of wealth, we have perverted our priorities and shifted our morals. Wealth is success, and anything less is failure. The world is richer than ever, yet we deny our fellow brethren their basic needs unless they can pay. Our central systems optimize for the few over the many, and its survival hinges on the many clinging to the hope of becoming the chosen few. All is fair in the pursuit of wealth.\nPatience is a virtue, kindness is a virtue, wisdom is a virtue, humility is a virtue, but wealth is not.\n","content_html":"\u003cp\u003eHey, remember when as a teenager you were suddenly hit with the crippling realization that you, contrary to what your 8 year old self thought, do not in fact know what you want to do as an adult? How come that 8 year old you knew exactly what you wanted to do, but your teenage self does not? is it the fact that you began to realize that firefighters \u0026amp; math teachers do not get paid well?\u003c/p\u003e\n\u003cp\u003eHonestly how come is it that every kid coming-of-age is forced to decide between earning a livable wage or doing what they love \u0026amp; might even be good at? And on that note, since when have universities, the supposed bastion of academia, become merely a pipeline for industries?\u003cbr\u003e\nIt seems that in the providence of capitalism, no individual nor institution is safe from monetization. Kids are nothing but future labour, and universities are nothing but skilled worker factories. Your worth is your productivity. And everything is merely fodder to the machinations of the capitalist machine.\u003c/p\u003e\n\u003cp\u003eIn university, I remember anyone that did not opt for a degree that is highly likely to be lucrative, were mocked for it. Science \u0026amp; Engineering students making fun of Liberal Art majors is the quintessential background noise of any campus. This is what we have decided as a society. To forsake the silly humanities and devote ourselves purely to economically sensible paths. More and more, every path that does not result in more money in the pockets of megacorp CEOs is seen as the fool\u0026rsquo;s path. Even STEM in its own way is not safe from this creeping monetization of education. The incentives for STEM students \u0026amp; faculty continues to be constantly wrapped by the whims of corporations, till it has become essentially hiring grounds for companies.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eSide note: as a student I found that I learned \u0026amp; practiced critical thinking more in Philosophy and History courses than I have in any Science or Math course. Aristotle\u0026rsquo;s Unmoved mover taught about the power of syllogistic demonstration, while investigating the fall of the Ottoman Empire taught me about the complex navigation required in attempting to alleviate the biased lenses of history. Other conclusions from each course is that infinity is really weird, and nation-state relations is an N-body problem.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eMoney has no intrinsic value. Instead its value is completely based on what you exchange it for. Obvious, I know. Yet somehow, we have glorified the pursuit of wealth, and prioritize working towards its attainment over almost everything else. If you go to school, you do it to find well-paying jobs, not pursue interests. If a job pays enough, we disregard any moral agency we have to justify the pay. This twisted glorification has infected every last vestibule of our society. In the dogged pursuit of wealth, we have perverted our priorities and shifted our morals. Wealth is success, and anything less is failure. The world is richer than ever, yet we deny our fellow brethren their basic needs unless they can pay. Our central systems optimize for the few over the many, and its survival hinges on the many clinging to the hope of becoming the chosen few. All is fair in the pursuit of wealth.\u003c/p\u003e\n\u003cp\u003ePatience is a virtue, kindness is a virtue, wisdom is a virtue, humility is a virtue, but wealth is not.\u003c/p\u003e\n","url":"http://anecdotalthoughts.com/posts/wealth-is-not-a-virtue/","image":"http://anecdotalthoughts.com/photos/","banner_image":"http://anecdotalthoughts.com/photos/","date_published":"13016-13-09T10:1313:00+00:00","date_modified":"13016-13-09T10:1313:00+00:00","author":{"name":"Anas Mohamed","url":"http://anecdotalthoughts.com/"}},{"id":"2be5cdd797557f8c1d8297379ce46388ab9c3696","title":"Things I Liked in 2021","summary":"","content_text":"2021 Patch Notes # \u0026ldquo;One of the strange things about adulthood is that you are your current self, but you are also all the selves you used to be, the ones you grew out of but can\u0026rsquo;t ever quite get rid of.\u0026rdquo;\n\u0026ndash; John Green, The Anthropocene Reviewed\nNotable events \u0026amp; personal changes in 2021.\nGraduated from University. Got a job at a local startup with cool people working on cool things. Finally checked out D\u0026amp;D for the first time, and as a DM nevertheless. Figured out the base identity of my blog. And really, what I want my public internet presence to be. My appreciation for the humanities grew more than anything else this past year. From philosophy and history to the written word and music. Learned # \u0026ldquo;Confusion is part of programming.\u0026rdquo;\n\u0026ndash; Felienne Hermans, The Programmer\u0026rsquo;s Brain\nFew programming languages \u0026amp; general concepts that I learned about that really stuck with me. The next iteration of this list will probably be even longer as I spend even more time programming as a full-time software developer in 2022. School is great for learning, but nothing beats practice.\nConcepts\nDistributed Systems ⇒ fault tolerance, SPMD/MPMD, and most importantly granularity \u0026amp; Amdahl\u0026rsquo;s law. Database Design ⇒ ACID transactions, on-disk storage, and created a DB Kagi Networking \u0026amp; Security Protocols ⇒ OSI model, hashes, symmetric \u0026amp; asymmetric cryptography, and learned how to smash the stack for fun. Programming Languages\nRust ⇒ fantastic community, best-in-class tooling, but the borrow checker still confuses me at times. Go ⇒ also amazing tooling, focus on simplicity \u0026amp; readability, and convention over configuration (which I kinda like) Watched # Watched way more movies this year, and way less anime. Consequently, got a ton of anime to binge through in 2022 (even though next year is looking packed for anime).\nArcane\nFantastic character-driven writing coupled with jaw-dropping animation and soundtrack. Easily my favorite media of the year.\nSoul\nPixar never disappoints. But this time they have went above \u0026amp; beyond. Exploration of death and the purpose of life has never been better animated.\nDune\nAlthough I haven\u0026rsquo;t read the books, Dune\u0026rsquo;s influence on sci-fi since 1965 oozes through and in the best of ways. Loved it.\nJojo Rabbit\nTaika Waititi is an absolute gem. Watch this now \u0026hellip; or when you have time, no rush.\nThe Grand Budapest Hotel\nNever have I ever laughed at mere camera shots \u0026amp; angles as much as I have in this movie. The Grand Budapest Hotel could\u0026rsquo;ve easily been a silent movie, and maintain its humor. The peak of visual comedy.\nPlayed # Played waaaay less singleplayer games this year. For better or worse, gaming has become a medium for hanging out with friends more than anything else. Regardless, it has been a fun year of gaming for me. Here is to more fun to be had in 2022.\nGuilty Gear -Strive-\nNothing more humbling than getting run-over by some crazy combos \u0026amp; getting absolutely outplayed in a 1v1 setting. The hardest soundtrack, characters that marry Anime \u0026amp; Rock in the best of ways, along with gorgeous animation, and a reliable rollback netcode. Oh, and the FGC is undefeated as a community. No other community is as welcoming, and supportive as the FGC. Really, just an all around a great time.\nDivinity: Original Sin 2\nTook ~90 hours to beat while playing with friends, and I loved every hour of playing as an undead tank necromancer. Fantastic intro into the world of CRPGs.\nNarita Boy\nRandomly picked this up on Game Pass, and was delightfully impressed by beautiful \u0026amp; colorful pixel art and tight gameplay. Story was intriguing, but dull at parts.\nValorant\nAnother year of dashing into headshots, and somehow I\u0026rsquo;ve enjoyed it even more. Tournaments were much more intense \u0026amp; entertaining to watch.\nHalo Infinite\nOnly played the Multiplayer, but as person that never played Halo growing up, I completely understand why the franchise is so beloved. Tight shooting with fun maps (some better than others). Great time with the boys.\nSlay the Spire\nA simple but addictive rogue-like. No wonder it has inspired so many card rogue-likes. One day I\u0026rsquo;ll beat that damn tower, but that day will not be in 2021.\nRead # \u0026ldquo;A good bookshop is just a genteel Black Hole that knows how to read.\u0026rdquo;\n\u0026ndash; Terry Pratchett, Guards! Guards!\nI read exactly 31 books in 2021.\nMe, I read 31 books, in a single year. Wow. Well mostly listened to, but still. For context, from a quick glance at goodreads, I have read 15 books in 2020, and since I have started tracking in 2016, a whopping total of 24 books.\n2021 was the year reading truly became one of my main hobbies if not the main one. And hopefully, 2022 will be just as great year for my to-be-read list as 2021 has been, so here\u0026rsquo;s to many more great reads to come.\nGuards! Guards! by Terry Pratchett\nReading Pratchett has taught made me stop, notice, and appreciate how beautiful prose can be and what it can do. Laughing at the least consequential of descriptions \u0026amp; hilarious choice of words was the norm while reading this, and any of Pratchett\u0026rsquo;s writing. That\u0026rsquo;s not even to mention Pratchett\u0026rsquo;s unqiue storytelling, engaging exploration of various themes from death to gender, capitalism, and a lot more. My newfound appreciation for the written form through Pratchett\u0026rsquo;s work was one of the main reasons for me wanting to write more. Pratchett was truly a literary genius, and his writing is what I aspire to emulate and will continue to learn from.\nGideon the Ninth by Tamsyn Muir\nSwordsmen \u0026amp; necromancer stuck in a gothic castle in space, where a locked-room murder mystery occurs, all bolstered by boisterous \u0026amp; charismatic characters. What\u0026rsquo;s not to love. Muir has a unique voice and her writing bleeds with personality and humor. A fantastic read, I have never been more invested in a series, I cannot wait for more books in The Locked Tomb series.\nA Gentleman in Moscow by Amor Towles\nI have never stepped a foot in Russia, nevermind Hotel Metropol in Moscow. Yet I have a vivid image of it. Of its bustling main lobby, and its world class restaurant. None of those images are based on photos, but rather Towles obsessive and meticulous description of every nook \u0026amp; cranny of the Hotel Metropol. And that\u0026rsquo;s just the setting. By the end of the book, Count Alexander Rostov, the main protagonist, felt like a friend I had known for years. His habits, routines, and philosophies. Towles has crafted an enthralling setting filled with characters that felt as more real as any out there.\nThe Myth of Sisyphus by Albert Camus\nA dense read that has left me thinking about it months later. And while I may not have been pursuaded into absurdism, I\u0026rsquo;m very much so intrigued to delve into more of Camus\u0026rsquo; work. This has also gotten me interested in exploring more of continental philosophy.\nProject Hail Mary by Andy Weir\nWeir struck lightening with The Martian, and he has done it again with this book. Amazing sci-fi narrative with fantastic storytelling, supported with characters that I truly cared for.\nPiranesi by Susanna Clarke\nI don\u0026rsquo;t think anything describes this book more than, beautiful. Bewitching prose, and such vivid scenery that has truly captured by heart. Just check it out, trust me.\nThe Anthropocene Reviewed by John Green\nI have watched many a Vlogbrothers video, and this was more of John exploring what is seemingly a random, highly specific, and highly personal set of things, before neatly connecting this exploration to the general human experience. And damn, is he good at it.\nScythe by Neal Shusterman\nA sci-fi exploration of death, the profound effect of its eventuality on the human experience, and the more profound effect of it no longer being absolute. Even when I wasn\u0026rsquo;t reading this book, I was thinking about it non-stop. That probably will remain the case for more time yet.\nBlue Period by Tsubasa Yamaguchi\nA manga about art, talent, hardwork, and the beauty in pursuing what you love. There are a handful number of manga that have truly inspired me, and few did it in as gorgeous panels as Blue Period did.\nListened to # In 2021, I have listened to and enjoyed more genres than I knew existed in previous years. It\u0026rsquo;s weird looking at my playlists from 2020 \u0026amp; before, and seeing how picky I was. So going into 2022, I know I will enjoy way more music than my past, extremely picky taste, would have allowed me to enjoy.\nNurture - Porter Robinson\nA beautiful \u0026amp; nostalgic electro-pop record, that both lifts me up \u0026amp; hits me right in the heart in the perfect balance. I love video games, anime, and EDM, and this record is a celebration of it all, with Porter Robinson\u0026rsquo;s own personal \u0026amp; heart-felt storytelling.\nSometimes I Be Introvert - Little Simz\nBeautiful production, and delivery that showcases Little Simz\u0026rsquo;s talent \u0026amp; range. Simply listen to Introvert, the opening track, and thank me later.\nUse Me - PVRIS\nEver since discovering PVRIS some month ago, I have had this entire album on repeat non-stop. Filled with earworms and tight synthwave production, this whole album continue to impress the more I listen to.\nNavy\u0026rsquo;s Reprise - Blue Navy\nQuality production, vulnerable lyrics and unique delivery on laid-back beats. Blue Navy has quickly rose up to being on of my favorite underground rappers.\nRoadrunner: New Light, New Machine - Brockhampton\nThe boys are back, and they got features this time. While it might not top the Saturation trilogy, Brockhampton still continues to deliver.\nUnseen World - Band-Maid\nCatchy \u0026amp; addictive J-Rock that I continue to playback constantly and never tire off.\n","content_html":"\u003ch2 id=\"2021-patch-notes\"\u003e2021 Patch Notes \u003ca href=\"#2021-patch-notes\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u0026ldquo;One of the strange things about adulthood is that you are your current self, but you are also all the selves you used to be, the ones you grew out of but can\u0026rsquo;t ever quite get rid of.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u0026ndash; John Green, \u003cem\u003eThe Anthropocene Reviewed\u003c/em\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eNotable events \u0026amp; personal changes in 2021.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eGraduated from University.\u003c/li\u003e\n\u003cli\u003eGot a job at a local startup with cool people working on cool things.\u003c/li\u003e\n\u003cli\u003eFinally checked out D\u0026amp;D for the first time, and as a DM nevertheless.\u003c/li\u003e\n\u003cli\u003eFigured out the base identity of my blog. And really, what I want my public internet presence to be.\u003c/li\u003e\n\u003cli\u003eMy appreciation for the humanities grew more than anything else this past year. From philosophy and history to the written word and music.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"learned\"\u003eLearned \u003ca href=\"#learned\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u0026ldquo;Confusion is part of programming.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u0026ndash; Felienne Hermans, The Programmer\u0026rsquo;s Brain\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eFew programming languages \u0026amp; general concepts that I learned about that really stuck with me. The next iteration of this list will probably be even longer as I spend even more time programming as a full-time software developer in 2022. School is great for learning, but nothing beats practice.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConcepts\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDistributed Systems ⇒ fault tolerance, SPMD/MPMD, and most importantly granularity \u0026amp; Amdahl\u0026rsquo;s law.\u003c/li\u003e\n\u003cli\u003eDatabase Design ⇒ ACID transactions, on-disk storage, and created a DB \u003ca href=\"https:/github.com/amohamed11/kagi\"\u003eKagi\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eNetworking \u0026amp; Security Protocols ⇒ OSI model, hashes, symmetric \u0026amp; asymmetric cryptography, and learned how to \u003ca href=\"https://inst.eecs.berkeley.edu/~cs161/fa08/papers/stack_smashing.pdf\"\u003esmash the stack for fun\u003c/a\u003e.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eProgramming Languages\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eRust ⇒ fantastic community, best-in-class tooling, but the borrow checker still confuses me at times.\u003c/li\u003e\n\u003cli\u003eGo ⇒ also amazing tooling, focus on simplicity \u0026amp; readability, and convention over configuration (which I kinda like)\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"watched\"\u003eWatched \u003ca href=\"#watched\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWatched way more movies this year, and way less anime. Consequently, got a ton of anime to binge through in 2022 (even though next year is looking packed for anime).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eArcane\u003c/strong\u003e\u003cbr\u003e\nFantastic character-driven writing coupled with jaw-dropping animation and soundtrack. Easily my favorite media of the year.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSoul\u003c/strong\u003e\u003cbr\u003e\nPixar never disappoints. But this time they have went above \u0026amp; beyond. Exploration of death and the purpose of life has never been better animated.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDune\u003c/strong\u003e\u003cbr\u003e\nAlthough I haven\u0026rsquo;t read the books, Dune\u0026rsquo;s influence on sci-fi since 1965 oozes through and in the best of ways. Loved it.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eJojo Rabbit\u003c/strong\u003e\u003cbr\u003e\nTaika Waititi is an absolute gem. Watch this now \u0026hellip; or when you have time, no rush.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThe Grand Budapest Hotel\u003c/strong\u003e\u003cbr\u003e\nNever have I ever laughed at mere camera shots \u0026amp; angles as much as I have in this movie. The Grand Budapest Hotel could\u0026rsquo;ve easily been a silent movie, and maintain its humor. The peak of visual comedy.\u003c/p\u003e\n\u003ch2 id=\"played\"\u003ePlayed \u003ca href=\"#played\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003ePlayed waaaay less singleplayer games this year. For better or worse, gaming has become a medium for hanging out with friends more than anything else. Regardless, it has been a fun year of gaming for me. Here is to more fun to be had in 2022.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eGuilty Gear -Strive-\u003c/strong\u003e\u003cbr\u003e\nNothing more humbling than getting run-over by some crazy combos \u0026amp; getting absolutely outplayed in a 1v1 setting. The hardest soundtrack, characters that marry Anime \u0026amp; Rock in the best of ways, along with gorgeous animation, and a reliable rollback netcode. Oh, and the FGC is undefeated as a community. No other community is as welcoming, and supportive as the FGC. Really, just an all around a great time.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDivinity: Original Sin 2\u003c/strong\u003e\u003cbr\u003e\nTook ~90 hours to beat while playing with friends, and I loved every hour of playing as an undead tank necromancer. Fantastic intro into the world of CRPGs.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNarita Boy\u003c/strong\u003e\u003cbr\u003e\nRandomly picked this up on Game Pass, and was delightfully impressed by beautiful \u0026amp; colorful pixel art and tight gameplay. Story was intriguing, but dull at parts.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eValorant\u003c/strong\u003e\u003cbr\u003e\nAnother year of dashing into headshots, and somehow I\u0026rsquo;ve enjoyed it even more. Tournaments were much more intense \u0026amp; entertaining to watch.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHalo Infinite\u003c/strong\u003e\u003cbr\u003e\nOnly played the Multiplayer, but as person that never played Halo growing up, I completely understand why the franchise is so beloved. Tight shooting with fun maps (some better than others). Great time with the boys.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSlay the Spire\u003c/strong\u003e\u003cbr\u003e\nA simple but addictive rogue-like. No wonder it has inspired so many card rogue-likes. One day I\u0026rsquo;ll beat that damn tower, but that day will not be in 2021.\u003c/p\u003e\n\u003ch2 id=\"read\"\u003eRead \u003ca href=\"#read\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u0026ldquo;A good bookshop is just a genteel Black Hole that knows how to read.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u0026ndash; Terry Pratchett, \u003cem\u003eGuards! Guards!\u003c/em\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eI read exactly 31 books in 2021.\u003cbr\u003e\nMe, \u003cem\u003eI read 31 books\u003c/em\u003e, in a single year. Wow. Well mostly listened to, but still. For context, from a quick glance at goodreads, I have read 15 books in 2020, and since I have started tracking in 2016, a whopping total of 24 books.\u003c/p\u003e\n\u003cp\u003e2021 was the year reading truly became one of my main hobbies if not the main one. And hopefully, 2022 will be just as great year for my to-be-read list as 2021 has been, so here\u0026rsquo;s to many more great reads to come.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eGuards! Guards! by Terry Pratchett\u003c/strong\u003e\u003cbr\u003e\nReading Pratchett has taught made me stop, notice, and appreciate how beautiful prose can be and what it can do. Laughing at the least consequential of descriptions \u0026amp; hilarious choice of words was the norm while reading this, and any of Pratchett\u0026rsquo;s writing. That\u0026rsquo;s not even to mention Pratchett\u0026rsquo;s unqiue storytelling, engaging exploration of various themes from death to gender, capitalism, and a lot more. My newfound appreciation for the written form through Pratchett\u0026rsquo;s work was one of the main reasons for me wanting to \u003ca href=\"https://anecdotalthoughts.com/writing-for-the-sake-of-writing\"\u003ewrite more\u003c/a\u003e. Pratchett was truly a literary genius, and his writing is what I aspire to emulate and will continue to learn from.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eGideon the Ninth by Tamsyn Muir\u003c/strong\u003e\u003cbr\u003e\nSwordsmen \u0026amp; necromancer stuck in a gothic castle in space, where a locked-room murder mystery occurs, all bolstered by boisterous \u0026amp; charismatic characters. What\u0026rsquo;s not to love. Muir has a unique voice and her writing bleeds with personality and humor. A fantastic read, I have never been more invested in a series, I cannot wait for more books in The Locked Tomb series.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eA Gentleman in Moscow by Amor Towles\u003c/strong\u003e\u003cbr\u003e\nI have never stepped a foot in Russia, nevermind Hotel Metropol in Moscow. Yet I have a vivid image of it. Of its bustling main lobby, and its world class restaurant. None of those images are based on photos, but rather Towles obsessive and meticulous description of every nook \u0026amp; cranny of the Hotel Metropol. And that\u0026rsquo;s just the setting. By the end of the book, Count Alexander Rostov, the main protagonist, felt like a friend I had known for years. His habits, routines, and philosophies. Towles has crafted an enthralling setting filled with characters that felt as more real as any out there.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThe Myth of Sisyphus by Albert Camus\u003c/strong\u003e\u003cbr\u003e\nA dense read that has left me thinking about it months later. And while I may not have been pursuaded into absurdism, I\u0026rsquo;m very much so intrigued to delve into more of Camus\u0026rsquo; work. This has also gotten me interested in exploring more of continental philosophy.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProject Hail Mary by Andy Weir\u003c/strong\u003e\u003cbr\u003e\nWeir struck lightening with The Martian, and he has done it again with this book. Amazing sci-fi narrative with fantastic storytelling, supported with characters that I truly cared for.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePiranesi by Susanna Clarke\u003c/strong\u003e\u003cbr\u003e\nI don\u0026rsquo;t think anything describes this book more than, beautiful. Bewitching prose, and such vivid scenery that has truly captured by heart. Just check it out, trust me.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThe Anthropocene Reviewed by John Green\u003c/strong\u003e\u003cbr\u003e\nI have watched many a Vlogbrothers video, and this was more of John exploring what is seemingly a random, highly specific, and highly personal set of things, before neatly connecting this exploration to the general human experience. And damn, is he good at it.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eScythe by Neal Shusterman\u003c/strong\u003e\u003cbr\u003e\nA sci-fi exploration of death, the profound effect of its eventuality on the human experience, and the more profound effect of it no longer being absolute. Even when I wasn\u0026rsquo;t reading this book, I was thinking about it non-stop. That probably will remain the case for more time yet.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBlue Period by Tsubasa Yamaguchi\u003c/strong\u003e\u003cbr\u003e\nA manga about art, talent, hardwork, and the beauty in pursuing what you love. There are a handful number of manga that have truly inspired me, and few did it in as gorgeous panels as Blue Period did.\u003c/p\u003e\n\u003ch2 id=\"listened-to\"\u003eListened to \u003ca href=\"#listened-to\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eIn 2021, I have listened to and enjoyed more genres than I knew existed in previous years. It\u0026rsquo;s weird looking at my playlists from 2020 \u0026amp; before, and seeing how picky I was. So going into 2022, I know I will enjoy way more music than my past, extremely picky taste, would have allowed me to enjoy.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNurture - Porter Robinson\u003c/strong\u003e\u003cbr\u003e\nA beautiful \u0026amp; nostalgic electro-pop record, that both lifts me up \u0026amp; hits me right in the heart in the perfect balance. I love video games, anime, and EDM, and this record is a celebration of it all, with Porter Robinson\u0026rsquo;s own personal \u0026amp; heart-felt storytelling.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSometimes I Be Introvert - Little Simz\u003c/strong\u003e\u003cbr\u003e\nBeautiful production, and delivery that showcases Little Simz\u0026rsquo;s talent \u0026amp; range. Simply listen to Introvert, the opening track, and thank me later.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUse Me - PVRIS\u003c/strong\u003e\u003cbr\u003e\nEver since discovering PVRIS some month ago, I have had this entire album on repeat non-stop. Filled with earworms and tight synthwave production, this whole album continue to impress the more I listen to.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNavy\u0026rsquo;s Reprise - Blue Navy\u003c/strong\u003e\u003cbr\u003e\nQuality production, vulnerable lyrics and unique delivery on laid-back beats. Blue Navy has quickly rose up to being on of my favorite underground rappers.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRoadrunner: New Light, New Machine - Brockhampton\u003c/strong\u003e\u003cbr\u003e\nThe boys are back, and they got features this time. While it might not top the Saturation trilogy, Brockhampton still continues to deliver.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUnseen World - Band-Maid\u003c/strong\u003e\u003cbr\u003e\nCatchy \u0026amp; addictive J-Rock that I continue to playback constantly and never tire off.\u003c/p\u003e\n","url":"http://anecdotalthoughts.com/posts/things-i-liked-2021/","image":"http://anecdotalthoughts.com/photos/","banner_image":"http://anecdotalthoughts.com/photos/","date_published":"31126-31-09T1230:3131:00+00:00","date_modified":"31126-31-09T1230:3131:00+00:00","author":{"name":"Anas Mohamed","url":"http://anecdotalthoughts.com/"}},{"id":"d05e397501a37772f2a1196a12d6bb1889adfd48","title":"The Abundance of Software Missionaries","summary":"Don't be a Software Missionary","content_text":"Few years ago, I was checking out the highly regarded Structure and Interpretation of Computer Programs textbook, mostly on a whim to see what made it so influential. And while it is not the most difficult read, suffice to say, my efforts were not long-lived. But regardless, a few ideas did stick with me, such as programming being the marriage of procedures and data. Yet, what stuck with me even moreso, was the opening dedication to Alan Perlis. I encourage you to read the whole paragraph as it is heartwarming \u0026amp; uplifting in its hopefulness, but the main part that stuck with me was Perlis saying\nI hope the field of computer science never loses its sense of fun. Above all, I hope we don’t become missionaries. Don\u0026rsquo;t feel as if you\u0026rsquo;re Bible salesmen. The world has too many of those already.\nBack then, the missionary metaphor did not really click with me, at least not the implications of it. But as of late, it has become more and more evident why Perlis used missionaries specifically and not preachers or something else. See, unlike with the preachers who preach what they believe to be true to willing listeners, there is a sense of urgency, an overwhelming duty and pressure to save the ignorant \u0026amp; the foolish, that comes with being a missionary. I mean, someone has got to save the souls of the dynamic language users and let them know that their language sucks, right?\nFor some context, I enjoy consuming \u0026ldquo;programming\u0026rdquo; content, be it talks, articles, or blog posts. And what I have concluded is that the only common theme among software developers, is their absolute disdain with the state of software development. And sometimes enough burning hatred for that state to fuel them to go an hour long rant. And often, shortly following each of these scathing tirades, a solution is presented that promises the world. Just follow this new methodology, use this programming language, implement these design patterns and not the other one, and you are set. In other words, join my church, and you\u0026rsquo;ll be spared the sin of bad code.\nThese derisive rants are often tempting to any soul that has to deal with the buggy, over-engineered and complex monstrosity that is modern software on daily basis. But they are also often dismissive, and pessimistic of any progress made. All these rants do instead, is create a tech-savvy doomsday cult. I mean, is most software really that bad? maybe, but is the modern web the cancerous death-spiral of all software that many tout it to be? probably not.\nA healthy skepticism is essential for any developer to manage navigating the trends \u0026amp; fads that plague the software, and to help distinguish what is better and what\u0026rsquo;s just hyped. But so is healthy optimism. So grounding that skepticism in an optimistic outlook of what we can do to help better direct these trends, is much more constructive than simply raging about mistakes done.\nI leave you with this Perlis quote from the same dedication\nWe began to feel as if we really were responsible for the successful, error-free perfect use of these machines. I don\u0026rsquo;t think we are. I think we\u0026rsquo;re responsible for stretching them, setting them off in new directions, and keeping fun in the house.\n","content_html":"\u003cp\u003eFew years ago, I was checking out the highly regarded \u003ca href=\"https://mitpress.mit.edu/sites/default/files/sicp/full-text/book/book.html\"\u003e\u003cem\u003eStructure and Interpretation of Computer Programs\u003c/em\u003e\u003c/a\u003e textbook, mostly on a whim to see what made it so influential. And while it is not the most difficult read, suffice to say, my efforts were not long-lived. But regardless, a few ideas did stick with me, such as programming being the marriage of procedures and data. Yet, what stuck with me even moreso, was the opening dedication to \u003ca href=\"https://en.wikipedia.org/wiki/Alan_Perlis\"\u003eAlan Perlis\u003c/a\u003e. I encourage you to read the whole paragraph as it is heartwarming \u0026amp; uplifting in its hopefulness, but the main part that stuck with me was Perlis saying\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eI hope the field of computer science never loses its sense of fun. Above all, I hope we don’t become missionaries. Don\u0026rsquo;t feel as if you\u0026rsquo;re Bible salesmen. The world has too many of those already.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eBack then, the missionary metaphor did not really click with me, at least not the implications of it. But as of late, it has become more and more evident why Perlis used \u003cem\u003emissionaries\u003c/em\u003e specifically and not preachers or something else. See, unlike with the preachers who preach what they believe to be true to willing listeners, there is a sense of urgency, an overwhelming duty and pressure to save the ignorant \u0026amp; the foolish, that comes with being a missionary. I mean, someone has got to save the souls of the dynamic language users and let them know that their language sucks, right?\u003c/p\u003e\n\u003cp\u003eFor some context, I enjoy consuming \u0026ldquo;programming\u0026rdquo; content, be it talks, articles, or blog posts. And what I have concluded is that the only common theme among software developers, is their absolute disdain with the state of software development. And sometimes enough burning hatred for that state to fuel them to go an hour long rant. And often, shortly following each of these scathing tirades, a solution is presented that promises the world. Just follow this new methodology, use this programming language, implement these design patterns and not the other one, and you are set. In other words, \u003cstrong\u003ejoin my church, and you\u0026rsquo;ll be spared the sin of bad code\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eThese derisive rants are often tempting to any soul that has to deal with the buggy, over-engineered and complex monstrosity that is modern software on daily basis. But they are also often dismissive, and pessimistic of any progress made. All these rants do instead, is create a tech-savvy doomsday cult. I mean, is most software really that bad? maybe, but is the modern web the cancerous death-spiral of all software that many tout it to be? probably not.\u003c/p\u003e\n\u003cp\u003eA healthy skepticism is essential for any developer to manage navigating the trends \u0026amp; fads that plague the software, and to help distinguish what is better and what\u0026rsquo;s just hyped. But so is healthy optimism. So grounding that skepticism in an optimistic outlook of what we can do to help better direct these trends, is much more constructive than simply raging about mistakes done.\u003c/p\u003e\n\u003cp\u003eI leave you with this Perlis quote from the same dedication\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eWe began to feel as if we really were responsible for the successful, error-free perfect use of these machines. I don\u0026rsquo;t think we are. I think we\u0026rsquo;re responsible for stretching them, \u003cstrong\u003esetting them off in new directions, and keeping fun in the house\u003c/strong\u003e.\u003c/p\u003e\n\u003c/blockquote\u003e\n","url":"http://anecdotalthoughts.com/posts/the-abundance-of-software-missionaries/","image":"http://anecdotalthoughts.com/photos/","banner_image":"http://anecdotalthoughts.com/photos/","date_published":"27116-27-09T1130:2727:00+00:00","date_modified":"27116-27-09T1130:2727:00+00:00","author":{"name":"Anas Mohamed","url":"http://anecdotalthoughts.com/"}},{"id":"d1fd47c031776f8c94057492716f7131287d793e","title":"Writing for the Sake of Writing","summary":"Writing is fun.","content_text":"I remember one occasion where younger me, bored in class while the teacher read from some novel that should have instead been a play, opened my notes and scribbled down an incredibly original origin story for a super-hero called Lotus, which innovatively involved explosions \u0026amp; laboratories. My teacher, having noticed my head in my notebook, approached me, took my notes and read it before briefly commenting \u0026ldquo;glad you enjoy writing, but do it some other time\u0026rdquo;.\nThere is no real significance to this story. Matter-of-fact I had completely forgotten about it, until I had started drafting this post. Yet, it encapsulate what my attitude towards writing, fiction or non, had become. The idea, that there is always something more important, more meaningful and more productive to do (god forbid you are not productive for a breath or two) than sitting down and simply writing.\nWriting is fun. Yet, somewhere I have forgotten that fact. That simple enjoyment that 6th grade me, despite my hatred for English class, knew and indulged in. When writing that laboratory explosion scene, the thought of writing comics for living never even came up, instead I was fully, and totally absorbed with the genuine act of writing for the sake of writing.\nNow that I have graduated and gotten a full-time job, with it crept a sneaky existential crisis concerning what I do with my free time. \u0026ldquo;I play video games. What? What do you mean I need a new hobby?\u0026rdquo; is what the voice in my head responds to the thought of doing anything else. Playing games has been, for the majority of my life, what I do with my free time. But gaming, just like reading, is a form of consumption, albeit really good ones (read: addictive).\nWhat I need, I have realized, is an outlet. A place to have a dialogue with my thoughts, think them through, and put them down on paper (or keyboard). And a place to look stupid.\nThanks for reading my first entry in Anecdotal Thoughts.\n","content_html":"\u003cp\u003eI remember one occasion where younger me, bored in class while the teacher read from some novel that should have instead been a play, opened my notes and scribbled down an incredibly original origin story for a super-hero called Lotus, which innovatively involved explosions \u0026amp; laboratories. My teacher, having noticed my head in my notebook, approached me, took my notes and read it before briefly commenting \u0026ldquo;glad you enjoy writing, but do it some other time\u0026rdquo;.\u003c/p\u003e\n\u003cp\u003eThere is no real significance to this story. Matter-of-fact I had completely forgotten about it, until I had started drafting this post. Yet, it encapsulate what my attitude towards writing, fiction or non, had become. The idea, that there is always something more important, more meaningful and more productive to do (god forbid you are not productive for a breath or two) than sitting down and simply writing.\u003c/p\u003e\n\u003cp\u003eWriting is fun. Yet, somewhere I have forgotten that fact. That simple enjoyment that 6th grade me, despite my hatred for English class, knew and indulged in. When writing that laboratory explosion scene, the thought of writing comics for living never even came up, instead I was fully, and totally absorbed with the genuine act of writing for the sake of writing.\u003c/p\u003e\n\u003cp\u003eNow that I have graduated and gotten a full-time job, with it crept a sneaky existential crisis concerning what I do with my free time. \u0026ldquo;I play video games. What? What do you mean I need a new hobby?\u0026rdquo; is what the voice in my head responds to the thought of doing anything else. Playing games has been, for the majority of my life, what I do with my free time. But gaming, just like reading, is a form of consumption, albeit really good ones (read: addictive).\u003c/p\u003e\n\u003cp\u003eWhat I need, I have realized, is an outlet. A place to have a dialogue with my thoughts, think them through, and put them down on paper (or keyboard). And a place to \u003ca href=\"https://danluu.com/look-stupid/\" title=\"Willingness to look stupid\"\u003elook stupid\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eThanks for reading my first entry in Anecdotal Thoughts.\u003c/p\u003e\n","url":"http://anecdotalthoughts.com/posts/writing-for-the-sake-of-writing/","image":"http://anecdotalthoughts.com/photos/","banner_image":"http://anecdotalthoughts.com/photos/","date_published":"6116-06-09T110:66:00+00:00","date_modified":"6116-06-09T110:66:00+00:00","author":{"name":"Anas Mohamed","url":"http://anecdotalthoughts.com/"}},{"id":"3e79b96491ebab4b5cef7f6ab6d59c5209062f45","title":"Building a Lightweight Key-Value Database in Go","summary":"On-disk B+trees \u0026 other midly extreme sports.","content_text":"This will be more of a high-level technical overview of key-value database design using notes from my research while working on Kagi, a on-disk key-value store.\nAt the bottom are some fantastic resources for better understanding B+Trees and Key-Value store design, definitely check those out.\nWhy build Kagi? # About a year and a half ago, I decided to take Databases II. And as soon as the first assignment, a run-of-the-mill embedded SQL assignment, was released, I remember thinking \u0026ldquo;why can\u0026rsquo;t we just create a database instead\u0026rdquo;. So this past holiday, I thought I\u0026rsquo;d finally sit down and do just that.\nDatabase design, and why a KV database? # For this, ironically, I ended up pulling out my lecture notes from said Databases II class. Despite the dreadful assignments, this course did touch on some essential concepts of database design. Here is the bullet-point summary:\nAccessing the data (on-disk vs in-memory, ordered vs unordered, etc.) Executing queries (SQL compiler, query optimizer, etc.) Managing transactions (ACID, scheduling, batching, etc.) Maintaining the database (index updates, tombstone cleanup, etc.) As you can already probably tell, databases can get pretty complicated.\nThankfully for a simple key-value database, we can focus on the first \u0026amp; the last parts to create a fully functional key-value store. And given my familiarity with B+Tree, it was an obvious choice for data access \u0026amp; organization to allow me to focus on the on-disk aspect. I would definitely\nRepresenting a database connection # To begin with, we create a struct that encompasses all the necessary attributes for a Kagi DB connection. First thing is the DB file itself. Second, we track the root node for our B+Tree. We also have a single mutex to make sure that changes to the DB file are synchronized. Lastly we have our loggers, one for general info, the other for fatal errors. With that we have this representation for a DB connection\ntype DB_CONNECTION struct { sync.Mutex file *os.File filePath string root *Node count uint32 infoLogger *log.Logger errorLogger *log.Logger } Hear me out, B+Tree \u0026hellip; but on-disk.🤯 # Since our B+Tree will be on disk, we have to deal with 2 integral changes to the traditional in-memory representation. The first is offsets instead of pointers, and the second is working with bytes instead of abstractions. Unix has a block size of 4096 bytes, as such we structure our B+Tree around that block size. To start off I settled on an Order=20 and a Degree=10 (Order is our upper limit for a Node\u0026rsquo;s bucket, and Degree is our lower limit) for the B+Tree. So this means we can fit a max of 19 key-value pairs in a 4KB node.\nHere is an early sketch of a Node as I was considering what headers are necessary, and how many bytes each part required. All elements of the header use either uint16 (2 bytes) or uint32 (4 bytes) as there is no need for negatives for boolean flags or for counts.\nAfter deciding on the headers, I settled on about 192 bytes for the key-value pairs. Where keys ar 48 bytes and values are 144 bytes. With that here is the set of constraints utilized in Kagi to make sure we are maximizing what we can get out of the 4KB blocks.\nconst ( Order int32 = 20 // the upper limit for number of keys/values that node can hold Degree int32 = 10 // the lower limit for number of keys/values that node must hold BlockSize int32 = 4096 // max size of a node Int32Size int32 = 4 // size of uint32 used for offsets in node Int16Size int32 = 2 // size of uint16 used for flags and counts in nodes MaxKeySize int32 = 48 MaxValueSize int32 = 144 ) Constructing our node # With our byte constraints set, we can setup our Node structure. One of the more confusing aspects of this project was in fact managing all the offsets necessary. In the end, it took some trial \u0026amp; error to figure out which offsets were necessary, and which were just waste of space.\nAs in all B+Trees, we will have two types of nodes:\nBranching Node: holds a bucket of keys to direct us in the tree Leaf Node: holds a bucket of key-value pairs Both type of nodes have the same headers, but different buckets. These headers are used to tell us about the state (leaf count, deleted marker, etc.). If a node ever has child offsets then we know that we it is a branching node, otherwise it\u0026rsquo;s a leaf node.\ntype Node struct { //----Header---- // Flags isRoot uint16 isDeleted uint16 // Counts dbCount uint32 numKeys uint16 numLeaves uint16 // Offsets offset uint32 parentOffset uint32 childOffsets []uint32 // ------------- // branching node keys [][]byte // leaf nodes leaves []*Leaf } type Leaf struct { // data key []byte value []byte } But first, some tests # With the Node struct done, we can start on B+Tree operations. But first, tests. Below is a simple test for setting/getting 100 key-value pairs. The test generates random values to use for key \u0026amp; value, each 5 bytes.\nfunc TestSet100Keys(t *testing.T) { db := Open(testClearOptions) rand.Seed(time.Now().UnixNano()) seq := randSeq(1000) for i := 0; i \u0026lt; 1000; i += 10 { k := seq[i : i+5] v := seq[i+5 : i+10] err := db.Set(k, v) db.logError(err) } db.Close() } func TestGet100Keys(t *testing.T) { ... // Same as test for Set ... for i := 0; i \u0026lt; 1000; i += 10 { k := seq[i : i+5] v := seq[i+5 : i+10] found, err2 := db.Get(k) if found != v { t.Error(err2) t.Errorf(`test %d, actual: \u0026#34;%s\u0026#34;, expected: \u0026#34;%s\u0026#34;`, i/10, found, v) } } db.Close() } Now, thanks to TDD, we work on flipping those failing tests upside down.\nTraversing the on-disk tree # First operation up is tree traversal. Traversing the on-disk B+tree is pretty similar to in-memory, with a slight twist, offsets. You start at the root node (which the DB always knows), then recursively traverse using child offsets - instead of pointers - with keys as direction markers. Below is a representation of our B+Tree.\nUpdating the on-disk tree # Now that we can freely traverse our tree, we move onto updating it. For adding a new key-value pair, we first find which node it fits into by traversing the tree. Once we insert the new data in the node\u0026rsquo;s bucket, there are 2 scenarios:\nIf it\u0026rsquo;s not full, then we are done. If it\u0026rsquo;s full, then we have to split the node up. Splitting a node up # If we ever fill up a node (number of keys equal Order), we split up the node into 3 nodes. In summary follow this basic algorithm:\nSplits leaves into 3 parts. Middle becomes a branching node. Left \u0026amp; Right become child nodes of the Middle. Add middle node as child to parent. If parent is now full, split parent node as well. Run it back ✔️ # With the traversal \u0026amp; updates for our B+Tree implemented - code too long to showcase here - we can now run our tests (added a test to make sure everything was saved on exit) to get those sweet, sweet passes.\nOn next episode # We got just two things left, and we got a fully functional key-value database, deletion \u0026amp; index maintenance. I plan to wrap up this features soon, and perhaps add a \u0026ldquo;Space\u0026rdquo; functionality to group a set of keys together. But this we\u0026rsquo;ll have to do for now.\nIf you\u0026rsquo;ve read this far, first of all thank you. Second, if you have any feedback please do feel free to hit me up about it, I\u0026rsquo;d really appreciate it.\nResources: # A fantastic B+Tree visualizer: https://www.cs.usfca.edu/~galles/visualization/BPlusTree.html Chapter from \u0026ldquo;Database Internals\u0026rdquo; by Alex Petrov on implementing on-disk B+Tree: https://www.oreilly.com/library/view/database-internals/9781492040330/ch04.html Look into B+Tree indexing in Oracle: http://web.archive.org/web/20161221112438/http://www.toadworld.com/platforms/oracle/w/wiki/11001.oracle-b-tree-index-from-the-concept-to-internals ","content_html":"\u003cp\u003eThis will be more of a high-level technical overview of key-value database design using notes from my research while working on \u003ca href=\"https://github.com/amohamed11/kagi\"\u003eKagi\u003c/a\u003e, a on-disk key-value store.\u003c/p\u003e\n\u003cp\u003eAt the bottom are some fantastic resources for better understanding B+Trees and Key-Value store design, definitely check those out.\u003c/p\u003e\n\u003ch2 id=\"why-build-kagi\"\u003eWhy build Kagi? \u003ca href=\"#why-build-kagi\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAbout a year and a half ago, I decided to take Databases II. And as soon as the first assignment, a run-of-the-mill embedded SQL assignment, was released, I remember thinking \u0026ldquo;why can\u0026rsquo;t we just create a database instead\u0026rdquo;. So this past holiday, I thought I\u0026rsquo;d finally sit down and do just that.\u003c/p\u003e\n\u003ch2 id=\"database-design-and-why-a-kv-database\"\u003eDatabase design, and why a KV database? \u003ca href=\"#database-design-and-why-a-kv-database\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003e\u003cimg src=\"/img/kv.png\" alt=\"Representation of a Key-Value database\" loading=\"lazy\" /\u003e\n\u003c/p\u003e\n\u003cp\u003eFor this, ironically, I ended up pulling out my lecture notes from said Databases II class. Despite the dreadful assignments, this course did touch on some essential concepts of database design. Here is the bullet-point summary:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eAccessing the data (on-disk vs in-memory, ordered vs unordered, etc.)\u003c/li\u003e\n\u003cli\u003eExecuting queries (SQL compiler, query optimizer, etc.)\u003c/li\u003e\n\u003cli\u003eManaging transactions (ACID, scheduling, batching, etc.)\u003c/li\u003e\n\u003cli\u003eMaintaining the database (index updates, tombstone cleanup, etc.)\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eAs you can already probably tell, databases can get pretty complicated.\u003c/p\u003e\n\u003cp\u003eThankfully for a simple key-value database, we can focus on the first \u0026amp; the last parts to create a fully functional key-value store. And given my familiarity with B+Tree, it was an obvious choice for data access \u0026amp; organization to allow me to focus on the on-disk aspect. I would definitely\u003c/p\u003e\n\u003ch2 id=\"representing-a-database-connection\"\u003eRepresenting a database connection \u003ca href=\"#representing-a-database-connection\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eTo begin with, we create a struct that encompasses all the necessary attributes for a Kagi DB connection. First thing is the DB file itself. Second, we track the root node for our B+Tree. We also have a single mutex to make sure that changes to the DB file are synchronized. Lastly we have our loggers, one for general info, the other for fatal errors. With that we have this representation for a DB connection\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-go\" data-lang=\"go\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003etype\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003eDB_CONNECTION\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003estruct\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e{\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003esync.Mutex\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003efile\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e*os.File\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003efilePath\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003estring\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eroot\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e*Node\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003ecount\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003euint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003einfoLogger\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e*log.Logger\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eerrorLogger\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e*log.Logger\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e}\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"hear-me-out-btree--but-on-disk\"\u003eHear me out, B+Tree \u0026hellip; but on-disk.🤯 \u003ca href=\"#hear-me-out-btree--but-on-disk\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eSince our B+Tree will be on disk, we have to deal with 2 integral changes to the traditional in-memory representation. The first is offsets instead of pointers, and the second is working with bytes instead of abstractions. Unix has a block size of \u003ccode\u003e4096 bytes\u003c/code\u003e, as such we structure our B+Tree around that block size. To start off I settled on an \u003ccode\u003eOrder=20\u003c/code\u003e and a \u003ccode\u003eDegree=10\u003c/code\u003e (Order is our upper limit for a Node\u0026rsquo;s bucket, and Degree is our lower limit) for the B+Tree. So this means we can fit a max of 19 key-value pairs in a 4KB node.\u003c/p\u003e\n\u003cp\u003eHere is an early sketch of a Node as I was considering what headers are necessary, and how many bytes each part required. All elements of the header use either \u003ccode\u003euint16\u003c/code\u003e (2 bytes) or \u003ccode\u003euint32\u003c/code\u003e (4 bytes) as there is no need for negatives for boolean flags or for counts.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"/img/node.png\" alt=\"Representation of a Node\" loading=\"lazy\" /\u003e\n\u003c/p\u003e\n\u003cp\u003eAfter deciding on the headers, I settled on about 192 bytes for the key-value pairs. Where keys ar 48 bytes and values are 144 bytes. With that here is the set of constraints utilized in Kagi to make sure we are maximizing what we can get out of the 4KB blocks.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-go\" data-lang=\"go\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003econst\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e(\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eOrder\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003eint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e20\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// the upper limit for number of keys/values that node can hold\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eDegree\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003eint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e10\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// the lower limit for number of keys/values that node must hold\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eBlockSize\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003eint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e4096\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// max size of a node\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eInt32Size\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003eint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e4\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// size of uint32 used for offsets in node\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eInt16Size\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003eint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e2\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// size of uint16 used for flags and counts in nodes\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eMaxKeySize\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003eint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e48\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eMaxValueSize\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003eint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e144\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e)\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"constructing-our-node\"\u003eConstructing our node \u003ca href=\"#constructing-our-node\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWith our byte constraints set, we can setup our Node structure. One of the more confusing aspects of this project was in fact managing all the offsets necessary. In the end, it took some trial \u0026amp; error to figure out which offsets were necessary, and which were just waste of space.\u003c/p\u003e\n\u003cp\u003eAs in all B+Trees, we will have two types of nodes:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eBranching Node: holds a bucket of keys to direct us in the tree\u003c/li\u003e\n\u003cli\u003eLeaf Node: holds a bucket of key-value pairs\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eBoth type of nodes have the same headers, but different buckets. These headers are used to tell us about the state (leaf count, deleted marker, etc.). If a node ever has child offsets then we know that we it is a branching node, otherwise it\u0026rsquo;s a leaf node.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-go\" data-lang=\"go\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003etype\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003eNode\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003estruct\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e{\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e\u003cspan style=\"color:#ff0007;background-color:#0f140f;font-weight:bold;font-style:italic\"\u003e//----Header----\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// Flags\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eisRoot\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003euint16\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eisDeleted\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003euint16\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// Counts\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003edbCount\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003euint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003enumKeys\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003euint16\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003enumLeaves\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003euint16\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// Offsets\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eoffset\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003euint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eparentOffset\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003euint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003echildOffsets\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e[]\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003euint32\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// -------------\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// branching node\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003ekeys\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e[][]\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003ebyte\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// leaf nodes\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eleaves\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e[]*Leaf\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e}\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003etype\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003eLeaf\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003estruct\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e{\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// data\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003ekey\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e[]\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003ebyte\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003evalue\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e[]\u003cspan style=\"color:#cdcaa9;font-weight:bold\"\u003ebyte\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e}\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"but-first-some-tests\"\u003eBut first, some tests \u003ca href=\"#but-first-some-tests\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWith the Node struct done, we can start on B+Tree operations. But first, tests. Below is a simple test for setting/getting 100 key-value pairs. The test generates random values to use for key \u0026amp; value, each 5 bytes.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#fff;background-color:#111;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-go\" data-lang=\"go\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003efunc\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eTestSet100Keys\u003c/span\u003e(t\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e*testing.T)\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e{\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003edb\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eOpen\u003c/span\u003e(testClearOptions)\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003erand.\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eSeed\u003c/span\u003e(time.\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eNow\u003c/span\u003e().\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eUnixNano\u003c/span\u003e())\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003eseq\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003erandSeq\u003c/span\u003e(\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e1000\u003c/span\u003e)\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003efor\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ei\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e0\u003c/span\u003e;\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ei\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u0026lt;\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e1000\u003c/span\u003e;\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ei\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e+=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e10\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e{\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\t\u003c/span\u003ek\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003eseq[i\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ei+\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e5\u003c/span\u003e]\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\t\u003c/span\u003ev\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003eseq[i+\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e5\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ei+\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e10\u003c/span\u003e]\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\t\u003c/span\u003eerr\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003edb.\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eSet\u003c/span\u003e(k,\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ev)\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\t\u003c/span\u003edb.\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003elogError\u003c/span\u003e(err)\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e}\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003edb.\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eClose\u003c/span\u003e()\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e}\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003efunc\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eTestGet100Keys\u003c/span\u003e(t\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e*testing.T)\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e{\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e...\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e\u003cspan style=\"color:#080;background-color:#0f140f;font-style:italic\"\u003e// Same as test for Set\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e...\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003efor\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ei\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e0\u003c/span\u003e;\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ei\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u0026lt;\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e1000\u003c/span\u003e;\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ei\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e+=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e10\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e{\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\t\u003c/span\u003ek\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003eseq[i\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ei+\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e5\u003c/span\u003e]\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\t\u003c/span\u003ev\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003eseq[i+\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e5\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ei+\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e10\u003c/span\u003e]\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\t\u003c/span\u003efound,\u003cspan style=\"color:#888\"\u003e \u003c/span\u003eerr2\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e:=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003edb.\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eGet\u003c/span\u003e(k)\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\t\u003c/span\u003e\u003cspan style=\"color:#fb660a;font-weight:bold\"\u003eif\u003c/span\u003e\u003cspan style=\"color:#888\"\u003e \u003c/span\u003efound\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e!=\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ev\u003cspan style=\"color:#888\"\u003e \u003c/span\u003e{\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\t\t\u003c/span\u003et.\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eError\u003c/span\u003e(err2)\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\t\t\u003c/span\u003et.\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eErrorf\u003c/span\u003e(\u003cspan style=\"color:#0086d2\"\u003e`test %d, actual: \u0026#34;%s\u0026#34;, expected: \u0026#34;%s\u0026#34;`\u003c/span\u003e,\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ei/\u003cspan style=\"color:#0086f7;font-weight:bold\"\u003e10\u003c/span\u003e,\u003cspan style=\"color:#888\"\u003e \u003c/span\u003efound,\u003cspan style=\"color:#888\"\u003e \u003c/span\u003ev)\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\t\u003c/span\u003e}\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003e}\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#888\"\u003e\t\u003c/span\u003edb.\u003cspan style=\"color:#ff0086;font-weight:bold\"\u003eClose\u003c/span\u003e()\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e}\u003cspan style=\"color:#888\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eNow, thanks to TDD, we work on flipping those failing tests upside down.\u003c/p\u003e\n\u003ch2 id=\"traversing-the-on-disk-tree\"\u003eTraversing the on-disk tree \u003ca href=\"#traversing-the-on-disk-tree\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eFirst operation up is tree traversal. Traversing the on-disk B+tree is pretty similar to in-memory, with a slight twist, offsets. You start at the root node (which the DB always knows), then recursively traverse using child offsets - instead of pointers - with keys as direction markers. Below is a representation of our B+Tree.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"/img/tree.png\" alt=\"Representation of the tree\" loading=\"lazy\" /\u003e\n\u003c/p\u003e\n\u003ch2 id=\"updating-the-on-disk-tree\"\u003eUpdating the on-disk tree \u003ca href=\"#updating-the-on-disk-tree\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eNow that we can freely traverse our tree, we move onto updating it. For adding a new key-value pair, we first find which node it fits into by traversing the tree. Once we insert the new data in the node\u0026rsquo;s bucket, there are 2 scenarios:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eIf it\u0026rsquo;s not full, then we are done.\u003c/li\u003e\n\u003cli\u003eIf it\u0026rsquo;s full, then we have to split the node up.\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch3 id=\"splitting-a-node-up\"\u003eSplitting a node up \u003ca href=\"#splitting-a-node-up\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eIf we ever fill up a node (number of keys equal \u003ccode\u003eOrder\u003c/code\u003e), we split up the node into 3 nodes. In summary follow this basic algorithm:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eSplits leaves into 3 parts.\u003c/li\u003e\n\u003cli\u003eMiddle becomes a branching node.\u003c/li\u003e\n\u003cli\u003eLeft \u0026amp; Right become child nodes of the Middle.\u003c/li\u003e\n\u003cli\u003eAdd middle node as child to parent.\u003c/li\u003e\n\u003cli\u003eIf parent is now full, split parent node as well.\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"run-it-back-\"\u003eRun it back ✔️ \u003ca href=\"#run-it-back-\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWith the traversal \u0026amp; updates for our B+Tree implemented - code too long to showcase here - we can now run our tests (added a test to make sure everything was saved on exit) to get those sweet, sweet passes.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"/img/tests_pass.png\" alt=\"Passing tests\" loading=\"lazy\" /\u003e\n\u003c/p\u003e\n\u003ch2 id=\"on-next-episode\"\u003eOn next episode \u003ca href=\"#on-next-episode\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWe got just two things left, and we got a fully functional key-value database, deletion \u0026amp; index maintenance. I plan to wrap up this features soon, and perhaps add a \u0026ldquo;Space\u0026rdquo; functionality to group a set of keys together. But this we\u0026rsquo;ll have to do for now.\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\nIf you\u0026rsquo;ve read this far, first of all thank you. Second, if you have any feedback please do feel free to hit me up about it, I\u0026rsquo;d really appreciate it.\u003c/p\u003e\n\u003ch2 id=\"resources\"\u003eResources: \u003ca href=\"#resources\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eA fantastic B+Tree visualizer: \u003ca href=\"https://www.cs.usfca.edu/~galles/visualization/BPlusTree.html\"\u003ehttps://www.cs.usfca.edu/~galles/visualization/BPlusTree.html\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eChapter from \u0026ldquo;Database Internals\u0026rdquo; by Alex Petrov on implementing on-disk B+Tree: \u003ca href=\"https://www.oreilly.com/library/view/database-internals/9781492040330/ch04.html\"\u003ehttps://www.oreilly.com/library/view/database-internals/9781492040330/ch04.html\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eLook into B+Tree indexing in Oracle: \u003ca href=\"http://web.archive.org/web/20161221112438/http://www.toadworld.com/platforms/oracle/w/wiki/11001.oracle-b-tree-index-from-the-concept-to-internals\"\u003ehttp://web.archive.org/web/20161221112438/http://www.toadworld.com/platforms/oracle/w/wiki/11001.oracle-b-tree-index-from-the-concept-to-internals\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n","url":"http://anecdotalthoughts.com/posts/building-a-lightweight-key-value-database-in-go/","image":"http://anecdotalthoughts.com/photos/","banner_image":"http://anecdotalthoughts.com/photos/","date_published":"18056-18-09T512:1818:00+00:00","date_modified":"18056-18-09T512:1818:00+00:00","author":{"name":"Anas Mohamed","url":"http://anecdotalthoughts.com/"}},{"id":"32db984adfca3bde690bce1167a7f27af4a4c07f","title":"Things I Liked in 2020 (So far)","summary":"Ah nothing but good times.","content_text":" Updated 02/02/2022:\nRemoved \u0026ldquo;to be continued\u0026rdquo; section, and a talk by Robert C. Martin. Turns out he\u0026rsquo;s quite the asshole.\nThis year has been something. Still though, I want to start of this blogging thing on a good note. So, here is a random compilation of things that I got to enjoy in 2020 so far.\nProgramming Talks # Every once and a while I go on a tech talk binge. Below are some of my recent favorites. Some I have watched for the first time in this year regardless of when they came out.\nThe Art of Code # Like most programmers, I think of coding only in the context of productivity. Whether it be work, school, or side projects, they are done for an external benefit or to solve a problem. Dylan Beattie instead explores coding as an art medium. He goes through many examples of the wild things that people have created. My favorite being the Quine Relay, a program that generates a program in different programming language which itself does the same for a continuous cycle that cover 128 programming languages.\nThis talk is just non-stop mind-blowing demonstrations of human creativity in this tech era.\nCompilers For Nothing, Executables For Free! # Lastly, a short (coming in at 8 minutes only) and fun talk that descends down the rabbit-hole that is specializers (specialized interpreters), all in the name of what\u0026rsquo;s right \u0026amp; Vim.\nVideo Games # Most of these games came out last year, but I was only able to get around to them this year. Also as a rule, I decided to only include single-player games, as such no Valorant (which I sunk an ungodly amount of time into), Among Us, etc.\nOuter Wilds # With the risk of sounding hyperbolic, this is probably my favorite game of all-time now. And that statement has remained the same since I finished it in January. The way Outer Wilds interweaves story and exploration in such a unique and delightful manner is an experience to be had.\nPersonally I\u0026rsquo;ve never been one for exploration, but that was because usually that directly interfered with my progress in the story. Exploring for the sake of exploring - though fun - just doesn;t strike me as enjoyable as unpeeling a good storyline is. And Outer Wilds manages to combine exploration directly with the story. So this way, you are rewarded two-fold when you explore, once for the amazing and unique planetary design that is a scene to behold, and once for the story bits that you pick up on along the way.\nOverall, Outer Wilds is just an absolute masterpiece in narrative and non-linear story-telling. And to me it is a testement to the unique storytelling capabilities of video games. I simply cannot recommend this game enough.\nBaba Is You # Also an Indie darling from 2019, Baba Is You manages to make puzzle games - typically my least favorite genre - fun for me. A simple but mind-bogglingly innovative gameplay coupled with tight level design and just the most adorable protagonist, what else can you ask for.\nBaba Is You isn\u0026rsquo;t the type of game you \u0026ldquo;finish\u0026rdquo; thanks to the plethora of levels (about 200) or at least that is if you are as bad as I am with puzzles.\nInstead it is the kind of game that you pop on every once and a while and solve as few levels before your brain melts and you get stuck and call it a night lest you give into googling a walkthrough. But god is it satisfying to figure out these puzzles, they truly force you to think outside the box and it just so good at it doing it consistently.\nHades # Lastly, Hades, a game that actually this year (if you ignore that it was in early access for a year).\nI\u0026rsquo;m still working through this game currently, but I can sing it nothing but praises. This game is truly embodies the meaning of polish. Every single part is just fantastic, from combat, voice-acting, to character design and storytelling. Hades just goes above and beyond to rewarding you for having a good or a bad run. The timings of when you unlock new upgrades is so well tuned, that you never feel stuck for too long. And when you die, you have often new dialogue and (sometimes) story to look forward to.\nIn a twisted way, Supergiant Games made dying in Hades somehow a truly enjoyable experience. If that sounds up your ally, than definitely give this a try, cause I truly believe there is something in this game for everyone to enjoy.\n","content_html":"\u003chr\u003e\n\u003cp\u003e\u003cem\u003eUpdated 02/02/2022\u003c/em\u003e:\u003cbr\u003e\nRemoved \u0026ldquo;to be continued\u0026rdquo; section, and a talk by Robert C. Martin. Turns out he\u0026rsquo;s \u003cem\u003equite\u003c/em\u003e the asshole.\u003c/p\u003e\n\u003chr\u003e\n\u003cp\u003eThis year has been something. Still though, I want to start of this blogging thing on a good note. So, here is a random compilation of things that I got to enjoy in 2020 so far.\u003c/p\u003e\n\u003c!-- raw HTML omitted --\u003e\n\u003ch2 id=\"programming-talks\"\u003eProgramming Talks \u003ca href=\"#programming-talks\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eEvery once and a while I go on a tech talk binge. Below are some of my recent favorites. Some I have watched for the first time in this year regardless of when they came out.\u003c/p\u003e\n\u003ch3 id=\"the-art-of-code\"\u003eThe Art of Code \u003ca href=\"#the-art-of-code\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003ca href=\"https://www.youtube.com/watch?v=6avJHaC3C2U\"\u003e\u003cimg src=\"https://i1.ytimg.com/vi/6avJHaC3C2U/hqdefault.jpg\" alt=\"Thumbnail for The Art of Code video\" loading=\"lazy\" /\u003e\n\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eLike most programmers, I think of coding only in the context of productivity. Whether it be work, school, or side projects, they are done for an external benefit or to solve a problem. Dylan Beattie instead explores coding as an art medium. He goes through many examples of the wild things that people have created. My favorite being the Quine Relay, a program that generates a program in different programming language which itself does the same for a continuous cycle that cover 128 programming languages.\u003cbr\u003e\nThis talk is just non-stop mind-blowing demonstrations of human creativity in this tech era.\u003c/p\u003e\n\u003ch3 id=\"compilers-for-nothing-executables-for-free\"\u003eCompilers For Nothing, Executables For Free! \u003ca href=\"#compilers-for-nothing-executables-for-free\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003ca href=\"https://www.youtube.com/watch?v=Dcug6bu_wUE\"\u003e\u003cimg src=\"https://i.ytimg.com/vi_webp/Dcug6bu_wUE/maxresdefault.webp\" alt=\"Thumbnail for Compilers For Nothing, Executables For Free! Video\" loading=\"lazy\" /\u003e\n\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eLastly, a short (coming in at 8 minutes only) and fun talk that descends down the rabbit-hole that is specializers (specialized interpreters), all in the name of what\u0026rsquo;s right \u0026amp; Vim.\u003c/p\u003e\n\u003c!-- raw HTML omitted --\u003e\n\u003ch2 id=\"video-games\"\u003eVideo Games \u003ca href=\"#video-games\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMost of these games came out last year, but I was only able to get around to them this year. Also as a rule, I decided to only include single-player games, as such no Valorant (which I sunk an ungodly amount of time into), Among Us, etc.\u003c/p\u003e\n\u003ch3 id=\"outer-wilds\"\u003eOuter Wilds \u003ca href=\"#outer-wilds\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eWith the risk of sounding hyperbolic, this is probably my favorite game of all-time now. And that statement has remained the same since I finished it in January. The way Outer Wilds interweaves story and exploration in such a unique and delightful manner is an experience to be had.\u003c/p\u003e\n\u003cp\u003ePersonally I\u0026rsquo;ve never been one for exploration, but that was because usually that directly interfered with my progress in the story. Exploring for the sake of exploring - though fun - just doesn;t strike me as enjoyable as unpeeling a good storyline is. And Outer Wilds manages to combine exploration directly with the story. So this way, you are rewarded two-fold when you explore, once for the amazing and unique planetary design that is a scene to behold, and once for the story bits that you pick up on along the way.\u003c/p\u003e\n\u003cp\u003eOverall, Outer Wilds is just an absolute masterpiece in narrative and non-linear story-telling. And to me it is a testement to the unique storytelling capabilities of video games. I simply cannot recommend this game enough.\u003c/p\u003e\n\u003ch3 id=\"baba-is-you\"\u003eBaba Is You \u003ca href=\"#baba-is-you\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eAlso an Indie darling from 2019, Baba Is You manages to make puzzle games - typically my least favorite genre - fun for me. A simple but mind-bogglingly innovative gameplay coupled with tight level design and just the most adorable protagonist, what else can you ask for.\u003cbr\u003e\nBaba Is You isn\u0026rsquo;t the type of game you \u0026ldquo;finish\u0026rdquo; thanks to the plethora of levels (about 200) or at least that is if you are as bad as I am with puzzles.\u003cbr\u003e\nInstead it is the kind of game that you pop on every once and a while and solve as few levels before your brain melts and you get stuck and call it a night lest you give into googling a walkthrough. But god is it satisfying to figure out these puzzles, they truly force you to think outside the box and it just so good at it doing it consistently.\u003c/p\u003e\n\u003ch3 id=\"hades\"\u003eHades \u003ca href=\"#hades\" class=\"hash\"\u003e#\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eLastly, Hades, a game that actually this year (if you ignore that it was in early access for a year).\u003cbr\u003e\nI\u0026rsquo;m still working through this game currently, but I can sing it nothing but praises. This game is truly embodies the meaning of polish. Every single part is just fantastic, from combat, voice-acting, to character design and storytelling. Hades just goes above and beyond to rewarding you for having a good or a bad run. The timings of when you unlock new upgrades is so well tuned, that you never feel stuck for too long. And when you die, you have often new dialogue and (sometimes) story to look forward to.\u003cbr\u003e\nIn a twisted way, Supergiant Games made dying in Hades somehow a truly enjoyable experience. If that sounds up your ally, than definitely give this a try, cause I truly believe there is something in this game for everyone to enjoy.\u003c/p\u003e\n\u003c!-- raw HTML omitted --\u003e\n","url":"http://anecdotalthoughts.com/posts/things-liked-2020/","image":"http://anecdotalthoughts.com/photos/","banner_image":"http://anecdotalthoughts.com/photos/","date_published":"11116-11-09T110:1111:00+00:00","date_modified":"11116-11-09T110:1111:00+00:00","author":{"name":"Anas Mohamed","url":"http://anecdotalthoughts.com/"}}]}