![\"The](\"https://rgbcu.be/assets/images/turkish-q-i.webp\")
So, you may have seen the HTMNIX project\nI've been working on the last few weeks. If not, no worries. Here is a Nix\nsnippet that uses it:
\n<html>\n <head>\n <title>\"Hello, Internet!\"<.title>\n <.head>\n <body>\n <p>\"Yep, this is 100% Nix!\"<.p>\n\n <img.>{src=\"/foo.png\"; alt=\"Attributes also work!\";}\n <.body>\n<.html>\nYou are probably thinking furiously right now, maybe you've noticed something:
\n\n\nAha! In Nix,
\n<foo>is used to find stuff from the Nix path like so:\nimport <nixpkgs> {}\nThat means you have to add hundreds of elements to your Nix Path to make this\nwork?
\n
You are somewhat correct. But not quite.
\nNix <foo> expressions actually boil down to a call of the builtin\n__findFile, like so:
# nix-instantiate --parse --expr \"<foo>\"\n\n(__findFile __nixPath \"foo\")\n\n\nIn case you didn't know,\n
\nnix-instantiate\nis a nice tool to see what your Nix code is desugared and un-precedence'd\ninto.
Aha! So this means we can override the builtin __findFile and put whatever we\nwould like in its place. So this will work:
let\n __findFile = nixPath: name: {\n content = \"<${name}>\";\n };\nin\n<foo>\nEvaluating this (by running nix eval -f test.nix), we get\n{ content = \"<foo>\"; }
So, then. How do we make it work for multiple tags, all coming after one another\n(and attribute sets, strings, etc.)?
\nAnother hack! We need to set the\nmagic __functor attribute of the\nattrset we return, so we can call our set and have it store the tags inside it\n(while also preserving its callability!).
We can do that like so:
\nlet\n __findFile = nixPath: name: {\n content = \"<${name}>\";\n __functor = self: next: self // {\n content = self.content + next;\n };\n };\nin\n<foo>\n\"bar\"\n\"baz\"\nGreat news! When we evaluate this, we get\n{ __functor = <LAMBDA>; content = \"<foo>barbaz\"; }.
We can also add a case to check if the next element is a tag, and use its\ncontent if it is:
\nlet\n __findFile = nixPath: name: {\n content = \"<${name}>\";\n __functor = self: next: self // {\n content = self.content + (if next ? content then next.content else next);\n };\n };\nin\n<foo>\n\"bar\"\n\"baz\"\n<endfoo>\nEnter another hack! We can utilize the outPath property that exists on\nderivations and gets returned whenever you call toString with an attrset that\nhas the property to make our code a little simpler:
let\n __findFile = nixPath: name: {\n outPath = \"<${name}>\";\n __functor = self: next: self // {\n outPath = self.outPath + toString next;\n };\n };\nin\n<foo>\n\"bar\"\n\"baz\"\n123\n<endfoo>\nWe also got support for other types for free, as well!
\nThese are all the hidden builtins that\nHTMNIX depends on and extends upon, making\nHTML in Nix an actually usable reality. It also has extra logic like turning\nattribute sets into HTML tags, which is fairly trivial compared to actually\ndiscovering these hidden builtins in the first place.
\nYou can read more about it in the project's README and see\nan example site using it.
\nI might even try to port this site to HTMNIX to ensure it is usable with more\ncomplex setups :-)
\nSoon, maybe...
\nThanks for reading my first ever proper blog post! :-)
\n\n\n \n","tags":["html","nix/os"],"date_published":"2024-03-04T00:00:00.000Z"},{"id":"https://rgbcu.be/blog/cosmic-drift/","url":"https://rgbcu.be/blog/cosmic-drift/","title":"Cosmic Drift","content_html":"\nSo, every day I wake up at 6:55, get dressed by 7, walk to the bus stop by 7:13\nand board the bus around 7:17. Today was different.
\nMy alarm that I have set for 6:55 rang at 7:12 and as a result, I missed the\nbus.
\n\n\nNo, I didn't sleep in. There was no note in the UI saying this was a repeat\nalarm, which there is if you snooze it or let it expire.
\n
Surely something was happening. Machines don't break whenever they want, they're\nmostly deterministic. And I doubt Samsung engineers wrote code to delay the\nalarm by 12 minutes on the date after April 1st.
\nSo, what was happening? I entered the Python repl to test out a theory:
\nalarm_time = 6*60*60 + 55*60\nalarm_time_real = 7*60*60 + 12*60\nalarm_time_real - alarm_time\n1020\nThe total time was off by about 1020 seconds. Give or take 60, as my phone\ndoesn't display the seconds of the time.
\n\n\nSince I'm using a Samsung SM-B310E, I assumed it uses seconds to store the\ntime. You can't even see seconds noted anywhere so I feel this is a normal\nassumption. Even if it is false, the math still adds up for milliseconds.
\n
Wow, I thought. That's really close to 1024 (which is 2 to the power of 10).\nMaybe the 11th bit got flipped, making it increment 1024?
\nbin(alarm_time)\n'0b110000101000100'\nalarm_time |= 1 << 10\nbin(alarm_time)\n'0b110010101000100'\nalarm_time - alarm_time_real\n-4\nAha! So the 11th bit got flipped by something. And that something was probably a\ncosmic ray.
\n\n\nEDIT: It was not a cosmic ray. As pointed out by\n@BenjaminRi's comment on lobste.rs\nit most likely was either a bug or storage corruption as the alarm ran late\nthe next day. You should still create more than one alarm just in case if you\nare using a phone prone to this, however.
\n
My main takeaway from this event is to create more than one alarm, for extra\nredundancy. Who knew being prone to sleeping in could save you from your alarm\ngetting shifted 12 minutes into the future :^).
\n\n\n \n","tags":["time","unix-timestamps"],"date_published":"2024-04-02T00:00:00.000Z"},{"id":"https://rgbcu.be/blog/swap-i/","url":"https://rgbcu.be/blog/swap-i/","title":"Swap the ı and i key on your keyboard for faster modal editing","content_html":"\nIf you have ever used a Turkish-Q keyboard in combination with a modal editor\nbefore, you might have noticed that the i key is pretty far off to the side:
This blog post will guide you on how to swap the ı key with the i key (but\nnot the I and İ keys). This will be a great change if you write primarily in\nEnglish but need the Turkish symbols sometimes.
\n\nThere is
\ntr(us)in thexkeyboard_configpackage that does something\nsimilar to this:\n// The basic Turkish layout with \"i\" and \"ı\" swapped.\n// Originally by Ali Riza KESKIN <parduscix@yandex.ru>, 2021.\npartial\nxkb_symbols \"us\" {\n\n include \"tr(basic)\"\n\n name[Group1]=\"Turkish (i and ı swapped)\";\n\n key <AC11> { type[group1] = \"FOUR_LEVEL_SEMIALPHABETIC\",\n [ i, I, paragraph, none ]};\n key <AD08> { type[group1] = \"FOUR_LEVEL_SEMIALPHABETIC\",\n [ idotless, Iabovedot, apostrophe, dead_caron ]};\n};\nHowever, this only swaps the uppercase letters, so the
\nikey is unchanged\nbut the uppercase of that key isIlike in English. However, this is usually\nnot desired as this still reduces your typing speed (as theiIkey is too\nfar).
Let's create our own layout that does something similar but swaps the lowercase\nletters instead. Here is the code for that:
\ndefault partial\nxkb_symbols \"basic\" {\n include \"tr(basic)\"\n\n name[Group1]=\"Turkish (i and ı swapped correctly)\";\n\n key <AC11> { type[group1] = \"FOUR_LEVEL_SEMIALPHABETIC\",\n [ idotless, Iabovedot, paragraph , none ]};\n key <AD08> { type[group1] = \"FOUR_LEVEL_SEMIALPHABETIC\",\n [ i , I , apostrophe, dead_caron ]};\n};\nThe default key is needed because we are going to make this a standalone file.\nSave this to ~/.config/xkb/symbols/tr-swapped-i and you can tell your WM/DE to\nuse the tr-swapped-i XKB layout.
In Hyprland (The WM I use) you can do it like this:
\n# In ~/.config/hypr/hyprland.conf\ninput {\n kb_layout = \"tr-swapped-i\"\n}\nThat should swap the ı and i keys on your WM/DE successfully. However, we\nare not done yet.
The TTY (swap to TTY 2 by doing CTRL-ALT-F2) still doesn't use this layout,\nwhich is a problem. And it seems that the format the TTY uses is a little\ndifferent. Looking at the kbd package, it uses a format called .map.
Here is a .map file that overrides the trq layout and swaps the ı and i\nkeys:
include \"/usr/share/keymaps/i386/qwerty/trq.map\"\n\nkeycode 23 = i\n altgr keycode 23 = +icircumflex\n altgr shift keycode 23 = +Icircumflex\n\nkeycode 40 = +dotlessi +Idotabove\nSave it to the directory where you store your maps.
\nNote that the path /usr/share/keymaps might differ based on your distro. You\ncan check what it is it by doing man loadkeys and scrolling to the bottom.
After that, consult your distro's docs on how to change the system keyboard\nlocale.
\nThis is how it is done on NixOS:
\n{ pkgs, ... }: {\n console.keyMap = pkgs.writeText \"trq-swapped-i.map\" ''\n include \"${pkgs.kbd}/share/keymaps/i386/qwerty/trq.map\"\n\n keycode 23 = i\n altgr keycode 23 = +icircumflex\n altgr shift keycode 23 = +Icircumflex\n\n keycode 40 = +dotlessi +Idotabove\n '';\n}\nAnd this is how it is done on Arch:
\ncat << :end\ninclude \"/usr/share/kbd/keymaps/i386/qwerty/trq.map\"\n\nkeycode 23 = i\n altgr keycode 23 = +icircumflex\n altgr shift keycode 23 = +Icircumflex\n\nkeycode 40 = +dotlessi +Idotabove\n:end > /usr/share/kbd/keymaps/i386/qwerty/trq-swapped-i.map\n\nlocalectl set-keymap --no-convert tr-swapped-i\nAfter that, your TTY should also use this locale.
\nGotta love having to mess with deep parts of your system because the Turkish-Q\nlayout did the big mistake of reusing the i key as ı!
At least I'll type and edit faster now...
\nHere is the commit where I implemented this, might be worth a look if you use NixOS.
\n\n\n \n","tags":["localisation","modal-editors"],"date_published":"2024-05-20T00:00:00.000Z"},{"id":"https://rgbcu.be/blog/gitignore/","url":"https://rgbcu.be/blog/gitignore/","title":".gitignore is inherently Sisyphean","content_html":"\nYou just started a new project. You ran cargo init, poetry init and\ngo mod init.
Those commands created the necessary files to work, it also added the following\nlines to your .gitignore:
\ntarget\n__pycache__\nbin\nAll great. You continue implementing features, and when the time comes, you\npublish your project to your Git hosting platform of choice.
\nPeople start to get interested in your project. One even decides that he's going\nto implement a new feature! Literally free work done for you!
\nAlright. That person uses his code editor and tools bundled with his operating\nsystem to implement a very cool new feature. He then submits the merge request.
\nYou start reviewing the code and notice a file quite out of place: .DS_Store.\nYou ask the person what it is, he says he has no clue.
![\"Hundreds](\"https://rgbcu.be/assets/images/github-ds-store-mr-list.webp\")
Whatever. You just delete the file from the branch and add the file's name to\nthe repositories gitignore:
\ntarget\n__pycache__\nbin\n.DS_Store\nNice. Now the code is on master, and your repository only contains relevant\ninformation.
\nThen, someone using an IDE created using web technologies submits another merge\nrequest. You look at it, and see that there is a whole directory that is\nirrelevant. You tell that person to delete the directory from the branch and add\nit to the gitignore. The gitignore lives on:
\ntarget\n__pycache__\nbin\n.DS_Store\n.vscode\nThen, someone that uses IntelliJ IDEA commits five hundred XML files and the\n.idea directory. You repeat this process:
target\n__pycache__\nbin\n.DS_Store\n.vscode\n.idea\nYears pass. Now your gitignore is hundreds of lines long, yet people keep\naccidentally committing in test scripts, foo, a, qux, data.tar.gz, start.sh,\nbin-release, cat, asd, fgsgskfh.
\nHell. You feel like a mythic god undergoing punishment for cheating death and\ndeceiving the underworld.
\n![\"Sisyphus](\"https://rgbcu.be/assets/images/sisyphus-ds-store.webp\")
How will you escape this endless loop of ignoring files that sneak in? Maybe by\neducating every single merge request author? Nope, that definitely won't work,\nthere should be a way to automatically handle this with tooling, rather than\nsubjective human communication.
\nLuckily, you realize that you can turn the blacklist of files (the gitignore)\ninto a whitelist, by just ignoring everything and manually un-ignoring desired\nfiles. You change your gitignore to this:
\n*\n\n!.gitignore\n\n# whitelist `src` directories and their children, regardless of place\n!src/\n!src/**/\n!src/**/*.rs\n!Cargo.{toml,lock}\n\n# whitelist root `pysrc` directory\n!/pysrc/\n!/pysrc/*.py\n!pyproject.toml\n!poetry.lock\n\n!/cmd/\n!/cmd/*.go\n!main.go\n!go.{mod,sum}\n\n!/docs/\n!/docs/*.md\nNow, nobody can accidentally commit undesired files, as git automatically\nignores them all and only allows the files that are explicitly whitelisted. It's\nalso future proof, future proof until an IDE decides to use the src/ide.rs\nfile as a convenient way of storing project specific configuration. And\nhopefully that future never comes.
You feel relieved.
\n\n\n \n","tags":["vcs"],"date_published":"2024-09-30T00:00:00.000Z"},{"id":"https://rgbcu.be/blog/why-cores/","url":"https://rgbcu.be/blog/why-cores/","title":"Why more /sys/class/hwmon//temp_label than CPU cores?","content_html":"\nSo, I am currently working on a Linux tool+daemon to replace\nauto-cpufreq with a more\nefficient, tunable and observable alternative.
And obviously, to tune a CPU well, knowing how hot it is is a requirement.
\nThen, how do you actually see how hot your CPU is in Linux? Using\n/sys/class/hwmon of course.
This path contains hardware monitoring devices. In the laptop I'm testing this\non, hwmon4 under this directory corresponds to the CPU.
Let's run a tree:
/sys/class/hwmon/hwmon4 -> ../../devices/platform/coretemp.0/hwmon/hwmon4\n├── device -> ../../../coretemp.0\n│ ├── driver_override\n│ ├── hwmon\n│ ├── modalias\n│ ├── power\n│ ├── subsystem -> ../../../bus/platform\n│ └── uevent\n├── name\n├── power\n│ ├── autosuspend_delay_ms\n│ ├── control\n│ ├── runtime_active_time\n│ ├── runtime_status\n│ └── runtime_suspended_time\n├── subsystem -> ../../../../../class/hwmon [recursive, not followed]\n├── temp10_crit\n├── temp10_crit_alarm\n├── temp10_input\n├── temp10_label\n├── temp10_max\n├── temp14_crit\n├── temp14_crit_alarm\n├── temp14_input\n├── temp14_label\n├── temp14_max\n├── temp18_crit\n├── temp18_crit_alarm\n├── temp18_input\n├── temp18_label\n├── temp18_max\n├── temp1_crit\n├── temp1_crit_alarm\n├── temp1_input\n├── temp1_label\n├── temp1_max\n├── temp22_crit\n├── temp22_crit_alarm\n├── temp22_input\n├── temp22_label\n├── temp22_max\n├── temp26_crit\n├── temp26_crit_alarm\n├── temp26_input\n├── temp26_label\n├── temp26_max\n├── temp2_crit\n├── temp2_crit_alarm\n├── temp2_input\n├── temp2_label\n├── temp2_max\n├── temp30_crit\n├── temp30_crit_alarm\n├── temp30_input\n├── temp30_label\n├── temp30_max\n├── temp34_crit\n├── temp34_crit_alarm\n├── temp34_input\n├── temp34_label\n├── temp34_max\n├── temp35_crit\n├── temp35_crit_alarm\n├── temp35_input\n├── temp35_label\n├── temp35_max\n├── temp36_crit\n├── temp36_crit_alarm\n├── temp36_input\n├── temp36_label\n├── temp36_max\n├── temp37_crit\n├── temp37_crit_alarm\n├── temp37_input\n├── temp37_label\n├── temp37_max\n├── temp38_crit\n├── temp38_crit_alarm\n├── temp38_input\n├── temp38_label\n├── temp38_max\n├── temp39_crit\n├── temp39_crit_alarm\n├── temp39_input\n├── temp39_label\n├── temp39_max\n├── temp40_crit\n├── temp40_crit_alarm\n├── temp40_input\n├── temp40_label\n├── temp40_max\n├── temp41_crit\n├── temp41_crit_alarm\n├── temp41_input\n├── temp41_label\n├── temp41_max\n├── temp42_crit\n├── temp42_crit_alarm\n├── temp42_input\n├── temp42_label\n├── temp42_max\n├── temp43_crit\n├── temp43_crit_alarm\n├── temp43_input\n├── temp43_label\n├── temp43_max\n├── temp44_crit\n├── temp44_crit_alarm\n├── temp44_input\n├── temp44_label\n├── temp44_max\n├── temp45_crit\n├── temp45_crit_alarm\n├── temp45_input\n├── temp45_label\n├── temp45_max\n├── temp46_crit\n├── temp46_crit_alarm\n├── temp46_input\n├── temp46_label\n├── temp46_max\n├── temp47_crit\n├── temp47_crit_alarm\n├── temp47_input\n├── temp47_label\n├── temp47_max\n├── temp48_crit\n├── temp48_crit_alarm\n├── temp48_input\n├── temp48_label\n├── temp48_max\n├── temp49_crit\n├── temp49_crit_alarm\n├── temp49_input\n├── temp49_label\n├── temp49_max\n├── temp6_crit\n├── temp6_crit_alarm\n├── temp6_input\n├── temp6_label\n├── temp6_max\n└── uevent\nLet's cat all the _label files:
/sys/class/hwmon/hwmon4/temp1_label:\nPackage id 0\n/sys/class/hwmon/hwmon4/temp2_label:\nCore 0\n/sys/class/hwmon/hwmon4/temp6_label:\nCore 4\n/sys/class/hwmon/hwmon4/temp10_label:\nCore 8\n/sys/class/hwmon/hwmon4/temp14_label:\nCore 12\n/sys/class/hwmon/hwmon4/temp18_label:\nCore 16\n/sys/class/hwmon/hwmon4/temp22_label:\nCore 20\n/sys/class/hwmon/hwmon4/temp26_label:\nCore 24\n/sys/class/hwmon/hwmon4/temp30_label:\nCore 28\n/sys/class/hwmon/hwmon4/temp34_label:\nCore 32\n/sys/class/hwmon/hwmon4/temp35_label:\nCore 33\n/sys/class/hwmon/hwmon4/temp36_label:\nCore 34\n/sys/class/hwmon/hwmon4/temp37_label:\nCore 35\n/sys/class/hwmon/hwmon4/temp38_label:\nCore 36\n/sys/class/hwmon/hwmon4/temp39_label:\nCore 37\n/sys/class/hwmon/hwmon4/temp40_label:\nCore 38\n/sys/class/hwmon/hwmon4/temp41_label:\nCore 39\n/sys/class/hwmon/hwmon4/temp42_label:\nCore 40\n/sys/class/hwmon/hwmon4/temp43_label:\nCore 41\n/sys/class/hwmon/hwmon4/temp44_label:\nCore 42\n/sys/class/hwmon/hwmon4/temp45_label:\nCore 43\n/sys/class/hwmon/hwmon4/temp46_label:\nCore 44\n/sys/class/hwmon/hwmon4/temp47_label:\nCore 45\n/sys/class/hwmon/hwmon4/temp48_label:\nCore 46\n/sys/class/hwmon/hwmon4/temp49_label:\nCore 47\nNotice something? I do:
\ntemp1_label is Package id 0 - what is that?The explanation for the first point is simple, looking at the kernel\ncoretemp.c\nimplementation, we can see that it is the temperature of the CPU as a whole:
static ssize_t show_label(\n struct device *dev,\n struct device_attribute *devattr,\n char *buf\n) {\n struct platform_data *pdata = dev_get_drvdata(dev);\n struct temp_data *tdata = container_of(devattr, struct temp_data, sd_attrs[ATTR_LABEL]);\n\n if (is_pkg_temp_data(tdata))\n return sprintf(buf, \"Package id %u\\n\", pdata->pkg_id);\n\n return sprintf(buf, \"Core %u\\n\", tdata->cpu_core_id);\n}\nThat leaves us the second question. Why do we have CPUs 1, 2, 6, 10, 14, 18, 22,\n26, 30, 34-49, instead of the expected 0-31?
\nIt turns out that CPU It's common for chip manufacturers to disable faulty or\ndegraded cores before shipping. Or if a lower tier SKU^1 is\nselling more, cores of higher tier SKUs are disabled to match expectations.
\nThis process is called binning, and it exists to not waste silicon. Since\nsilicon manufacturing is not perfect, some CPUs are more faulty, thus slower\nthan others. The process determines how faulty a CPU is and sorts them into\n\"bins\". This is also why lower tier CPUs who are almost exactly the same as\nhigher tier CPUs exist.
The likelihood of faulty silicon also increases with the smaller the\narchitecture size gets (the Apple M4 is 4nm, which is crazy), so this method of\nrecycling worse chips is becoming much more valuable by the day.
\nSo, in summary the 32 core CPU I was testing this on was most likely just the 64\ncore version with a bunch of the cores disabled.
\n\n\nNginx automatically creates\nETag\nheaders for static files. But it does it in such a way that is suboptimal.
\nI'll explain why that is, and how the\nNixpkgs distribution of Nginx\ndoes it better.
\nFirst, let's take a look at how\nNginx sets Etags:
\n// in ngx_http_set_etag @ src/http/ngx_http_core_module.c:1681\n\n1\netag = ngx_list_push(&r->headers_out.headers);\nif (etag == NULL) {\n return NGX_ERROR;\n}\n\n2\netag->hash = 1;\netag->next = NULL;\nngx_str_set(&etag->key, \"ETag\");\n\n3\netag->value.data = ngx_pnalloc(r->pool, NGX_OFF_T_LEN + NGX_TIME_T_LEN + 3);\nif (etag->value.data == NULL) {\n etag->hash = 0;\n return NGX_ERROR;\n}\n\n4\netag->value.len = ngx_sprintf(etag->value.data, \"\\\"%xT-%xO\\\"\",\n r->headers_out.last_modified_time,\n r->headers_out.content_length_n)\n - etag->value.data;\nHere we create an empty header entry in the requests's outgoing headers.
\nWe initialize it, and set its key to \"ETag\".
\nWe then allocate memory for the ETag value from the current request's arena.\nThe size is calculated to be large enough to hold the timestamp\n(NGX_TIME_T_LEN), the content length (NGX_OFF_T_LEN), and extra\ncharacters (two quotes \" and a hyphen -).
And then we fill the header's value to a string like \"5f7e1b2a-1c8f\" using\nthe last modified time of the file being served and its content length.\nSomething weird is - etag->value.data, which is required because\nngx_sprintf returns a pointer to the end of the string. We do it to get the\nstart.
Result: The outgoing request now contains an Etag header, which is based off\nof the file's length and last modified date.
What if we use a tool that aims to produce\nreproducible build artifacts to build our\nwebsite, and we just so happen to modify a file in such a way that doesn't\nchange the size of its contents? (more common than you think)
\n\n\nFor example:\nNix^1,\nwhich always sets the last-modified timestamps for files under the Nix\nstore^2 to
\n1, which corresponds to January 1st, 1970,\n00:00:01 UTC.
In that case, neither the last modified date or the content length would change.\nSo the ETag header would stay constant, which would make the client assume\nthe asset has not changed.
The caching on your website being silently broken is a monumental issue, and has\nto be fixed. The way Nixpkgs does it is through\nthis patch:
\nThis patch makes it possible to serve static content from Nix store paths, by\nusing the hash of the store path for the ETag header.\n\ndiff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c\nindex 97a91aee2..2d07d71e6 100644\n--- a/src/http/ngx_http_core_module.c\n+++ b/src/http/ngx_http_core_module.c\n ngx_http_set_etag(ngx_http_request_t *r)\n {\n ngx_table_elt_t *etag;\n ngx_http_core_loc_conf_t *clcf;\n 2\n+ u_char *real, *ptr1, *ptr2;\n+ ngx_err_t err;\n\n clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);\n\n ngx_http_set_etag(ngx_http_request_t *r)\n etag->next = NULL;\n ngx_str_set(&etag->key, \"ETag\");\n\n 1\n- etag->value.data = ngx_pnalloc(r->pool, NGX_OFF_T_LEN + NGX_TIME_T_LEN + 3);\n- if (etag->value.data == NULL) {\n- etag->hash = 0;\n- return NGX_ERROR;\n+ // Upstream nginx uses file mod timestamp and content-length for Etag, but\n+ // files in the Nix store have their timestamps reset, so that doesn't work.\n+ // Instead, when serving from the Nix store, we use the hash from the store\n+ // path and content-length.\n+ //\n+ // Every file in under the given store path will share the same store path\n+ // hash. It is fine to serve different resources with the same Etag, but\n+ // different representations of the same resource (eg the same file, but\n+ // gzip-compressed) should have different Etags. Thus, we also append\n+ // content-length, which should be different when the response is compressed\n+\n+ err = ngx_errno; 3\n+ real = ngx_realpath(clcf->root.data, NULL); 4\n+ ngx_set_errno(err); 5\n+\n+ #define NIX_STORE_DIR \"@nixStoreDir@\" 6\n+ #define NIX_STORE_LEN @nixStoreDirLen@\n+\n+ if (r->headers_out.last_modified_time == 1 7\n+ && real != NULL\n+ && !ngx_strncmp(real, NIX_STORE_DIR, NIX_STORE_LEN)\n+ && real[NIX_STORE_LEN] == '/'\n+ && real[NIX_STORE_LEN + 1] != '\\0')\n+ {\n+ // extract the hash from a path formatted like\n+ // /nix/store/hashhere1234-pname-1.0.0\n+ // +1 to skip the leading /\n+ ptr1 = real + NIX_STORE_LEN + 1; 8\n+\n+ ptr2 = (u_char *) ngx_strchr(ptr1, '-');\n+\n+ if (ptr2 == NULL) {\n+ ngx_free(real);\n+ etag->hash = 0;\n+ return NGX_ERROR;\n+ }\n+\n+ *ptr2 = '\\0'; 9\n+\n+ // hash + content-length + quotes and hyphen. Note that the\n+ // content-length part of the string can vary in length.\n+ etag->value.data = ngx_pnalloc(r->pool, ngx_strlen(ptr1) + NGX_OFF_T_LEN + 3);\n+\n+ if (etag->value.data == NULL) {\n+ ngx_free(real);\n+ etag->hash = 0;\n+ return NGX_ERROR;\n+ }\n+\n+\n+ // set value.data content to \"{hash}-{content-length}\" (including quote\n+ // marks), and set value.len to the length of the resulting string\n+ etag->value.len = ngx_sprintf(etag->value.data, \"\\\"\\%s-%xO\\\"\",\n+ ptr1,\n+ r->headers_out.content_length_n)\n+ - etag->value.data;\n+\n+ ngx_http_clear_last_modified(r);\n+ } else { 10\n+ // outside of Nix store, use the upstream Nginx logic for etags\n+\n+ etag->value.data = ngx_pnalloc(r->pool, NGX_OFF_T_LEN + NGX_TIME_T_LEN + 3);\n+\n+ if (etag->value.data == NULL) {\n+ ngx_free(real);\n+ etag->hash = 0;\n+ return NGX_ERROR;\n+ }\n+\n+ etag->value.len = ngx_sprintf(etag->value.data, \"\\\"%xT-%xO\\\"\",\n+ r->headers_out.last_modified_time,\n+ r->headers_out.content_length_n)\n+ - etag->value.data;\n }\n\n- etag->value.len = ngx_sprintf(etag->value.data, \"\\\"%xT-%xO\\\"\",\n- r->headers_out.last_modified_time,\n- r->headers_out.content_length_n)\n- - etag->value.data;\n+ ngx_free(real);\n\n r->headers_out.etag = etag;\nThe way this works is as follows:
\nThe old logic, explained above is deleted.
\nWe create 4 new variables, 3 used for path manipulation and 1 for storing\nerrno.
Save the current value of errno (ugh) to a local.
Then we try to resolve clcf->root.data. clcf->root.data is the path\nspecified in the root directive of the Nginx configuration and could be a\nsymbolic link (e.g., it could be /var/www which might be a symlink to a Nix\nstore path^3). ngx_realpath resolves all symbolic\nlinks and returns the canonical, absolute path. This is necessary to reliably\ncheck if the path is inside the Nix store.
Restore the value of errno, as the previous function call might have\nchanged it. We handle the error case of ngx_realpath later.
These are Nix placeholders, aka they get replaced by the path of the Nix\nstore at build time. The reason we do not hardcode /nix/store is because\nNix's store path, when on the local filesystem, is arbitrary.\nThis makes our patch portable.
The main condition: We check if the r->headers_out.last_modified_time\nis 1, which is what Nix sets file's last modified times to, we then check\nif the canonical path resolution has succeeded (real != NULL), if the\ncanonical path starts with the Nix store path (NIX_STORE_DIR), and if the\nstore path has anything after it (@[next] != '/' && @[next + 1] != NULL).
(If the condition is true) We then extract the hash. A path like\n/nix/store/6bxcxc6xvg5xv70z55adcwhgik5m41a0-package-1.0.0/bin/package gets\nturned into 6bxcxc6xvg5xv70z55adcwhgik5m41a0.
(If the condition is true) SIDENOTE: This is a nice hack for one of many\ndesign failures in C, this one being NULL-terminated strings. We just place a\n0 byte at the index we want to end our string out for efficiency, instead\nof creating a copy.
(If the condition is false) We continue on with the old logic, as explained\nat the top.
\nWe then set the value of the ETag header to \"{hash}-{content-length}\", the\nreason for the content length being explained in the comment in the patch:
\n\nIt is fine to serve different resources with the same Etag, but different\nrepresentations of the same resource (eg the same file, but gzip-compressed)\nshould have different Etags. Thus, we also append content-length, which should\nbe different when the response is compressed
\n
This way, by depending on the Nix store path instead of the last modification\ndate, we can serve files more reliably and with less^4 issues.
\nI hope you learned something in this blog post, and perhaps a new way to\nfingerprint Nginx installs :).
\n![\"Nixpkgs](\"https://rgbcu.be/assets/images/nixpkgs-nginx-etag.webp\")
/nix/store. Every\n\"Nix store path\" is represented like so:\n/nix/store/6bxcxc6xvg5xv70z55adcwhgik5m41a0-package-1.0.0, where the hash is\nderived from its inputs..?use std::sync::Arc;\n\nstruct NoClone;\n\n\nstruct WrapArc<T>(Arc<T>);\n\nfn main() {\n let foo = WrapArc(Arc::new(NoClone));\n let foo_ = foo.clone();\n}\nDo you think this code should compile?
\nWhat about the following code:
\nstruct AlwaysEq<T>(T);\n\nimpl<T> PartialEq for AlwaysEq<T> {\n fn eq(&self, _other: &Self) -> bool {\n true\n }\n}\n\nimpl<T> Eq for AlwaysEq<T> {}\n\nstruct NotEq;\n\n\nstruct WrapAlwaysEq<T>(AlwaysEq<T>);\n\nfn assert_is_eq(_: impl Eq) {}\n\nfn main() {\n let x = WrapAlwaysEq(AlwaysEq(NotEq));\n assert_is_eq(x);\n}\nThe second example is a bit far fetched, but you probably answered yes.
\n\nThe\nimplementation of #[derive(Clone)] in the Rust compiler\ngenerates a Clone implementation with the following requirements on the\nderived type:
Clone.Clone.Can you spot the issue here? It's the latter requirement: we cannot just\nrequire all generic parameters to be Clone, as we cannot assume they are used\nin such a way that requires them to be cloned.^1
This applies to practically all builtin derive traits, such as Clone,\nPartialEq, Eq, or\neven Debug.
There are two solutions to this. Both require deleting that second requirement.
\nWe could create a Rust RFC, hopefully not bikeshed it to death, and get it\nstabilized in the next Rust edition as it is a breaking change.^2
\nThis would take 4+ years to stabilize and be available to everyone. That sucks,\nbut is the correct thing to do in the long-term.
\nWe can just write our own macro that generates the following code:
\n/* input */\n\nstruct WrapArc<T>(Arc<T>);\n\n/* generated code */\nimpl<T> Clone for WrapArc<T>\nwhere\n Arc<T>: Clone,\n // and so on, `FieldType: DerivedTrait` for each field\n{\n // ...\n}\nThis does the job correctly.
\nAnd it's not even hard to do. I know people who do this internally in their\ncompany codebases - it's not much code.
\nSo I've opened an issue\nabout replicating the builtin derive traits in a less restrictive and thus\ncorrect way in the derive_more crate's GitHub repository. The reason I chose\nthis crate is because it already has a lot of users and is the main place for\nderive implementations.
Replicating already-existing behaviour of the std may not be in the scope of the\ncrate, which is a perfectly fine stance to take. If that doesn't get accepted,\nI'll probably create my own crate and release it on\ncrates.io.
\nStay tuned, I'll update this blog post.
\n\n\nIf you are a GNU maintainer and are not willing to be criticized by people sick\nof your bullshit software, do not read this post.
\nAlso, obligatory notice on every post that touches GNU: No, nothing in this post\nis political. I am perfectly fine with using GNU licensed software, and find it\nmore favourable than MIT licensed software as it protects user freedoms. (My\npersonal choice however is MPL, as it doesn't deem your software dead on arrival\nat places that actually make money and feed hungry mouths)
\nThe commonly overlooked but also very important argument for dropping GNU\ncoreutils (or any other GNU tool that depends on Gnulib) for anything other than\nthe most core-level bootstrapping needs is that it depends on Gnulib.
\nGnulib is a huge blob of C and 80 thousand lines of m4 that has crusted over 30+\nyears and is almost impossible to build correctly and hermetically, has to be\nmanually patched in every single program that embeds it (if trying to be\nhermetic, and correct), ^1 rewards bad OSes, makes good OSes shrivel in pain,\nmakes packagers go bald & makes issues hard to diagnose and debug.
\nIt's so incredibly easy to build it wrong and create a shitty non-reproducible\ndistro (and it is built non-hermetically by default). At least in Rust (or in\nother ecosystems, such as Go) and the general ecosystem of Rust, Cargo.toml is\npretty well-defined and build.rs scripts don't do anything that insane. (Hell,\neven the C compilation tools used inside crates are shared\ndeps and are well-defined).
I don't trust the average distro to build any toolchain made by GNU properly,\nand I do not trust them to produce a proper set of system tools either because\nof Gnulib and other bad build tools.
\nI do however trust the average distro (not Debian, they are lower than average\nand suck at packaging (Yes, really.)) to build\nUutils tools & any other Rust tool correctly, because it is pretty relatively\nstraightforward compared to hundreds of thousands of lines of hard to audit &\nancient m4. It also won't misbehave or segfault when you interact with a\nmisbehaving NTFS partition.
\nI hope Uutils coreutils & Uutils findutils and so on achieve near perfect\ncompliance so I do not need to serve GNU tools to my users.
\n\n\nAlso, keep in mind that it doesn't have to be Rust or uutils or anything that\nis shilled by the \"Rust Strike Force\" in particular. I would also welcome the\nGNU people fixing up their shitty build tooling & making their software behave\ncorrectly. But that isn't happening any time soon, so we are starting from\nscratch & fixing (quite) a few itches on the way.
\n
Read more about this on the\nSortix wiki.
\nYes, really.
\nFrom\nPhoronix, on bcachefs-tools being \"impossible to maintain in a package collection\":
\n\n\nSo, back in April the Rust dependencies for bcachefs-tools in Debian didn’t at\nall match the build requirements. I got some help from the Rust team who says\nthat the common practice is to relax the dependencies of Rust software so that\nit builds in Debian. So errno, which needed the exact version 0.2, was relaxed\nso that it could build with version 0.4 in Debian, udev 0.7 was relaxed for\n0.8 in Debian, memoffset from 0.8.5 to 0.6.5, paste from 1.0.11 to 1.08 and\nbindgen from 0.69.9 to 0.66.
\nI found this a bit disturbing, but it seems that some Rust people have lots of\nconfidence that if something builds, it will run fine. And at least it did\nbuild, and the resulting binaries did work, although I’m personally still not\nvery comfortable or confident about this approach (perhaps that might change\nas I learn more about Rust).
\nWith that in mind, at this point you may wonder how any distribution could\nsanely package this. The problem is that they can’t. Fedora and other\ndistributions with stable releases take a similar approach to what we’ve done\nin Debian, while distributions with much more relaxed policies (like Arch)\ninclude all the dependencies as they are vendored upstream.
\n
Incredibly foolish. You are not supposed to package every single crate manually,\nand you should not be anyway.
\nThe way you should package any programming language that has a widely used and\ngenerally well-defined and static build system is to generate (unlisted) package\ndefinitions from packages using a script or tool (such as\ncargo metadata -> parse json -> puke out package manifests), and only add\nextra configuration to packages that depend on anything external (such as a C\nlibrary, or CMake, or perl, or Go for some godforsaken reason (Why,\naws-lc-sys, why?)).
It's also not hard to use a dependency solver algorithm to try and deduplicate\nall crates required in the whole package repository using a pre-made library,\nsuch as lib.rs/pubgrub. You can have the best\nof all worlds.
In general - Debian is a distro stuck in the 90s that assumes every language\necosystem is as fragmented, differing and inconsistent as C's. That's not the\ncase anymore, Debian maintainers should wake up from their slumber and modernize\ntheir tools, automating way more. The future is not C^2, and a good distro\ncannot assume that.
\n\n\nAnd so, my adventure with bcachefs-tools comes to an end. I’d advise that if\nyou consider using bcachefs for any kind of production use in the near future,\nyou first consider how supportable it is long-term, and whether there’s really\nanyone at all that is succeeding in providing stable support for it.
\n
It's trivial to support!\nHere is what Nixpkgs, the biggest Nix package\ncollection, Nixpkgs, does -\nlook, it's all 140 lines of code!
\nNixpkgs has pkgs.buildRustCrate,\nto build crates without Cargo, but currently doesn't use it for most packages,\nso it doesn't have crate-level incremental rebuilds. This may change in the\nfuture, and when it does, compiling Rust programs will take a fraction of the\ntime because you aren't building dependencies over and over and over and over\nagain, and can utilize cache.nixos.org, or any other cache.
It will also decrease the amount of lines you'll have to write in Nixpkgs\npackage specifications, because you'll no longer have to specify all external\ndependencies for a program. Why? Because external, non-Cargo managed\ndependencies will be configured in a\nper-crate basis,\nso a top-level Rust program that uses a crate that requires liburing won't\nactually see liburing when being compiled.
And here's another great part: Since the pkgs view is not a 1:1 map of all the\nderivations (build units, but in a fancy Nix / Functional Software Deployment\nModel way), these generated definitions won't pollute search or interfere with\nusers. It's just a plus in almost every way imagineable.^3
\n\nYou can actually use this right now with\n
\ncargo2nixor\ncrate2nix, though expect some\nheadaches. (I recently spent a few hours gettingcrate2nixto work on a huge\ncodebase with a lot of native dependencies, andaws-lc-syswas the biggest\nheadache on darwin)\n\nThe extra crate overrides I ended up with, if it helps.
\n\n{ pkgs, ... }:\n{\n openssl = old: {\n buildInputs = old.buildInputs or [ ] ++ [\n pkgs.openssl\n ];\n };\n\n openssl-sys = old: {\n RUSTFLAGS = \"--cfg ossl111 --cfg ossl110 --cfg ossl101\";\n\n \"${pkgs.stdenv.buildPlatform.rust.cargoEnvVarTarget}_OPENSSL_DIR\" = \"${pkgs.openssl.dev}\";\n OPENSSL_NO_VENDOR = \"1\";\n OPENSSL_STATIC = \"0\";\n };\n\n aws-lc-sys = old: {\n nativeBuildInputs = old.nativeBuildInputs or [ ] ++ [\n pkgs.cmake\n pkgs.go\n pkgs.libclang\n pkgs.perl\n pkgs.nasm\n ];\n env.CFLAGS = \"-D_DARWIN_C_SOURCE\"; # Hours of my life wasted to figure this out.\n };\n\n librocksdb-sys = old: {\n buildInputs = old.buildInputs or [ ] ++ [\n pkgs.zlib\n pkgs.zstd\n pkgs.bzip2\n pkgs.lz4\n ];\n };\n\n tokio = old: {\n extraRustcOpts = old.extraRustcOpts or [ ] ++ [\n \"--cfg tokio_unstable\"\n ];\n };\n\n uvm_syn = old: {\n extraRustcOpts = old.extraRustcOpts or [ ] ++ [\n \"-Awarnings\"\n ];\n };\n\n pyo3_build_config = old: {\n buildInputs = old.buildInputs or [ ] ++ [\n pkgs.python313\n ];\n };\n}\nI also had to patch the
\nCargo.nixgenerator because it was not handling or\neven exposing a way to enable extra cfg flags separately, but maybe this is\nfixed incargo2nix?Anyway, we do not talk about the general state of Nix/Nixpkgs/NixOS tooling.\n(But it's still better than every other distro out there)
\n
Package management and build systems aren't hard, your tools are just bad.
\nThat's why I'm working on a new system named \"Cull\", which will hopefully solve\na lot of these problems (and thus fix the mistakes of Nix). Stay tuned!
\nIt will also be cross platform\n(Linux/BSDs/Darwin/Windows/<insert-your-favourite-os-that-has-rust-support-here>),\nand cacheable at the expression level. No waiting for your system closure to\nevaluate for 5 minutes.
\n\n\nnix-daemon, but that shouldn't be significant\nfor most of Rust builds from scratch, and for those that are, the already-cached\ncrates will mitigate the losses pretty easily...?From the\ntree(1) release notes,\nversion 2.0.0:
\n\n\n
\n- Output un-indented JSON on file descriptor 3 (\"stddata\") automatically if\nfile descriptor 3 is present (currently Linux only.) Maybe switch to BSON.
\n
Surely, nothing will happen if I just assume that the existence of a specific\nfile descriptor implies something, as nobody is crazy or stupid enough to\nhardcode such a thing? <insert sounds of cars crashing in the\nbackground>
\nOops (from the treeversion 2.0.2 release notes):
\n\n\n
\n- Okay, apparently the stddata addition is causing havoc (who knew how many\nscripts just haphazardly hand programs random file descriptors, that's\nsurely not a problem.) Going forward the stddata option will only work if\nthe environment variable STDDATA_FD is present or set to the descriptor to\nproduce the JSON output on.
\n
Assuming anything about the users environment is and will always be a mistake.\nBut it's still cool we can have tree output structured data using\nSTDDATA_FD:
> STDDATA_FD=1 tree\n\n[{\"type\":\"directory\",\"name\":\".\",\"contents\":[{\"type\":\"file\",\"name\":\"2024-01-01-test.md\"},{\"type\":\"file\",\"name\":\"2024-03-04-htmnix.md\"},{\"type\":\"file\",\"name\":\"2024-04-02-cosmic-drift.md\"},{\"type\":\"file\",\"name\":\"2024-04-15-nix-iceberg.md\"},{\"type\":\"file\",\"name\":\"2024-05-20-swap-i.md\"},{\"type\":\"file\",\"name\":\"2024-09-30-gitignore.md\"},{\"type\":\"file\",\"name\":\"2025-06-05-why-cores.md\"},{\"type\":\"file\",\"name\":\"2025-06-09-nginix.md\"},{\"type\":\"file\",\"name\":\"2025-06-20-intro-cab.md\"},{\"type\":\"file\",\"name\":\"2025-07-04-derive-broken.md\"},{\"type\":\"file\",\"name\":\"2025-07-20-no-gnulib.md\"},{\"type\":\"file\",\"name\":\"_data.ts\"}]},{\"type\":\"report\",\"directories\":1,\"files\":12}]\nAlso, obligatory nushell mention (using the markdown\ntable style because for some reason the fancy characters don't render properly\nwith my webfont):
> STDDATA_FD=1 tree | from json\n\n|-#-|---type----|-name-|------------------contents-------------------|-directories-|-files-|\n| 0 | directory | . | |-#--|-type-|------------name-------------| | nope | nope |\n| | | | | 0 | file | 2024-01-01-test.md | | | |\n| | | | | 1 | file | 2024-03-04-htmnix.md | | | |\n| | | | | 2 | file | 2024-04-02-cosmic-drift.md | | | |\n| | | | | 3 | file | 2024-04-15-nix-iceberg.md | | | |\n| | | | | 4 | file | 2024-05-20-swap-i.md | | | |\n| | | | | 5 | file | 2024-09-30-gitignore.md | | | |\n| | | | | 6 | file | 2025-06-05-why-cores.md | | | |\n| | | | | 7 | file | 2025-06-09-nginix.md | | | |\n| | | | | 8 | file | 2025-06-20-intro-cab.md | | | |\n| | | | | 9 | file | 2025-07-04-derive-broken.md | | | |\n| | | | | 10 | file | 2025-07-20-no-gnulib.md | | | |\n| | | | | 11 | file | _data.ts | | | |\n| 1 | report | nope | nope | 1 | 12 |\nInspired by the legendary\nQc Na\nand Git Koans, which was in\nturn inspired by Vim Koans.
\nA site reliability engineer rushed into the Nix Store.
\n\"The whole front aisle is rotten!\" he cried to the clerk.
\nThey seized the tainted goods, but the goods were fastened to the shelves, and\nthe shelves to the floor.
\nA passerby said: \"Build a new shelf, and move the door.\"
\n\"There is no time\", cried the clerk. Then she shouted: \"mount -o remount,rw\n/nix/store!\"
\nAt last, something budged. The building collapsed, its foundations swapped for\nenlightenment.
\nA novice was running a free advertising campaign for Master Nix.
\n\"What Master Nix builds today, he will build in ten years!\" he shouted to the\nmarketplace. \"A build within his garden knows neither the day, nor the hour!\"
\nMaster Nix, who was passing, handed the novice a derivation. \"Print my\npamphlets,\" he said.
\nThe novice took the derivation to the garden, and printed the pamphlets. Each\npamphlet bore its day and hour of printing.
\n\"A breach in the walls!\" cried the novice.
\nMaster Nix said nothing. He struck the day and hour from the template, and\nprinted again with --impure. The pamphlets agreed. He tore down the garden\nwalls with __noChroot = true, and printed once more, in the open air. The\npamphlets agreed.
\"Master, with the walls torn, what keeps the world out of the build?\"
\n\"What kept the hour out, when they stood?\"
\nThe marketplace went quiet.
\nAt that moment, the novice was enlightened.
\nA functional programmer was trying to convince Master Nix that his project was\nworthy of a derivation.
\nMaster Nix eventually answered the call, evaluated the programmer, and fulfilled\nhis wish.
\nThe programmer copied the bestowed gift onto a portable storage device, and as\nhe turned to depart, Master Nix asked:
\n\"And its closure?\"
\nThe programmer rummaged through his code, and only found self-contained\nfunctions. \"There are no closures in my program\" he said.
\nMaster Nix said nothing.
\nAt the demonstration, the programmer was englightened.
\nA new student of Master Nix was learning the basics.
\nProud of his first package, he showed the master his work: \"nix-build hello.nix\"
\nMaster Nix struck him on the head with a metal bat. \"We do not speak the old\ntongue.\"
\nThe student learned the new tongue, in which a silence stands where the hyphen\nstood. He spoke it beautifully: nix build, nix shell, nix run. In time he grew\nzealous, and corrected the older students when they slipped.
\nOne day, wishing he knew how the temple was kept, he opened the scripts of the\nmasters. nix-instantiate, nix-env, nix-prefetch-url, nix-store. The old tongue,\nfrom the first scroll to the last.
\n\"Master!\" he cried. \"Your own scriptures-\"
\nMaster Nix threw him into the well.
\nAt the bottom, stood enlightenment.
\nA poor devops engineer who was assigned a teletype of unusual thinness was\nlimping along in despair, for his masters demanded x86_64-linux, and the\nteletype knew it not.
\nOne day, he heard a colleague speak of pkgsCross, by which a machine may build\nfor a land it will never see. That night, he evaluated pkgsCross.gnu64, and\nthe teletype began to build the world.
binutils crossed. glibc crossed. Then a package, wishing to test what it had\nbuilt, tried to run it; but what it had built was of the far land, and the\nteletype knew it not. A thousand errors followed. The engineer read the first,\nand wept.
\nAn old operator looked over his shoulder.
\n\"A fine terminal,\" he said. \"But one needs a computer to build.\"
\nThe engineer wrote in his configuration:\nbuilders = ssh-ng://elsewhere x86_64-linux. The teletype clattered through the\nnight, printing the labors of a distant machine.
At dawn, the engineer was enlightened.
\n\n\n \n","tags":["nix/os"],"date_published":"2026-06-13T00:00:00.000Z"}]}