I Explored the Linux File System for a Week — Here's What Broke My Brain

I thought I knew Linux. Then I actually started poking around the file system — and realized I barely scratched the surface. This isn't a textbook walkthrough. These are genuine "wait, WHAT?" moments I had while investigating Linux directories one by one.

Why I Wrote This

Most Linux resources explain what a directory is. Very few explain why it's designed that way, or what happens when it breaks.

I spent a week deliberately exploring the Linux file system — reading files, tracing commands to their sources, and sometimes (accidentally) breaking things. What came out is this: a collection of insights that actually matter, not just facts you could Google.

Let's go.

---

🔧 /etc/fstab — The File That Can Break Your Boot

When I first opened /etc/fstab, I thought: "This is just a list of drives."

Then I made a typo — a wrong UUID — and my system dropped into emergency recovery mode on the next boot.

That's when it hit me: this plain-text config file has more power than most system services.

What It Does

Every line in /etc/fstab tells Linux:

• What to mount (device by UUID or label)

• Where to mount it (mount point like /, /home, /boot)

• How to mount it (filesystem type + options)

# Example /etc/fstab entry
UUID=abc1234-...  /       ext4    errors=remount-ro  0  1
UUID=def5678-...  /boot   vfat    umask=0077         0  1

💡 The Insight That Surprised Me

Linux doesn't use device names like /dev/sda1 in modern systems — it uses UUIDs.

Why? Because device names can silently change between boots depending on which drives are plugged in and detected first. UUIDs are stable, permanent identifiers tied to the filesystem itself.

Practical Rule: Always use UUIDs in /etc/fstab, never /dev/sdX paths. Always validate with sudo mount -a before rebooting.

---

📁 /dev — Hardware Isn't Hardware, It's Files

Here's something that fundamentally changed how I think about operating systems:

In Linux, hardware devices are exposed as files.

Your hard disk? A file (/dev/sda).
Your terminal? A file (/dev/tty).
A place to dump data you want to discard? A file (/dev/null).

The Philosophy Behind It

This is the "everything is a file" design principle at its best. It means programs can interact with hardware using the same read/write operations they use for regular files — no special APIs needed for each device.

# Write to terminal directly
echo "hello" > /dev/tty

# Discard output
command > /dev/null 2>&1

# Even random number generation is a file
head -c 16 /dev/urandom | xxd

💡 The Insight That Surprised Me

When you do ls -l /dev, you'll see file types b and c in the permissions column:

brw-rw---- 1 root disk  8, 0 /dev/sda     ← block device
crw-rw-rw- 1 root tty   5, 0 /dev/tty     ← character device

Block devices (disks) buffer data. Character devices (terminals, serial ports) stream data one character at a time.

This distinction isn't just theory — it determines how the kernel schedules I/O for that device. Understanding it helped me debug a performance issue where I was treating a block device like a stream.

---

🔍 /proc — A Live Window Into the Kernel

This one genuinely blew my mind.

/proc looks like a directory full of files. But none of those files exist on disk. They're generated on demand, in real time, by the kernel.

Every time you read /proc/cpuinfo, the kernel generates that output right then. The file has no permanent storage — it's a live interface.

Real Examples

# CPU details — generated live by kernel
cat /proc/cpuinfo

# Memory stats
cat /proc/meminfo

# Running processes (each PID has its own directory!)
ls /proc/1234/     # 1234 = process ID

# Network routing table
cat /proc/net/route

💡 The Insight That Surprised Me

Commands you use every day are just pretty wrappers over /proc:

Command	What It Actually Reads
ps	/proc/[PID]/status
top	/proc/meminfo, /proc/stat
route	/proc/net/route
lscpu	/proc/cpuinfo

When I realized that ps is just reading files, it stopped feeling like magic and started feeling like engineering.

---

🌐 DNS Resolution Isn't Just /etc/resolv.conf

This is probably the most practically useful thing in this entire post.

I used to think: "DNS is broken? Check /etc/resolv.conf."

That's only one piece of a much bigger puzzle. Actual DNS resolution in Linux goes through several layers:

The Full Resolution Stack

Request → /etc/hosts → NSS (nsswitch.conf) → systemd-resolved → DNS server

1. /etc/hosts — static local mappings (checked first by default)

2. /etc/nsswitch.conf — controls the order of name resolution

3. systemd-resolved — a local DNS cache running at 127.0.0.53

4. /etc/resolv.conf — points to upstream DNS server

# Check resolution order
cat /etc/nsswitch.conf | grep hosts
# hosts: files mdns4_minimal [NOTFOUND=return] dns

💡 The Insight That Surprised Me

Even with a perfectly correct /etc/resolv.conf, DNS can fail if /etc/nsswitch.conf is misconfigured.

I spent 45 minutes debugging a DNS issue in a Docker container once. The nameserver was set correctly. The problem? nsswitch.conf was set to only check local files (files) and not dns. A single word was missing.

Pro Tip: When DNS breaks, check nsswitch.conf before you start changing nameservers.

---

⚙️ systemd — After= Is NOT Requires=

I had a service that was starting before its database dependency was ready. My After=database.service wasn't working. Why?

Because I confused ordering with dependency.

The Critical Difference

Directive	What It Does
After=	Controls start order only
Requires=	Declares a hard dependency (if dep fails, this fails too)
Wants=	Declares a soft dependency (starts even if dep fails)

[Unit]
Description=My App
After=database.service       # Start after database, but don't care if it fails
Requires=network.target      # MUST have network; fail if not present
Wants=redis.service          # Try to start redis, but proceed even if it doesn't start

💡 The Insight That Surprised Me

A service configured with only After=database.service will start even if the database service fails.

It just waits for the database service to reach a finished state (success or failure) before starting. If you want your app to not start when the database is unavailable, you need Requires=.

This subtle design allows systemd to be flexible — you can have loose ordering without tight coupling.

---

🚀 /proc/net/route — The Routing Table Is Just a File

Most people use ip route to see the routing table. I decided to check where that data comes from.

cat /proc/net/route

Iface  Destination  Gateway   Flags  RefCnt  Use  Metric  Mask      MTU  Window  IRTT
eth0   00000000     0101A8C0  0003   0       0    100     00000000  0    0       0

It's hexadecimal and raw — but it's the same data that ip route prettifies for you.

💡 The Insight That Surprised Me

This is Linux's design philosophy at its purest: the kernel exposes state through the filesystem. No special syscall, no daemon to query — just a file read.

This is why lightweight containers and embedded systems can run networking tools with minimal overhead. The tools are just file readers.

---

🔐 /etc/passwd vs /etc/shadow — A Security Evolution in Two Files

The name is misleading. /etc/passwd does not store passwords anymore.

Here's the historical backstory that makes this make sense:

How It Used to Work (The Insecure Way)

Originally, /etc/passwd stored both user info and the encrypted password hash. The problem? /etc/passwd is world-readable — any user on the system can open it.

# Anyone can read this
cat /etc/passwd
# debesh:x:1000:1000:Debesh:/home/debesh:/bin/bash
#         ↑
#         This 'x' means "password is in /etc/shadow"

The Fix: /etc/shadow

Passwords were moved to /etc/shadow, which is readable only by root:

# Only root can read this
sudo cat /etc/shadow
# debesh:\(6\)rounds=656000\(salt\)hash...:19500:0:99999:7:::

The shadow file contains:

• The hashed password (not plain text)

• Password aging info (expiry, minimum age, etc.)

• Account lock status

💡 The Insight That Surprised Me

Even today, the x placeholder in /etc/passwd exists purely for backward compatibility. Old programs that expected a password in that field still work — they just see x and know to look elsewhere.

Linux maintained compatibility while improving security. That's good systems design.

---

📜 /var/log — The System's Long-Term Memory

Every meaningful event on your Linux system leaves a trace in /var/log. When something breaks and you don't know why — start here.

Key Log Files

File	What It Captures
auth.log	SSH logins, sudo usage, failed auth attempts
syslog	General system messages
kern.log	Kernel-level events (hardware errors, driver messages)
dpkg.log	Package install/remove history
/var/log/nginx/	Web server access + error logs

# Watch auth logs live
sudo tail -f /var/log/auth.log

# Check kernel errors
sudo grep -i error /var/log/kern.log

# Find failed SSH attempts
sudo grep "Failed password" /var/log/auth.log | tail -20

💡 The Insight That Surprised Me

Modern Ubuntu/Debian systems don't only use text files in /var/log. systemd's journal stores logs in a binary format and manages them via journalctl:

# View logs for a specific service
journalctl -u nginx.service

# Boot logs only
journalctl -b

# Last 100 lines, live
journalctl -n 100 -f

The journal and /var/log coexist — some daemons still write to files, while systemd-managed services use the journal. Knowing both is essential.

---

⚡ /run — The Directory That Vanishes on Reboot

Here's a directory I completely ignored for months: /run.

ls /run
# dbus/  lock/  systemd/  user/  nginx.pid  sshd.pid  ...

This directory is not stored on disk. It's mounted as tmpfs — meaning it lives entirely in RAM.

What Lives in /run

• PID files — records of which process owns a service

• Unix sockets — for inter-process communication

• Lock files — to prevent duplicate processes

• Runtime state — anything a service needs while running

# Check the filesystem type
df -T /run
# Filesystem  Type   Size  Used
# tmpfs       tmpfs  1.6G  1.2M

💡 The Insight That Surprised Me

/run replaced the older /var/run. For years, /var/run was used for the same purpose — but it was on disk, meaning stale PID files could survive reboots and cause services to fail to start.

By moving to a RAM-based filesystem (tmpfs), Linux guarantees a clean slate after every boot. The old /var/run is now just a symlink to /run.

---

🖥️ /sys — You Can Actually Control Hardware With Files

While /proc shows you kernel internals, /sys goes further: it lets you interact with hardware directly.

# Battery level (on a laptop)
cat /sys/class/power_supply/BAT0/capacity

# Network interface stats
cat /sys/class/net/eth0/statistics/rx_bytes

# CPU frequency scaling
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

💡 The Insight That Surprised Me

You can write to some of these files to change hardware behavior at runtime.

# Switch CPU governor to performance mode
echo "performance" | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# Temporarily disable a network interface
echo "0" | sudo tee /sys/class/net/eth0/flags

This is how power management tools, performance tuners, and hardware monitoring daemons work — they're just reading and writing files in /sys.

The abstraction is elegant: you don't need device-specific APIs. You just need file operations.

---

👤 /home vs /root — Not All Users Are Equal

Every regular user gets a subdirectory in /home:

/home/debesh/
/home/alice/
/home/bob/

But the root user doesn't live there. Root has its own separate home: /root.

Why the Separation?

This is a deliberate security and reliability decision:

1. Security — /home is often mounted from a separate partition or network share. If regular users can write anywhere in /home, you don't want root's config files mixed in there.

2. Reliability — If /home gets corrupted, unmounted, or fills up, root can still log in. Without /root, a full /home partition could lock out your only admin account.

# Verify
ls -la / | grep -E "^d.*(home|root)"
# drwxr-xr-x  /home
# drwx------  /root   ← only root can access this

💡 The Insight That Surprised Me

In Docker containers, you'll often see the root user's home set to /root even in minimal images. It's baked into the user setup because Linux assumes root needs a private, separate space — even in containers.

---

🌱 /etc/environment vs .bashrc — Why Your Variable Works in Terminal But Not in Scripts

Ever exported a variable in terminal, confirmed it worked, then watched a script fail because it couldn't find that variable?

The answer is in the difference between these files:

File	Scope	When It Loads
/etc/environment	System-wide (all users)	Every login, all sessions
/etc/profile	System-wide	Login shells only
~/.profile	Per-user	Login shells
~/.bashrc	Per-user	Interactive non-login shells

# This works in terminal (interactive shell)
export MY_VAR="hello"
echo $MY_VAR   # works ✅

# But a cron job or systemd service won't see it
# because those aren't interactive shells

💡 The Insight That Surprised Me

A cron job, a systemd service, or a shell script run via exec operates in a non-login, non-interactive shell. It won't load .bashrc.

For truly global variables available to all processes, use /etc/environment:

# /etc/environment — simple KEY=VALUE format (no export keyword!)
NODE_ENV=production
DATABASE_URL=postgresql://localhost/mydb

This is why "it works on my machine" bugs happen — your terminal has vars from .bashrc, but your deployment script runs in a clean environment.

---

🚀 /boot — Where Linux Actually Begins

This is the most foundational directory of them all. Without /boot, your system is an expensive paperweight.

ls /boot
# vmlinuz-6.1.0-generic    ← the Linux kernel (it's just a file!)
# initrd.img-6.1.0-generic ← initial RAM disk
# grub/                    ← bootloader config
# System.map-6.1.0-generic ← kernel symbol table

What Each File Does

File	Purpose
vmlinuz	The compressed Linux kernel — the core OS
initrd.img / initramfs	Temporary root filesystem used during boot before real root is mounted
grub/	GRUB bootloader config — controls boot menu
System.map	Maps kernel symbol names to memory addresses (used for debugging)

💡 The Insight That Surprised Me

The kernel is just a file. I always imagined the kernel as something deep and hidden, woven into the hardware. But it's literally a compressed binary sitting in /boot/vmlinuz.

When GRUB boots your system, it loads that file into RAM and hands control to it. The entire Linux operating system starts from reading one file.

Also: if /boot is on a separate partition (which is common), and you fill it up (by keeping too many old kernels), your system will refuse to install updates. Always keep /boot clean.

# Check /boot usage
df -h /boot

# Clean old kernels (Ubuntu/Debian)
sudo apt autoremove --purge

---

🎯 Key Takeaways

After a week of digging through the Linux file system, here's what I actually internalized:

1. Linux's "everything is a file" philosophy isn't a gimmick — it's a powerful unifying abstraction that simplifies every layer of the stack.

2. /proc and /sys are live kernel interfaces, not just directories. Reading them is literally querying the kernel.

3. DNS has multiple resolution layers — /etc/resolv.conf is just one config point in a pipeline.

4. systemd ordering ≠ dependency — a subtle distinction with real-world consequences.

5. Security evolved through history — /etc/shadow exists because /etc/passwd was world-readable.

6. Know where your variables are defined — it determines which processes can actually see them.

7. /boot and /etc/fstab are fragile points — be careful with both. One wrong config, and your system won't start.

---

📚 What to Explore Next

If this sparked your curiosity, here's where to go deeper:

• man hier — the official Linux file system hierarchy documentation

• man fstab, man nsswitch.conf, man systemd.unit — deep dives into each topic

• The Linux Documentation Project — classic, comprehensive Linux docs

• Arch Wiki — arguably the best Linux reference on the internet

---

If you found this useful, share it with someone learning Linux. And if I got something wrong or missed a discovery worth adding — drop it in the comments. I'm still learning too.

— Written with genuine curiosity and at least two broken VM instances 🐧