# I Explored the Linux File System for a Week — Here's What Broke My Brain 

> I thought I knew Linux. Then I actually started poking around the file system — and realized I barely scratched the surface. This isn't a textbook walkthrough. These are genuine "wait, WHAT?" moments I had while investigating Linux directories one by one.

## Why I Wrote This

Most Linux resources explain *what* a directory is. Very few explain *why it's designed that way*, or *what happens when it breaks*.

I spent a week deliberately exploring the Linux file system — reading files, tracing commands to their sources, and sometimes (accidentally) breaking things. What came out is this: a collection of **insights that actually matter**, not just facts you could Google.

Let's go.

* * *

## 🔧 `/etc/fstab` — The File That Can Break Your Boot

When I first opened `/etc/fstab`, I thought: *"This is just a list of drives."*

Then I made a typo — a wrong UUID — and my system dropped into **emergency recovery mode** on the next boot.

That's when it hit me: this plain-text config file has more power than most system services.

### What It Does

Every line in `/etc/fstab` tells Linux:

*   **What** to mount (device by UUID or label)
    
*   **Where** to mount it (mount point like `/`, `/home`, `/boot`)
    
*   **How** to mount it (filesystem type + options)
    

```bash
# Example /etc/fstab entry
UUID=abc1234-...  /       ext4    errors=remount-ro  0  1
UUID=def5678-...  /boot   vfat    umask=0077         0  1
```

### 💡 The Insight That Surprised Me

Linux doesn't use device names like `/dev/sda1` in modern systems — **it uses UUIDs**.

Why? Because device names can silently change between boots depending on which drives are plugged in and detected first. UUIDs are stable, permanent identifiers tied to the filesystem itself.

> **Practical Rule:** Always use UUIDs in `/etc/fstab`, never `/dev/sdX` paths. Always validate with `sudo mount -a` before rebooting.

* * *

## 📁 `/dev` — Hardware Isn't Hardware, It's Files

Here's something that fundamentally changed how I think about operating systems:

**In Linux, hardware devices are exposed as files.**

Your hard disk? A file (`/dev/sda`).  
Your terminal? A file (`/dev/tty`).  
A place to dump data you want to discard? A file (`/dev/null`).

### The Philosophy Behind It

This is the *"everything is a file"* design principle at its best. It means programs can interact with hardware using the **same read/write operations** they use for regular files — no special APIs needed for each device.

```bash
# Write to terminal directly
echo "hello" > /dev/tty

# Discard output
command > /dev/null 2>&1

# Even random number generation is a file
head -c 16 /dev/urandom | xxd
```

### 💡 The Insight That Surprised Me

When you do `ls -l /dev`, you'll see file types `b` and `c` in the permissions column:

```plaintext
brw-rw---- 1 root disk  8, 0 /dev/sda     ← block device
crw-rw-rw- 1 root tty   5, 0 /dev/tty     ← character device
```

**Block devices** (disks) buffer data. **Character devices** (terminals, serial ports) stream data one character at a time.

This distinction isn't just theory — it determines how the kernel schedules I/O for that device. Understanding it helped me debug a performance issue where I was treating a block device like a stream.

* * *

## 🔍 `/proc` — A Live Window Into the Kernel

This one genuinely blew my mind.

`/proc` looks like a directory full of files. But **none of those files exist on disk**. They're generated **on demand**, in real time, by the kernel.

Every time you read `/proc/cpuinfo`, the kernel generates that output right then. The file has no permanent storage — it's a live interface.

### Real Examples

```bash
# CPU details — generated live by kernel
cat /proc/cpuinfo

# Memory stats
cat /proc/meminfo

# Running processes (each PID has its own directory!)
ls /proc/1234/     # 1234 = process ID

# Network routing table
cat /proc/net/route
```

### 💡 The Insight That Surprised Me

Commands you use every day are just pretty wrappers over `/proc`:

| Command | What It Actually Reads |
| --- | --- |
| `ps` | `/proc/[PID]/status` |
| `top` | `/proc/meminfo`, `/proc/stat` |
| `route` | `/proc/net/route` |
| `lscpu` | `/proc/cpuinfo` |

When I realized that `ps` is just reading files, it stopped feeling like magic and started feeling like engineering.

* * *

## 🌐 DNS Resolution Isn't Just `/etc/resolv.conf`

This is probably the most practically useful thing in this entire post.

I used to think: *"DNS is broken? Check* `/etc/resolv.conf`*."*

That's only **one piece** of a much bigger puzzle. Actual DNS resolution in Linux goes through several layers:

### The Full Resolution Stack

```plaintext
Request → /etc/hosts → NSS (nsswitch.conf) → systemd-resolved → DNS server
```

1.  `/etc/hosts` — static local mappings (checked first by default)
    
2.  `/etc/nsswitch.conf` — controls the *order* of name resolution
    
3.  `systemd-resolved` — a local DNS cache running at `127.0.0.53`
    
4.  `/etc/resolv.conf` — points to upstream DNS server
    

```bash
# Check resolution order
cat /etc/nsswitch.conf | grep hosts
# hosts: files mdns4_minimal [NOTFOUND=return] dns
```

### 💡 The Insight That Surprised Me

Even with a perfectly correct `/etc/resolv.conf`, DNS can fail if `/etc/nsswitch.conf` is misconfigured.

I spent 45 minutes debugging a DNS issue in a Docker container once. The nameserver was set correctly. The problem? `nsswitch.conf` was set to only check local files (`files`) and not `dns`. A single word was missing.

> **Pro Tip:** When DNS breaks, check `nsswitch.conf` before you start changing nameservers.

* * *

## ⚙️ systemd — `After=` Is NOT `Requires=`

I had a service that was starting before its database dependency was ready. My `After=database.service` wasn't working. Why?

Because I confused **ordering** with **dependency**.

### The Critical Difference

| Directive | What It Does |
| --- | --- |
| `After=` | Controls **start order** only |
| `Requires=` | Declares a **hard dependency** (if dep fails, this fails too) |
| `Wants=` | Declares a **soft dependency** (starts even if dep fails) |

```ini
[Unit]
Description=My App
After=database.service       # Start after database, but don't care if it fails
Requires=network.target      # MUST have network; fail if not present
Wants=redis.service          # Try to start redis, but proceed even if it doesn't start
```

### 💡 The Insight That Surprised Me

A service configured with only `After=database.service` will start **even if the database service fails**.

It just waits for the database service to reach a finished state (success or failure) before starting. If you want your app to *not start* when the database is unavailable, you need `Requires=`.

This subtle design allows systemd to be flexible — you can have loose ordering without tight coupling.

* * *

## 🚀 `/proc/net/route` — The Routing Table Is Just a File

Most people use `ip route` to see the routing table. I decided to check *where that data comes from*.

```bash
cat /proc/net/route
```

```plaintext
Iface  Destination  Gateway   Flags  RefCnt  Use  Metric  Mask      MTU  Window  IRTT
eth0   00000000     0101A8C0  0003   0       0    100     00000000  0    0       0
```

It's hexadecimal and raw — but it's the same data that `ip route` prettifies for you.

### 💡 The Insight That Surprised Me

This is Linux's design philosophy at its purest: **the kernel exposes state through the filesystem**. No special syscall, no daemon to query — just a file read.

This is why lightweight containers and embedded systems can run networking tools with minimal overhead. The tools are just file readers.

* * *

## 🔐 `/etc/passwd` vs `/etc/shadow` — A Security Evolution in Two Files

The name is misleading. `/etc/passwd` does **not** store passwords anymore.

Here's the historical backstory that makes this make sense:

### How It Used to Work (The Insecure Way)

Originally, `/etc/passwd` stored **both** user info and the encrypted password hash. The problem? `/etc/passwd` is **world-readable** — any user on the system can open it.

```bash
# Anyone can read this
cat /etc/passwd
# debesh:x:1000:1000:Debesh:/home/debesh:/bin/bash
#         ↑
#         This 'x' means "password is in /etc/shadow"
```

### The Fix: `/etc/shadow`

Passwords were moved to `/etc/shadow`, which is readable only by root:

```bash
# Only root can read this
sudo cat /etc/shadow
# debesh:$6$rounds=656000$salt$hash...:19500:0:99999:7:::
```

The shadow file contains:

*   The **hashed** password (not plain text)
    
*   Password aging info (expiry, minimum age, etc.)
    
*   Account lock status
    

### 💡 The Insight That Surprised Me

Even today, the `x` placeholder in `/etc/passwd` exists purely for **backward compatibility**. Old programs that expected a password in that field still work — they just see `x` and know to look elsewhere.

Linux maintained compatibility while improving security. That's good systems design.

* * *

## 📜 `/var/log` — The System's Long-Term Memory

Every meaningful event on your Linux system leaves a trace in `/var/log`. When something breaks and you don't know why — **start here**.

### Key Log Files

| File | What It Captures |
| --- | --- |
| `auth.log` | SSH logins, sudo usage, failed auth attempts |
| `syslog` | General system messages |
| `kern.log` | Kernel-level events (hardware errors, driver messages) |
| `dpkg.log` | Package install/remove history |
| `/var/log/nginx/` | Web server access + error logs |

```bash
# Watch auth logs live
sudo tail -f /var/log/auth.log

# Check kernel errors
sudo grep -i error /var/log/kern.log

# Find failed SSH attempts
sudo grep "Failed password" /var/log/auth.log | tail -20
```

### 💡 The Insight That Surprised Me

Modern Ubuntu/Debian systems don't only use text files in `/var/log`. **systemd's journal** stores logs in a binary format and manages them via `journalctl`:

```bash
# View logs for a specific service
journalctl -u nginx.service

# Boot logs only
journalctl -b

# Last 100 lines, live
journalctl -n 100 -f
```

The journal and `/var/log` coexist — some daemons still write to files, while systemd-managed services use the journal. Knowing both is essential.

* * *

## ⚡ `/run` — The Directory That Vanishes on Reboot

Here's a directory I completely ignored for months: `/run`.

```bash
ls /run
# dbus/  lock/  systemd/  user/  nginx.pid  sshd.pid  ...
```

This directory is **not stored on disk**. It's mounted as `tmpfs` — meaning it lives entirely in RAM.

### What Lives in `/run`

*   **PID files** — records of which process owns a service
    
*   **Unix sockets** — for inter-process communication
    
*   **Lock files** — to prevent duplicate processes
    
*   **Runtime state** — anything a service needs while running
    

```bash
# Check the filesystem type
df -T /run
# Filesystem  Type   Size  Used
# tmpfs       tmpfs  1.6G  1.2M
```

### 💡 The Insight That Surprised Me

`/run` replaced the older `/var/run`. For years, `/var/run` was used for the same purpose — but it was on disk, meaning stale PID files could survive reboots and cause services to fail to start.

By moving to a RAM-based filesystem (`tmpfs`), Linux guarantees a clean slate after every boot. The old `/var/run` is now just a symlink to `/run`.

* * *

## 🖥️ `/sys` — You Can Actually Control Hardware With Files

While `/proc` shows you kernel internals, `/sys` goes further: **it lets you interact with hardware directly**.

```bash
# Battery level (on a laptop)
cat /sys/class/power_supply/BAT0/capacity

# Network interface stats
cat /sys/class/net/eth0/statistics/rx_bytes

# CPU frequency scaling
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
```

### 💡 The Insight That Surprised Me

You can **write** to some of these files to change hardware behavior at runtime.

```bash
# Switch CPU governor to performance mode
echo "performance" | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# Temporarily disable a network interface
echo "0" | sudo tee /sys/class/net/eth0/flags
```

This is how power management tools, performance tuners, and hardware monitoring daemons work — they're just reading and writing files in `/sys`.

The abstraction is elegant: you don't need device-specific APIs. You just need file operations.

* * *

## 👤 `/home` vs `/root` — Not All Users Are Equal

Every regular user gets a subdirectory in `/home`:

```plaintext
/home/debesh/
/home/alice/
/home/bob/
```

But the root user doesn't live there. Root has its own separate home: `/root`.

### Why the Separation?

This is a **deliberate security and reliability decision**:

1.  **Security** — `/home` is often mounted from a separate partition or network share. If regular users can write anywhere in `/home`, you don't want root's config files mixed in there.
    
2.  **Reliability** — If `/home` gets corrupted, unmounted, or fills up, **root can still log in**. Without `/root`, a full `/home` partition could lock out your only admin account.
    

```bash
# Verify
ls -la / | grep -E "^d.*(home|root)"
# drwxr-xr-x  /home
# drwx------  /root   ← only root can access this
```

### 💡 The Insight That Surprised Me

In Docker containers, you'll often see the root user's home set to `/root` even in minimal images. It's baked into the user setup because Linux assumes root needs a private, separate space — even in containers.

* * *

## 🌱 `/etc/environment` vs `.bashrc` — Why Your Variable Works in Terminal But Not in Scripts

Ever exported a variable in terminal, confirmed it worked, then watched a script fail because it couldn't find that variable?

**The answer is in the difference between these files:**

| File | Scope | When It Loads |
| --- | --- | --- |
| `/etc/environment` | System-wide (all users) | Every login, all sessions |
| `/etc/profile` | System-wide | Login shells only |
| `~/.profile` | Per-user | Login shells |
| `~/.bashrc` | Per-user | Interactive non-login shells |

```bash
# This works in terminal (interactive shell)
export MY_VAR="hello"
echo $MY_VAR   # works ✅

# But a cron job or systemd service won't see it
# because those aren't interactive shells
```

### 💡 The Insight That Surprised Me

A cron job, a systemd service, or a shell script run via `exec` operates in a **non-login, non-interactive shell**. It won't load `.bashrc`.

For truly global variables available to all processes, use `/etc/environment`:

```bash
# /etc/environment — simple KEY=VALUE format (no export keyword!)
NODE_ENV=production
DATABASE_URL=postgresql://localhost/mydb
```

This is why "it works on my machine" bugs happen — your terminal has vars from `.bashrc`, but your deployment script runs in a clean environment.

* * *

## 🚀 `/boot` — Where Linux Actually Begins

This is the most foundational directory of them all. Without `/boot`, your system is an expensive paperweight.

```bash
ls /boot
# vmlinuz-6.1.0-generic    ← the Linux kernel (it's just a file!)
# initrd.img-6.1.0-generic ← initial RAM disk
# grub/                    ← bootloader config
# System.map-6.1.0-generic ← kernel symbol table
```

### What Each File Does

| File | Purpose |
| --- | --- |
| `vmlinuz` | The compressed Linux kernel — the core OS |
| `initrd.img` / `initramfs` | Temporary root filesystem used during boot before real root is mounted |
| `grub/` | GRUB bootloader config — controls boot menu |
| `System.map` | Maps kernel symbol names to memory addresses (used for debugging) |

### 💡 The Insight That Surprised Me

**The kernel is just a file.** I always imagined the kernel as something deep and hidden, woven into the hardware. But it's literally a compressed binary sitting in `/boot/vmlinuz`.

When GRUB boots your system, it loads that file into RAM and hands control to it. The entire Linux operating system starts from reading one file.

Also: if `/boot` is on a separate partition (which is common), and you fill it up (by keeping too many old kernels), **your system will refuse to install updates**. Always keep `/boot` clean.

```bash
# Check /boot usage
df -h /boot

# Clean old kernels (Ubuntu/Debian)
sudo apt autoremove --purge
```

* * *

## 🎯 Key Takeaways

After a week of digging through the Linux file system, here's what I actually internalized:

1.  **Linux's "everything is a file" philosophy** isn't a gimmick — it's a powerful unifying abstraction that simplifies every layer of the stack.
    
2.  `/proc` **and** `/sys` **are live kernel interfaces**, not just directories. Reading them is literally querying the kernel.
    
3.  **DNS has multiple resolution layers** — `/etc/resolv.conf` is just one config point in a pipeline.
    
4.  **systemd ordering ≠ dependency** — a subtle distinction with real-world consequences.
    
5.  **Security evolved through history** — `/etc/shadow` exists because `/etc/passwd` was world-readable.
    
6.  **Know where your variables are defined** — it determines which processes can actually see them.
    
7.  `/boot` **and** `/etc/fstab` **are fragile points** — be careful with both. One wrong config, and your system won't start.
    

* * *

## 📚 What to Explore Next

If this sparked your curiosity, here's where to go deeper:

*   `man hier` — the official Linux file system hierarchy documentation
    
*   `man fstab`, `man nsswitch.conf`, `man systemd.unit` — deep dives into each topic
    
*   [**The Linux Documentation Project**](https://tldp.org/) — classic, comprehensive Linux docs
    
*   [**Arch Wiki**](https://wiki.archlinux.org/) — arguably the best Linux reference on the internet
    

* * *

*If you found this useful, share it with someone learning Linux. And if I got something wrong or missed a discovery worth adding — drop it in the comments. I'm still learning too.*

*— Written with genuine curiosity and at least two broken VM instances 🐧*