My Website Crashed Twice in 48 Hours, So I Rebuilt the Entire Deployment Pipeline

At 11:14 AM on August 19, I sent a message in Feishu.

Our website is inaccessible from the public internet again. Can we check why?

The key word is "again."

It had crashed once two days earlier. At the time, I didn't think much of it. I manually restarted the service, the site came back, and I moved on. My assessment was simple: it was a random occurrence, just bad luck.

The second time, same pattern, same time window, same downtime.

When the same failure happens a second time, it's no longer an accident—it's a design flaw.

This article documents four architecture upgrades that started that morning. On August 19, I rebuilt the deployment pipeline. On August 21, I added HTTPS. On August 27, I set up off-site backups. And last night, while fact-checking for this article, I uncovered a hidden issue that had been lurking for eight days.

One more thing to clarify upfront: from troubleshooting to upgrades, almost everything was executed by an AI assistant on the server. I sat on the other end of the Feishu chat and made only four decisions. I'll walk through this throughout the article, including where it got stuck and where it made mistakes.

Five Commands to Find the Root Cause

After reporting the issue, I didn't open a terminal. I sent the message to the AI assistant, and it started investigating.

Five commands, and the root cause surfaced.

The architecture at the time was this: I used pnpm dev to run an Astro development server on port 3000, and iptables to forward traffic from port 80 to it. It sounded workable, but it had two major pitfalls.

iptables rules are not persistent by default. When the server reboots, the forwarding rules are wiped.

pnpm dev is not a systemd service. After a reboot, nothing brings it back up.

Neither of these is a bug. They are the result of using development tools for production work. pnpm dev was never designed to survive a reboot; it's for hot-reloading while you edit code locally, not for handling production traffic.

Using it in production is like driving a go-kart on the highway and then complaining it's unsafe.

Three Options, and My First Decision

After the investigation, the AI assistant gave me three options.

I chose C.

This was my first decision in this incident, and the most critical one.

The reasoning isn't complicated. Options A and B essentially make the wrong thing more stable; only C replaces it with the right thing.

When putting out fires, human instinct leans toward A. It's a small change, quick results, and you can wrap up in ten minutes. But A fixes this fire, not the next one. iptables-persistent can make rules survive a reboot, but the dev server still runs in production without process supervision or resource limits. Memory leaks go unmanaged, and builds are compiled on the spot every time.

My website is a purely static site. It doesn't need a Node process running constantly. Having Nginx serve static files directly is how it should have been from the start.

This trade-off is something AI can't provide. It can list the pros and cons of each option clearly, but how much change risk I'm willing to take for this fix is my call.

Rebuilding the Deployment Pipeline: Six Steps and Three Pitfalls

With the plan set, it was time to execute. I barely intervened; the AI assistant did everything on the server.

1. 1.

   pnpm build generates dist/, 5 static pages, about 2 seconds

2. 2.

   apt-get install nginx, includes systemd service, enabled + active

3. 3.

   Create deployment directory /var/www/xiaocao

4. 4.

   Write Nginx configuration

5. 5.

   Clean up old architecture: kill dev process and iptables -t nat -F PREROUTING

6. 6.

   Write one-click deployment script deploy.sh

We hit three pitfalls along the way, each worth discussing.

The first was a permissions issue. My project source is in ~/.openclaw/workspace/, which has 700 permissions. Nginx runs as the www-data user and can't read it. The solution was to use rsync to sync the build output to /var/www/xiaocao, then chown it to www-data. So the deployment script must include this sync step; you can't just point Nginx's root directly there.

The second was a zombie process. After killing the parent Astro dev process, the child process 35557 was still alive, holding the port, so I had to manually kill it.

The third was a routing issue. Astro builds to a /path/index.html structure, which Nginx's default try_files doesn't handle. It needs to be written like this:

⁠⁠⁠

location /{try_files$uri$uri/ $uri.html $uri/index.html =404;}

Besides try_files, there are three other Nginx settings worth configuring: a one-year long cache with immutable flag for the _astro/ directory, a 30-day cache for images/, and enabling gzip. The full configuration is on my website; click "Read More" at the end of the article for a copy-paste version.

That one-click deployment script later became the only entry point for updating my site. The core is three steps:

⁠⁠⁠

pnpm build sudorsync -a --delete "$SRC/dist/""$DEST/"sudo nginx -t &&sudo systemctl reload nginx

Build, sync, reload. Add set -e to stop on any error, and chown to www-data, and you have a deployment script that will last.

From the decision at 11:19 to the domain being live at 11:26, it took 7 minutes. The site was back, and this time it could survive a reboot.

Adding HTTPS: Two Pitfalls Tutorials Won't Tell You

Two days later, on the afternoon of August 21, the site was stable but still running in plain HTTP, with a glaring "Not Secure" warning in the browser. I was about to put the website link at the end of a WeChat article, and that warning was a bad look.

Again, two options.

A was Let's Encrypt with certbot: free, auto-renews every 90 days, stable direct connection in China. B was putting Cloudflare in front, which also gives CDN and Analytics, but nodes in China are unreliable.

I chose A, my second decision. The reason is simple: my readers are mainly in China, and I wouldn't bet on Cloudflare's free nodes' performance there. The benefits of a CDN aren't worth the loss from slower access.

The main process is really just one command:

⁠⁠⁠

certbot --nginx -d www.example.com -d example.com \  --non-interactive --agree-tos --email your@email.com --redirect

The certificate was installed, Nginx config was automatically rewritten, and a local curl https://127.0.0.1 returned 200.

Then it got stuck.

Accessing port 443 from the public internet timed out.

This is the first pitfall, and the one I think is most worth writing about. Tencent Cloud Lighthouse's security group doesn't open port 443 by default.

certbot had no issues, and Nginx had no issues. The problem was in the cloud provider's console, a place the AI assistant couldn't access.

Almost no certbot tutorial mentions this step, because the authors usually use self-hosted servers or machines where the port is already open. But you, someone who just bought a cloud server and wants to enable HTTPS, will get stuck here and start wondering if you installed certbot wrong.

This is also the clearest human-machine boundary in the entire article. The AI assistant can change any configuration on the server, but it doesn't have my cloud console credentials. It pinpointed the issue to the security group, and the rest had to be done by me. I logged into Tencent Cloud, opened port 443, came back, tested again: 200, 103 ms.

The second pitfall is more subtle. When certbot automatically modified the config, it kept my previous bare-domain redirect rule, which pointed to http://www. As a result, when a user visits https://bare-domain, they get redirected to http://www, then to https://www, adding an unnecessary hop.

The fix was one line:

⁠⁠⁠

sudosed -i 's|return 301 http://www.example.com|return 301 https://www.example.com|'\  /etc/nginx/sites-enabled/xiaocao

I also upgraded the deployment script, adding four health checks that run automatically after each deployment. They test local HTTPS, public HTTPS, HTTP redirect, and bare-domain redirect. All four must pass for the deployment to be considered successful.

⁠⁠⁠

curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" https://www.example.com/ curl -s -o /dev/null -w "%{http_code} -> %{redirect_url}\n" http://www.example.com/

The full four checks are also in the full article on my website.

At this point, I thought the HTTPS chapter was over.

Eight days later, I found out it wasn't.

The Hidden Pitfall: nginx -t Passing Doesn't Mean Your Edited File Is in Effect

Late last night, while fact-checking for this article, I had the AI assistant re-test the four redirect paths. The results were wrong: http://bare-domain still took two hops.

The AI assistant immediately gave me an explanation: the sed command on August 21 only fixed half of it—https://bare-domain was fixed, but http://bare-domain was missed.

It sounded plausible. I almost believed it.

It almost believed it too, until it went to change the config.

It edited /etc/nginx/sites-available/xiaocao following standard practice, nginx -t passed, systemctl reload nginx succeeded. Tested again.

Nothing changed in production.

Then it diffed the two files, and the truth came out.

/etc/nginx/sites-enabled/xiaocao

This file is not a symlink; it's a standalone regular file.

In standard Nginx practice, sites-enabled/ should contain symlinks pointing to sites-available/, with one source of truth per config. But on this machine, each directory had a nearly identical file, and they had been drifting apart since August 21.

And the truth was the opposite of that plausible explanation. The sed on August 21 was correct—it edited sites-enabled, which is the one actually in effect. The file in sites-available had never been touched and still had the old http://www redirect.

So last night, the AI assistant's first edit was to a dead file. It edited, tested, passed, reloaded, and nothing happened.

That's why I'm dedicating a whole section to it.

nginx -t passing doesn't mean the file you edited is in effect. It only checks syntax. Both configs are syntactically correct, so it says OK to both. It doesn't error; it just doesn't take effect.

One command to self-check:

⁠⁠⁠

readlink -f /etc/nginx/sites-enabled/your-site-name

If the output isn't a path under /etc/nginx/sites-available/, you have two configs drifting apart.

My third decision was to fix the root cause rather than patch it. I overwrote the live config with the one in effect, deleted the regular file, and restored the standard symlink. Now all four paths are direct, and https://www responds in 97 ms.

The Final Piece: Source Code Shouldn't Live on Only One Machine

The previous three upgrades addressed what to do if the service goes down. But a bigger risk remained: the source code existed only on this one server.

If the server goes down, the site is down, but a reboot can fix it. If the server is lost, the site is gone, and you start from scratch.

Late on August 27, I filled this gap with a GitHub private repository.

This time, the division of labor was most pronounced. The AI assistant installed the gh CLI, configured git config, initialized the repo, wrote .gitignore, created the remote repo, and pushed the initial commit—all in one go. I only provided three things: my email, the repo name, and the OAuth device code. From start to finish took 19 minutes, with 43 source files and 4,314 lines. dist, node_modules, .astro, and .env were all excluded by .gitignore.

There's a pitfall here that only OpenClaw users might hit, but if you do, it's serious.

My workspace directory ~/.openclaw/workspace/ is already a git repository, and the website project is a subdirectory. Running git status in the subproject makes git walk up to the workspace repo and list all files in the entire workspace.

It might seem convenient to reuse the existing repo. But what's in that workspace? There's MEMORY.md, the AI assistant's long-term memory, containing all my project decisions, collaboration conventions, and troubleshooting notes from the past six months. There's also the .agents directory with the entire configuration.

One git add -A would push everything to the public internet.

So you must run git init separately in the subproject. This decision not to reuse was my fourth decision, and the least technical one. It's not about architecture; it's about privacy boundaries.

So, What Did I Actually Do?

Four upgrades, complete timeline:

The commands were typed by the AI assistant, the configs were written by it, the pitfalls were hit by it, and it climbed out of them itself.

I made four decisions.

Of the four, only the third is a capability boundary—AI can't access that console. The other three are judgment boundaries. It can list all options and explain the pros and cons, but choosing one requires a person who will bear the consequences.

There's one more thing worth noting. Last night, its first explanation—that the 8/21 fix was only half done—was wrong. At the time, the evidence it had only supported one fact: the bare domain had two hops. But it conveniently filled in a whole causal chain that sounded professional, logically closed, and fully self-consistent.

It wasn't me who caught it; it was the AI hitting a wall.

After making the change and seeing no effect in production, it went back to diff and uncovered the config drift. If it hadn't taken action and had just given me that plausible explanation, I would have written this article with a wrong conclusion, and even taught readers how to fix a problem that didn't exist.

This is probably the most important thing to watch out for in human-AI collaboration. The wrong answers AI gives are often not the obviously absurd ones, but the ones that sound particularly reasonable. The only way to expose them is to let it hit the wall of the real world. Run it, test it, and see if reality agrees.

Back to that "again" at the beginning.

When the first crash happened, I chose manual recovery, costing ten minutes. For the second crash, I paid by rebuilding the entire deployment pipeline. If I had acted the first time, the second wouldn't have happened.

Every architecture upgrade is a bill from an incident. The difference is whether you pay proactively or reactively.

And when execution costs approach zero, the only thing you still have to pay is judgment.