Troubleshooting

This section provides common solutions to issues that may occur while setting up or running Twilight Validator or Relayer Nodes. Before making changes, always check logs and verify configuration settings.

Overview

Twilight nodes rely on multiple interlinked services (Docker, Nyks, zkOS, PostgreSQL, Redis, Kafka). Issues can arise from configuration errors, missing dependencies, or network problems. This guide lists the most frequent problems and their fixes.

1

Node Startup Issues

Problem: Node fails to start or terminates immediately. Possible causes and fixes:

• Docker not running:

commands

sudo systemctl status docker
sudo systemctl enable --now docker

• Missing dependencies: Ensure docker and docker-compose are installed and at the correct versions.

• Incorrect architecture: Confirm you’re using the correct binary line in nyks/Dockerfile (arm64, amd64, etc.).

• .env syntax error: Check for missing quotes or equal signs in .env.

2

Network & Synchronization Problems

Problem: Node stays in “catching_up” state or cannot connect to peers.

Possible causes:

• Peers not reachable

• Firewall blocking required ports

• Outdated genesis or chain configuration

Solutions:

commands

curl http://localhost:26657/status
sudo ufw allow 26656,26657/tcp
git pull && docker-compose restart

If peer count remains zero, ensure your .env PEERS variable is properly set and validator addresses are valid.

3

Relayer Connectivity Issues

Problem: Relayer cannot reach the validator RPC or REST API.

Solutions:

• Confirm the validator node is running and RPC is active:

check-rpc

curl http://localhost:26657/status

• Verify relayer logs:

relayer-logs

docker logs -f relayer-core

• Check that Docker containers for Kafka, Redis, and PostgreSQL are running:

docker-ps

docker ps

If Kafka or Zookeeper repeatedly restart, increase available memory or check your docker-compose.yml for port conflicts.

4

Storage & Data Problems

Problem: Node or database fails to initialize due to corrupted data. Solution: Safely reset and rebuild node state:

reset

docker-compose down -v
docker-compose up -d

This deletes local blockchain and database data. Back up your .env and wallet keys before performing this step.

5

Log Diagnostics

Logs are your most powerful troubleshooting tool.

Common Commands

common-log-commands

docker logs -f validator
docker logs -f relayer
docker logs validator | grep error

To inspect configuration paths inside a container:

inspect-container

docker exec -it <container_id> /bin/bash
cd /root/.nyks/config

6

Port Conflicts

If Docker reports an error like “port already in use,” identify the conflicting process:

lsof-ports

sudo lsof -i :26656
sudo lsof -i :26657

Stop the process or change the port in your .env file before restarting.

7

Quick Recovery Commands

When in doubt, these commands resolve most runtime issues:

quick-recovery

git pull
docker-compose down
docker-compose build
docker-compose up -d

To completely reset the environment:

full-reset

docker-compose down -v

8

Additional Tips

• Verify Docker Compose file indentation — YAML errors can cause silent startup failures.

• If metrics are unavailable, ensure telemetry ports (e.g., 26660) are exposed.

• Use docker stats to monitor container resource usage.

• Keep system time synchronized:

ntp

sudo timedatectl set-ntp true

Related Sections

• Configuration

• Monitoring & Maintenance

• Validator Setup

• Relayer Setup

If issues persist, open a GitHub issue under twilight-project/testnets with your log excerpts and system details.