ssh-monitor/readme_eng.md

# ssh-monitor

Bash script that monitors **SSH**, **SUDO**, and **`systemd-logind`** events (local/graphical sessions, etc.), sends alerts via **Telegram** and **SMTP email**, optional backup webhook, automatic IP blocking (**ipset** + **iptables** / **ip6tables**: `hash:ip` sets with timeout, single `-m set` rule; details in **`IPSET-NOTES.txt`**), and a daily report.

## Repository contents

| File | Purpose |
|------|---------|
| **`ssh-monitor`** | Main loop: parses logs (**journalctl**: sshd, sudo, security events, **systemd-logind**), notification chain (Telegram, SMTP via **python3**, backup webhook); with **`ENABLE_IP_BAN=1`** tracks failed attempts, bans via **ipset**, reconciles with **iptables**/**ip6tables**, daily report, heartbeat, optional **Prometheus** textfile and JSON health. Modes: **`--check-config`**, **`--dry-run`**. |
| **`ssh-monitor-watchdog`** | Standalone script (usually via **systemd timer**): checks that the monitor service is active and the heartbeat file is fresh; on failure runs **`systemctl restart`** and sends Telegram or **`BACKUP_WEBHOOK_URL`**. Reads the same `TELEGRAM_*` / webhook from `/etc/ssh-monitor.conf`. |
| **`update_ssh_monitor.sh`** | Optional auto-update: `git pull` / `clone`, SHA256 compare with `/usr/local/bin/ssh-monitor`, restart `ssh-monitor.service`. See [docs/auto-update.ru.md](docs/auto-update.ru.md) (Russian). |

**System packages and dependencies** (install on the server before running): step-by-step in [docs/install-prerequisites.ru.md](docs/install-prerequisites.ru.md) (Russian).

## Configuration

The script reads `/etc/ssh-monitor.conf` in `KEY="value"` format.

1. Copy the example:
   - `sudo cp ./ssh-monitor.conf.example /etc/ssh-monitor.conf`
2. Restrict permissions:
   - `sudo chmod 600 /etc/ssh-monitor.conf`
3. Fill in at least **one notification channel** (otherwise the script exits immediately; see **«Mandatory requirement»** below and **«Notification channels and empty NOTIFY_CHAIN»**):
   - for **Telegram**: `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID`;
   - or configure **SMTP** using the variables below with `NOTIFY_ORDER=""` (auto-selected channels).

**Mandatory requirement:** after loading the config, **`NOTIFY_CHAIN`** must contain **at least one** channel (`telegram` or `email`). If there are **none** (empty `NOTIFY_ORDER` and neither Telegram nor SMTP qualify as “configured”, or `NOTIFY_ORDER` contains only unknown names), the script **does not** enter the main loop and exits with code **1**, stderr: **`Не настроен ни один канал отправки оповещений`**. The same applies to **`--check-config`** and **`--dry-run`**. The backup **`BACKUP_WEBHOOK_URL`** is **not** part of this check — it is used only if **every** channel in **`NOTIFY_CHAIN`** fails to deliver.

Supported parameters:

- `TELEGRAM_BOT_TOKEN`, `TELEGRAM_CHAT_ID`
- `NOTIFY_ORDER` — channel list (`telegram`, `email`, or **tg**, **mail**); for each alert the script **tries all listed** channels in order (not “first success stops”). Empty = only configured channels are chained (default order in `ssh-monitor.conf.example`). If the resulting chain is empty — the script does not start (see **«Notification channels and empty NOTIFY_CHAIN»**).
- `MAIL_SMTP_HOST`, `MAIL_SMTP_PORT`, `MAIL_SMTP_USER`, `MAIL_SMTP_PASSWORD`, `MAIL_FROM`, `MAIL_TO`, `MAIL_SMTP_STARTTLS`, `MAIL_SMTP_SSL` — mail via **python3** (see channel table below)
- `BACKUP_WEBHOOK_URL` — fallback JSON `{"text":"..."}` (e.g. Slack Incoming Webhook) if **all** **`NOTIFY_CHAIN`** channels fail; **does not** replace the need for at least one primary channel (see step 3)
- `LOG_FILE`, `LAST_HEARTBEAT_FILE`, `LAST_REPORT_FILE`, `LAST_SSH_CHECK_FILE`, `LAST_SUDO_CHECK_FILE`, `LAST_SECURITY_EVENTS_FILE`, `LAST_LOGIND_CHECK_FILE`, `BAN_LIST_FILE`
- `ENABLE_LOGIND_MONITOR`, `LOGIND_NOTIFY_NEW`, `LOGIND_NOTIFY_REMOVED`, `LOGIND_NOTIFY_FAILED`, `LOGIND_SKIP_REMOTE` — see **«systemd-logind monitoring»** below
- `DAILY_REPORT_HOUR` (0..23), `DAILY_REPORT_TZ` (optional), `NOTIFY_TZ` (optional), `SERVER_DISPLAY_NAME` (optional), `SSH_ACCEPT_NOTIFY_DEDUP_SEC`, `DAILY_REPORT_TOP_IPS`
- `BRUTE_WINDOW_SEC`, `BRUTE_MIN_FAILS`, `BRUTE_NOTIFY_COOLDOWN_SEC`
- `PROMETHEUS_TEXTFILE_DIR` — directory for `ssh_monitor.prom` (node_exporter textfile collector)
- `HEALTHCHECK_STATUS_FILE` — path to JSON with last loop iteration marker
- `ENABLE_IP_BAN` — `1` (default): monitoring and auto-ban via ipset; `0`: monitoring and alerts only (no ipset/ban)
- `BAN_TIME`, `MAX_ATTEMPTS`, `BAN_CHECK_INTERVAL`, `MONITOR_INTERVAL`
- `WHITELIST_IPS`, `WHITELIST_SUBNETS` (CSV, e.g. `ip1,ip2`)
- `WATCHDOG_MAX_HEARTBEAT_AGE`, `WATCHDOG_LOG_FILE`, `WATCHDOG_SERVICE_NAME`, `WATCHDOG_NOTIFY_ON_RECOVERY`

If `/etc/ssh-monitor.conf` is missing or some values are empty, script defaults apply.

### `/etc/ssh-monitor.conf` variables

- `TELEGRAM_BOT_TOKEN` — Telegram bot token for alerts.
- `TELEGRAM_CHAT_ID` — chat/user ID for notifications.
- `NOTIFY_ORDER` — CSV of channel names (`telegram`, `email`, `tg`, `mail`). For **each** alert, delivery is attempted **to all** listed channels (if one fails, others still run; failures go to **`LOG_FILE`**). Empty string: build chain from configured channels only (default order telegram → email; “configured” criteria — **table** in **«Notification channels and empty NOTIFY_CHAIN»**). If no channels remain, the script exits: **«Не настроен ни один канал отправки оповещений»** (including `--check-config` and `--dry-run`).
- `MAIL_SMTP_HOST`, `MAIL_SMTP_PORT`, `MAIL_SMTP_USER`, `MAIL_SMTP_PASSWORD`, `MAIL_FROM`, `MAIL_TO`, `MAIL_SMTP_STARTTLS`, `MAIL_SMTP_SSL` — SMTP for `email` channel (**python3**; see “configured” table).
- `BACKUP_WEBHOOK_URL` — backup POST URL (JSON `{"text":"..."}`); fallback only, **not** counted as a configured channel (see step 3).
- `LOG_FILE` — main script log path.
- `LAST_HEARTBEAT_FILE` — file with last heartbeat timestamp.
- `LAST_REPORT_FILE` — date of last daily report `YYYY-MM-DD` (legacy Unix time on first run is interpreted automatically).
- `LAST_SSH_CHECK_FILE` — last SSH events check timestamp.
- `LAST_SUDO_CHECK_FILE` — last sudo events check timestamp.
- `LAST_SECURITY_EVENTS_FILE` — last “heavy” security journal check.
- `LAST_LOGIND_CHECK_FILE` — last `systemd-logind` journal processing (Unix time); ~30 minutes of history on first run.
- `BAN_LIST_FILE` — ban state file (IP, ban end time, metadata).
- `ENABLE_LOGIND_MONITOR` — `1` enables `journalctl -u systemd-logind`, `0` disables this block.
- `LOGIND_NOTIFY_NEW` — `1` send Telegram on new logind session (*New session … of user …*).
- `LOGIND_NOTIFY_REMOVED` — `1` send Telegram on session end (*Removed session …*); default `0` (log only to avoid spam).
- `LOGIND_NOTIFY_FAILED` — `1` send Telegram on logind lines containing *failed* (broad filter; disable if needed).
- `LOGIND_SKIP_REMOTE` — `1` (default): for a new logind session, if `loginctl` is available, skip duplicate Telegram when already covered by **`monitor_ssh`**: **`Type=ssh`** or **`Service`** points to **sshd** (often **`Type=tty` + `Service=sshd`**); short retries if `loginctl` returns empty. Set `0` for separate logind alerts for SSH sessions too.
- `DAILY_REPORT_HOUR` — hour (0..23) after which at most one daily report is sent in the **current calendar day** (in the zone below).
- `DAILY_REPORT_TZ` — optional **IANA** zone (`Europe/Moscow`, …). If **empty**, report uses the monitor process zone (usually matches `timedatectl` / `/etc/localtime`). If the unit sets `TZ=UTC`, without `DAILY_REPORT_TZ` the report uses **UTC** — set an explicit zone in config.
- `NOTIFY_TZ` — **IANA** zone for **«🕐 Время»** lines in Telegram/email. **Empty** — use **`DAILY_REPORT_TZ`** if set, else process zone. Common mismatch: **`ssh-monitor.service`** has **`Environment=TZ=UTC`** — set **`NOTIFY_TZ`** or **`DAILY_REPORT_TZ`** to your locale, e.g. `Asia/Vladivostok`.
- `SERVER_DISPLAY_NAME` — label in **«🖥️ Server»** line on every alert. **Empty** — `hostname` plus primary IPv4 when available. Set a fixed name when many hosts share one Telegram chat. Details: [docs/notifications.ru.md](docs/notifications.ru.md) (Russian).
- `SSH_ACCEPT_NOTIFY_DEDUP_SEC` — at most one Telegram per successful SSH (**Accepted**) per **user + IP** pair within **N** seconds (default **5**). Removes duplicates when journal has two lines with **different client ports** for one login. **`0`** disables this cross-iteration dedup (in-pass dedup by `user|ip|port` remains).
- `DAILY_REPORT_TOP_IPS` — how many IPs in the failed-attempts top for 24 hours.
- `BRUTE_WINDOW_SEC` — window (seconds) for “mass” brute-force evaluation via `journalctl`.
- `BRUTE_MIN_FAILS` — minimum failures in the window to alert.
- `BRUTE_NOTIFY_COOLDOWN_SEC` — cooldown between repeat alerts for the same IP.
- `PROMETHEUS_TEXTFILE_DIR` — if set to an existing directory, writes `ssh_monitor.prom` with `ssh_monitor_last_loop_unixtime` each iteration.
- `HEALTHCHECK_STATUS_FILE` — if set, updates JSON `{ "ts", "hostname", "dry_run" }` each iteration.
- `ENABLE_IP_BAN` — enable auto-ban (`1`) or monitor-only with alerts (`0`; **ipset** not required).
- `BAN_TIME` — ban duration in seconds (`ENABLE_IP_BAN=1`).
- `MAX_ATTEMPTS` — failed SSH attempts before auto-ban.
- `BAN_CHECK_INTERVAL` — interval to check expired bans and reconcile ban list with firewall (seconds).
- `MONITOR_INTERVAL` — pause between main loop iterations (seconds).
- `WHITELIST_IPS` — comma-separated whitelist (IPv4/IPv6 exact match).
- `WHITELIST_SUBNETS` — **IPv4-only** CIDR whitelist, comma-separated (e.g. `10.0.0.0/24`).
- `WATCHDOG_MAX_HEARTBEAT_AGE` — seconds after which heartbeat is stale.
- `WATCHDOG_LOG_FILE` — watchdog log file.
- `WATCHDOG_SERVICE_NAME` — systemd service watched by watchdog (default `ssh-monitor.service`).
- `WATCHDOG_NOTIFY_ON_RECOVERY` — `1` enables watchdog “OK” messages when healthy, `0` disables.

### Notification channels and empty NOTIFY_CHAIN

What counts as a **configured** channel (only these join the auto chain when **`NOTIFY_ORDER=""`**):

| Channel | “Configured” condition |
|---------|------------------------|
| `telegram` | **`TELEGRAM_BOT_TOKEN`** and **`TELEGRAM_CHAT_ID`** set |
| `email` | non-empty **`MAIL_SMTP_HOST`**, **`MAIL_FROM`**, **`MAIL_TO`**, **`python3`** in PATH |

If **`NOTIFY_CHAIN` is empty**, the script exits with **`Не настроен ни один канал отправки оповещений`** (see step 3 in **Configuration**).

### systemd-logind monitoring

- Collects **`systemd-logind`** unit messages via **`journalctl`** (`-o cat`): new/ended sessions, and lines containing *failed* (if enabled).
- Without **`journalctl`**, this block **does not run** (like other journal-based features).
- “New session” parsing expects typical English formats (`New session … of user …`). On mismatch, the line is still logged with a parse warning.
- **`loginctl`** is used for **`LOGIND_SKIP_REMOTE`**: **Type** and **Service** decide whether to avoid duplicate Telegram with **`monitor_ssh`** (see `LOGIND_SKIP_REMOTE` above).
- **`monitor_ssh`**: **sshd** events mostly **`journalctl _COMM=sshd`** (fallback units `sshd` / `ssh`), lines **`sort -u`**, **`Accepted`** dedup per **user|IP|port** within one pass; between passes **`SSH_ACCEPT_NOTIFY_DEDUP_SEC`** for **user|IP** (see config).

**`ssh-monitor --check-config`** prints current values, including logind settings.

Hosts **without journald** and **`/var/log/auth.log`** caveats: section 4 in [docs/install-prerequisites.ru.md](docs/install-prerequisites.ru.md).

### Server identification in alerts

**All** notifications sent via **`notify_send()`** get a **«🖥️ Сервер: …»** line (Russian label in messages) right after the title: `hostname (IPv4)` or hostname only. Implemented centrally in **`message_ensure_server_line()`**; no need to add the line in each alert builder.

- Requires **`hostname`** and **`ip`** (see prerequisites doc).
- Optional **`SERVER_DISPLAY_NAME`** in config for a fixed label.
- Message types and examples: [docs/notifications.ru.md](docs/notifications.ru.md) (Russian).

### Auto-update

**`update_ssh_monitor.sh`** can run on a schedule to `git pull`, compare SHA256 with `/usr/local/bin/ssh-monitor`, copy on change, and restart **`ssh-monitor.service`**. Does not modify `/etc/ssh-monitor.conf`.

Setup guide: [docs/auto-update.ru.md](docs/auto-update.ru.md) (Russian).

## Run modes

Normal run (root):

```bash
sudo bash ./ssh-monitor
```

Check config and syntax **without** starting the loop, **without** changing `iptables`, no mandatory Telegram send:

```bash
sudo bash ./ssh-monitor --check-config
```

**Dry-run** (firewall rules unchanged; notifications go to stderr instead of sending):

```bash
sudo bash ./ssh-monitor --dry-run
```

## Syntax check

```bash
bash -n ./ssh-monitor
```

## Release tarball

Version is set in the script as **`SSH_MONITOR_VERSION`** (as of last doc update — **1.1.3-server-label**; see the **`ssh-monitor`** file for the current value). Build from the git tree:

```bash
make dist
```

Produces `ssh-monitor-<version>.tar.gz` (via `git archive`). Prebuilt archives are attached to [GitHub releases](https://github.com/PTah/ssh-monitor/releases).

## Installing the script, config, and systemd

1. Install the script:
   - `sudo install -m 750 ./ssh-monitor /usr/local/bin/ssh-monitor`
2. Ensure config exists (see **Configuration**):
   - `sudo cp ./ssh-monitor.conf.example /etc/ssh-monitor.conf`
   - `sudo chmod 600 /etc/ssh-monitor.conf`
3. Create `/etc/systemd/system/ssh-monitor.service`:

```ini
[Unit]
Description=SSH Monitor with Telegram alerts
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/ssh-monitor
Restart=always
RestartSec=5
User=root

[Install]
WantedBy=multi-user.target
```

4. Enable and start:

```bash
sudo systemctl daemon-reload
sudo systemctl enable ssh-monitor
sudo systemctl start ssh-monitor
```

5. Status and logs:

```bash
sudo systemctl status ssh-monitor
sudo journalctl -u ssh-monitor -f
```

## Log rotation

Example `logrotate` config: `contrib/logrotate.d/ssh-monitor`. Copy to `/etc/logrotate.d/` and adjust paths for your `LOG_FILE` / `WATCHDOG_LOG_FILE` and state files under `/var/log/` (including `LAST_*` if you rotate them separately).

## Watchdog (automatic recovery)

The watchdog checks `ssh-monitor.service` activity and heartbeat freshness (`LAST_HEARTBEAT_FILE`). On failure it runs `systemctl restart` and sends a notification (Telegram, or `BACKUP_WEBHOOK_URL` if Telegram fails).

Step-by-step **ssh-monitor-watchdog** and **timer** setup: section 6 in [docs/install-prerequisites.ru.md](docs/install-prerequisites.ru.md).

**Keywords (search):** SSH, **sshd**, **bash** on **Linux**, **Telegram**, **SMTP**, **systemd-logind**, **sudo**, **journalctl**, **iptables** / **ip6tables**, automatic **IP ban** and **whitelist**, **systemd** unit, **daily report**, **heartbeat**, **OpenSSH**, server security, **Prometheus** textfile, optional **watchdog**.

## License

This project is licensed under the **MIT License** — see [LICENSE](LICENSE).