Skip to content

Commit 4d2fb17

Browse files
hzoppettihzoppetti
authored
new guide and images for logging and system monitoring (linode#3746)
1 parent f219b83 commit 4d2fb17

File tree

6 files changed

+210
-0
lines changed

6 files changed

+210
-0
lines changed

‎ci/vale/dictionary.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,7 @@ bram
130130
brd
131131
browserify
132132
bs4
133+
btmp
133134
bufio
134135
bugtracker
135136
buildbot
@@ -1670,6 +1671,7 @@ whitelisting
16701671
whitespace
16711672
whitespaces
16721673
whois
1674+
whowatch
16731675
wi
16741676
widgits
16751677
wildfly
@@ -1693,6 +1695,7 @@ writecaps
16931695
ws
16941696
wsgi
16951697
wsrep
1698+
wtmp
16961699
www
16971700
wx
16981701
xabbix

‎docs/guides/security/monitoring/logging-and-system-monitoring/Logging_and_system_monitoring.png‎

719 KB
Loading

‎docs/guides/security/monitoring/logging-and-system-monitoring/index.md‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
---
2+
slug: logging-and-system-monitoring
3+
author:
4+
name: Hackersploit
5+
description: 'Learn to analyze system logs like auth.log, wtmp, and btmp, and monitor system processes with utilities like top, htop, and whowatch.'
6+
og_description: 'Learn to analyze system logs like auth.log, wtmp, and btmp, and monitor system processes with utilities like top, htop, and whowatch.'
7+
keywords: ["linux log monitor", "linux system monitoring tools", "system monitor linux"]
8+
tags: ["monitoring","security"]
9+
license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)'
10+
published: 2020-10-29
11+
modified: 2020-10-29
12+
modified_by:
13+
name: Linode
14+
title: 'Logging and System Monitoring'
15+
h1_title: 'Logging and System Monitoring'
16+
aliases: ['/security/monitoring/logging-and-system-monitoring/']
17+
image: Logging_and_system_monitoring.png
18+
---
19+
20+
## Analyzing system logs
21+
22+
The following table highlights the most important log files and the data that is logged in them. These log files are particularly helpful/useful for monitoring and analyzing authentication attempts.
23+
24+
| Log File | Function |
25+
| -------- | -------- |
26+
| `/var/log/auth.log` | Contains information about security-related events, including authentication successes or failures and the IP addresses where the requests came from. |
27+
| `/var/log/wtmp` | The wtmp log file contains all login and logout events. It shows each instance of user login and logouts, how long the session was active for, and which host the connection came from. |
28+
| `/var/log/btmp` | Similar to wtmp, btmp is also a binary file you can touch to create if it doesn’t exist. This file can be used to find out bad login attempts. |
29+
30+
## Analyzing /var/log/auth.log
31+
32+
The `auth.log` file contains information about security-related events, including authentication successes or failures, and the IP addresses where the requests came from. You can use a utility like `grep` to display successful and failed SSH authentication attempts.
33+
34+
This can be done by running the following command:
35+
36+
sudo cat /var/log/auth.log | grep -e "sshd"
37+
38+
Example output:
39+
40+
{{< output >}}
41+
Oct 29 16:07:35 localhost sshd[406]: Server listening on 0.0.0.0 port 22.
42+
Oct 29 16:07:35 localhost sshd[406]: Server listening on :: port 22.
43+
Oct 29 16:07:35 localhost sshd[406]: Received signal 15; terminating.
44+
Oct 29 16:07:37 localhost sshd[625]: Server listening on 0.0.0.0 port 22.
45+
Oct 29 16:07:37 localhost sshd[625]: Server listening on :: port 22.
46+
Oct 29 16:08:42 localhost sshd[646]: Connection closed by 203.0.113.0 port 54614 [preauth]
47+
Oct 29 16:09:37 localhost sshd[648]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=203.0.113.0 user=root
48+
Oct 29 16:09:39 localhost sshd[648]: Failed password for root from 203.0.113.0 port 54840 ssh2
49+
Oct 29 16:09:44 localhost sshd[648]: Accepted password for root from 203.0.113.0 port 54840 ssh2
50+
Oct 29 16:09:44 localhost sshd[648]: pam_unix(sshd:session): session opened for user root by (uid=0)
51+
Oct 29 16:10:10 localhost sshd[648]: Received disconnect from 203.0.113.0 port 54840:11: disconnected by user
52+
Oct 29 16:10:10 localhost sshd[648]: Disconnected from user root 203.0.113.0 port 54840
53+
Oct 29 16:10:10 localhost sshd[648]: pam_unix(sshd:session): session closed for user root
54+
Oct 29 16:10:21 localhost sshd[686]: Accepted password for exampleuser from 203.0.113.0 port 55060 ssh2
55+
Oct 29 16:10:21 localhost sshd[686]: pam_unix(sshd:session): session opened for user exampleuser by (uid=0)
56+
Oct 29 16:52:53 localhost sshd[14049]: Invalid user admin from 192.0.2.0 port 45429
57+
Oct 29 16:52:53 localhost sshd[14049]: pam_unix(sshd:auth): check pass; user unknown
58+
Oct 29 16:52:53 localhost sshd[14049]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.0.2.0
59+
60+
...
61+
{{</ output >}}
62+
63+
## Analyzing /var/log/wtmp
64+
65+
The wtmp log file contains all login and logout events. It shows each instance of user login and logouts, how long the session was active for, and which host the connection came from.
66+
67+
The `last` command shows a chronological history of user logins into the system since the `/var/log/wtmp` file was created. It’s a binary file, so you can’t `cat` or `grep` this file.
68+
69+
Instead, run the last command to view the output:
70+
71+
last -aiF
72+
73+
Sample output:
74+
75+
{{< output >}}
76+
exampleuser pts/0 Thu Oct 29 20:10:17 2020 still logged in 203.0.113.0
77+
exampleuser pts/0 Thu Oct 29 16:10:21 2020 - Thu Oct 29 16:53:29 2020 (00:43) 203.0.113.0
78+
root pts/0 Thu Oct 29 16:09:44 2020 - Thu Oct 29 16:10:10 2020 (00:00) 203.0.113.0
79+
reboot system boot Thu Oct 29 16:07:33 2020 still running 0.0.0.0
80+
81+
wtmp begins Thu Oct 29 16:07:33 2020
82+
{{</ output >}}
83+
84+
## Analyzing /var/log/btmp
85+
86+
Similar to wtmp, the `/var/log/btmp` log is also a binary file that can be used to view bad login attempts.
87+
88+
To access the contents of the file, use the `lastb` command:
89+
90+
sudo lastb -adF
91+
92+
Sample output:
93+
94+
{{< output >}}
95+
root ssh:notty Thu Oct 29 19:19:35 2020 - Thu Oct 29 19:19:35 2020 (00:00) 198.51.100.0
96+
support ssh:notty Thu Oct 29 16:53:37 2020 - Thu Oct 29 16:53:37 2020 (00:00) 198.51.100.1
97+
support ssh:notty Thu Oct 29 16:53:35 2020 - Thu Oct 29 16:53:35 2020 (00:00) 198.51.100.1
98+
guest ssh:notty Thu Oct 29 16:53:34 2020 - Thu Oct 29 16:53:34 2020 (00:00) 198.51.100.2
99+
ubnt ssh:notty Thu Oct 29 16:53:30 2020 - Thu Oct 29 16:53:30 2020 (00:00) 198.51.100.3
100+
ubnt ssh:notty Thu Oct 29 16:53:28 2020 - Thu Oct 29 16:53:28 2020 (00:00) 198.51.100.3
101+
guest ssh:notty Thu Oct 29 16:53:27 2020 - Thu Oct 29 16:53:27 2020 (00:00) 198.51.100.2
102+
guest ssh:notty Thu Oct 29 16:53:25 2020 - Thu Oct 29 16:53:25 2020 (00:00) 198.51.100.2
103+
root ssh:notty Thu Oct 29 16:53:23 2020 - Thu Oct 29 16:53:23 2020 (00:00) 198.51.100.0
104+
105+
btmp begins Thu Oct 29 16:09:39 2020
106+
{{</ output >}}
107+
108+
## List logged in users
109+
110+
You can list the active logged in users on the system by running the who command.
111+
112+
who
113+
114+
Sample output:
115+
116+
{{< output >}}
117+
exampleuser pts/0 2020-10-29 20:10 (203.0.113.0)
118+
{{</ output >}}
119+
120+
This displays the users logged in, their IP address, the time at which they logged in, and the duration of their session.
121+
122+
## Monitoring System Processes
123+
124+
You can monitor and analyze system processes on Linux with utilities like `top` or `htop`. Both of these tools provide a simple and effective view of the system processes and their resource consumption.
125+
126+
### top
127+
128+
To start the `top` utility, use the following command:
129+
130+
top
131+
132+
Sample output:
133+
134+
{{< output >}}
135+
top - 20:21:46 up 4:14, 1 user, load average: 0.00, 0.00, 0.00
136+
Tasks: 79 total, 1 running, 78 sleeping, 0 stopped, 0 zombie
137+
%Cpu(s): 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
138+
MiB Mem : 987.2 total, 276.1 free, 74.6 used, 636.5 buff/cache
139+
MiB Swap: 512.0 total, 512.0 free, 0.0 used. 761.2 avail Mem
140+
141+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
142+
1 root 20 0 23008 10256 7832 S 0.0 1.0 0:02.41 systemd
143+
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
144+
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
145+
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
146+
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H-kblockd
147+
8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
148+
9 root 20 0 0 0 0 S 0.0 0.0 0:00.14 ksoftirqd/0
149+
10 root 20 0 0 0 0 I 0.0 0.0 0:00.17 rcu_sched
150+
11 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_bh
151+
12 root rt 0 0 0 0 S 0.0 0.0 0:00.06 migration/0
152+
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
153+
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
154+
16 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 netns
155+
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kauditd
156+
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
157+
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
158+
20 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 writeback
159+
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
160+
22 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
161+
23 root 39 19 0 0 0 S 0.0 0.0 0:00.14 khugepaged
162+
24 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 crypto
163+
25 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kintegrityd
164+
26 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kblockd
165+
166+
...
167+
{{</ output >}}
168+
169+
Type `q` to quit out of the top utility.
170+
171+
### htop
172+
173+
You can also use `htop`. htop is a utility similar to top that offers much more functionality, like the ability to kill processes.
174+
175+
1. Install htop by running the following command:
176+
177+
sudo apt install htop
178+
179+
1. Launch htop by running the htop command in the terminal.
180+
181+
htop
182+
183+
1. Htop lists out all processes and sorts out processes in terms of resource consumption.
184+
185+
![htop lists processes and sorts by consumption](logging-and-monitoring-htop.png "htop lists processes and sort by consumption")
186+
187+
You can search for processes with the `F3` key and you can kill processes with the `F9` key. Exit htop by pressing `F10`.
188+
189+
## Monitoring SSH Connections
190+
191+
You can monitor active SSH connections with a utility called `whowatch`. Whowatch allows you to view all authenticated SSH sessions, their IP addresses, and what commands or processes they are running.
192+
193+
1. Install whowatch with the following command:
194+
195+
sudo apt install whowatch
196+
197+
1. Run it by using the command:
198+
199+
whowatch
200+
201+
1. From the list, you can select any user account by navigating with the arrow keys and pressing `Enter` or `Return`.
202+
203+
![Whowatch user list](logging-and-monitoring-whowatch-list.png "Whowatch user list")
204+
205+
1. You can see information about all the programs that a user is running.
206+
207+
![Whowatch detail view](logging-and-monitoring-whowatch-detail.png "Whowatch detail view")

‎docs/guides/security/monitoring/logging-and-system-monitoring/logging-and-monitoring-htop.png‎

854 KB
Loading

‎docs/guides/security/monitoring/logging-and-system-monitoring/logging-and-monitoring-whowatch-detail.png‎

181 KB
Loading

‎docs/guides/security/monitoring/logging-and-system-monitoring/logging-and-monitoring-whowatch-list.png‎

102 KB
Loading

0 commit comments

Comments
 (0)