Study: DeveloperTools(DevTool)/DevTool: Container

[Docker] ๋„์ปค ์šฉ๋Ÿ‰ ํ™•์ธ ๋ฐ ๊ด€๋ฆฌ (feat. ์„œ๋ฒ„์˜ ์šฉ๋Ÿ‰์ด ๊ฝ‰ ์ฐจ๋ฒ„๋ ธ๋‹ค)

DrawingProcess 2022. 10. 29. 13:25
๋ฐ˜์‘ํ˜•
๐Ÿ’ก ๋ณธ ๋ฌธ์„œ๋Š” 'Docker ์šฉ๋Ÿ‰ ํ™•์ธ ๋ฐ ๊ด€๋ฆฌ'์— ๋Œ€ํ•ด ์ •๋ฆฌํ•ด๋†“์€ ๊ธ€์ž…๋‹ˆ๋‹ค.
์ตœ๊ทผ์— 1TB์˜ ์„œ๋ฒ„์˜ ์šฉ๋Ÿ‰ ๊ฝ‰์ฐจ์„œ DB์— ๋ฐ์ดํ„ฐ๊ฐ€ ๋“ค์–ด๊ฐ€์ง€ ์•Š๋Š” ์ƒํ™ฉ์ด ๋ฐœ์ƒํ–ˆ์Šต๋‹ˆ๋‹ค. ์ด์œ ๋ฅผ ์ฐพ์•„๋ณด๋‹ˆ docker์˜ ๋ฌธ์ œ๋กœ ๋ฐœ๊ฒฌ๋˜์—ˆ์œผ๋ฉฐ ์ด๋ฅผ ํ•ด๊ฒฐํ•œ ๊ณผ์ •๋ฅผ ์ •๋ฆฌํ•˜๊ณ  ๊ณผ์ •์—์„œ ์–ป์€ Docker ์šฉ๋Ÿ‰ ํ™•์ธ ๋ฐ ๊ด€๋ฆฌ ๋ฐฉ๋ฒ•์„ ์ •๋ฆฌํ•ด๋†“์•˜์œผ๋‹ˆ ์ฐธ๊ณ ๋ถ€ํƒ๋“œ๋ฆฝ๋‹ˆ๋‹ค.

1. OS Level (Debian ๊ณ„์—ด)

(๋งˆ์šดํŠธ ๋œ) ๋””์Šคํฌ ์šฉ๋Ÿ‰ ํ™•์ธ

df -h

๋””๋ ‰ํ† ๋ฆฌ ๋””์Šคํฌ ์‚ฌ์šฉ๋Ÿ‰ ํ™•์ธ: ํŒŒ์ผ๋ณ„ ์šฉ๋Ÿ‰ ํฐ์ˆœ์„œ๋กœ

sudo du -sm * | sort -nr

๋””๋ ‰ํ† ๋ฆฌ ๋””์Šคํฌ ์‚ฌ์šฉ๋Ÿ‰ ํ™•์ธ: ํŒŒ์ผ๋ณ„ ์šฉ๋Ÿ‰ ์ฝ์„ ์ˆ˜ ์žˆ๊ฒŒ

sudo du -sh * | sort -nr

2. Docker ๊ด€๋ จ

์•„๋ž˜ ๋ช…๋ น์–ด๋Š” ์ „๋ถ€ ํ˜„์žฌ ์‹คํ–‰๋˜๊ณ  ์žˆ์ง€ ์•Š์€ ์ด๋ฏธ์ง€, ์ปจํ…Œ์ด๋„ˆ, ์บ์‹œ ๋“ฑ์„ ์‚ญ์ œํ•˜๋Š” ๋ช…๋ น์–ด์ž…๋‹ˆ๋‹ค.

2.1 Docker ์ปจํ…Œ์ด๋„ˆ, ์ด๋ฏธ์ง€, ์บ์‹œ ๊ด€๋ฆฌ

๋„์ปค ์šฉ๋Ÿ‰ ํ™•์ธ

docker system df --verbose

๋„์ปค ๋ฏธ์‚ฌ์šฉ ์ปจํ…Œ์ด๋„ˆ ์‚ญ์ œ

docker container prune

๋„์ปค ๋ฏธ์‚ฌ์šฉ ์ด๋ฏธ์ง€ ์‚ญ์ œ

docker image prune

๋„์ปค ๋ฏธ์‚ฌ์šฉ ๋ณผ๋ฅจ ์‚ญ์ œ

docker volume prune

๋„์ปค ๋ฏธ์‚ฌ์šฉ ์˜ค๋ธŒ์ ํŠธ ์ „์ฒด ์‚ญ์ œ (๋กœ๊ทธ๋Š” ์‚ญ์ œ ์•ˆ๋จ)

docker system prune

2.2 Docker ๋กœ๊ทธ ๊ด€๋ฆฌ

2.2.1 Docker ๋กœ๊ทธ (๊ธฐ๋ณธ)

๋„์ปค ๋กœ๊ทธ ์šฉ๋Ÿ‰ ํ™•์ธ

sudo du -h $(docker inspect --format='{{.LogPath}}' $(docker ps -qa))

๋„์ปค ๋กœ๊ทธ ์ „์ฒด ์‚ญ์ œ

sudo sh -c "truncate -s 0 /var/lib/docker/containers/*/*-json.log"

๋„์ปค ๋กœ๊ทธ ์šฉ๋Ÿ‰ ์ œํ•œ ๋ฐฉ๋ฒ•(compose ํŒŒ์ผ)

2.2.2 Docker ๋กœ๊ทธ (Loki ํ”Œ๋Ÿฌ๊ทธ์ธ ์„ค์ •)

grafana/loki-docker-driver๋ฅผ ์„ค์น˜ํ•ด์„œ docker logger๋กœ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ๋‹ค๋ฉด ์•ž์„  ๋ฐฉ๋ฒ•์€ ๋จนํžˆ์ง€ ์•Š์„ ๊ฒƒ์ด๋‹ค. ์ด๋Ÿด ๋•Œ๋Š” ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•ด์•ผ ํ•œ๋‹ค.

๋„์ปค Loki ํ”Œ๋Ÿฌ๊ทธ์ธ ๋กœ๊ทธ ํ™•์ธ: sudo su๋กœ ๊ด€๋ฆฌ์ž ๊ณ„์ •์œผ๋กœ ๋ฐ”๊ฟ”์ค€ ํ›„ ๊ฐ€๋Šฅ

sudo su
sudo du -h /var/lib/docker/plugins/*/rootfs/var/log/docker/*/json.log

๋„์ปค ๋กœ๊ทธ ์ „์ฒด ์‚ญ์ œ

truncate -s 0 /var/lib/docker/plugins/*/rootfs/var/log/docker/*/json.log

๋„์ปค ๋กœ๊ทธ ์šฉ๋Ÿ‰ ์ œํ•œ ๋ฐฉ๋ฒ•(daemon.json ํŒŒ์ผ)

์ž์„ธํ•œ ๋‚ด์šฉ์€ Loki docker log driver logs are never cleaned up, eventually filling up the disk์— ์–ธ๊ธ‰๋œ ๋ฐฉ๋ฒ•์„ ์ฐธ๊ณ ํ•˜์‹œ๊ธฐ ๋ฐ”๋ž๋‹ˆ๋‹ค.

 

์ฐธ๊ณ 

๋ฐ˜์‘ํ˜•