Category: Tech

This is a cookbook style on how to set a limit (ulimit style) on your custom services that is managed by systemd.

Usecase

Why would you want to do something like this?

You might be running on a small server (or instance if you are using cloud services) and want to prevent your application from affecting other services sharing that server (think of noisy neighbor problem).

Generally, Linux kernel scheduler does a good job of fairly sharing system resources, but that is assuming you have a well behaved application.

Sometime you want to pack applications tightly and don’t mind less performant applications.

In summary, there are lots of reasons why you might want to tune the resources allocated to your applications.

Luckily, if you are using systemd as the controller, you can take advantage of its capabilities.

Note:

There are some caveats. You need to be using a fairly recent kernel and Linux distrob, either Ubuntu/Debian or recent CentOS/RedHat/Fedora.

What

I am going to show you how to get cloudquery run under systemd on an Ubuntu 20.04 LTS. The reason that I want to do this is because cloudquery will use as much memory as it can and trigger Linux OOM killer.

How

There are 3 files needed:

• /etc/default/cloudquery
  
  • This file contains definition for CQ_SERVICE_ACCOUNT_KEY_JSON, the value of which is the json content of your service account key file.
  • Example:
    • CQ_SERVICE_ACCOUNT_KEY_JSON='{ "type": "service_account", "project_id": "foobar", "private_key_id": "1a23b456cd134", "private_key": "-----BEGIN PRIVATE KEY-----\n.....vA8r\n-----END PRIVATE KEY-----\n", "client_email": "[email protected]", "client_id": "1234567890", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/foobar-sa%40foobar.iam.gserviceaccount.com" }'

• /lib/systemd/system/cloudquery_limit.slice
  
  • [Unit]
Description=Slice that limits memory for all my services

[Slice]
# MemoryHigh works only in "unified" cgroups mode, NOT in "hybrid" mode
# Must add 'systemd.unified_cgroup_hierarchy=1' to GRUB_CMDLINE_LINUX_DEFAULT
# in /etc/default/grub
MemoryHigh=10240M
# MemoryMax works in "hybrid" cgroups mode, too
MemoryMax=10240M

• /etc/systemd/system/cloudquery.service
  
  • [Unit]
Description=Cloud Query
Documentation=cloudquery.README.md
After=network.target

[Service]
Slice=cloudquery_limit.slice
EnvironmentFile=-/etc/default/cloudquery
ExecStart=/usr/local/bin/cloudquery --config /data/cq/config.hcl fetch
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=simple
WorkingDirectory=/data/cq
RuntimeDirectory=cq
RuntimeDirectoryMode=0755
LimitNOFILE=64000
user=cloudquery
group=cloudquery

[Install]
WantedBy=multi-user.target
Alias=cloudquery.service

Once you have all 3 files in place and edited the values to match your particular system, you need to tell systemd to check its directory for the new service, by running

systemctl daemon-reload

Once you have done that, you can check to see if systemd see your new service, by running

systemctl list-unit-files|grep query

Smoke Test

Test to see if everything works by starting your service.

systemctl start cloudquery

Check (and debug) the status of your new service via

systemctl status cloudquery

and journalctl -xe

Thanks to the posts from https://unix.stackexchange.com/questions/436791/limit-total-memory-usage-for-multiple-instances-of-systemd-service for pointing me in the right direction.