Extending cpuset to include user processes

Extending cpuset to include user processes - Printable Version

+- Snakeoil Forums (https://forums.snakeoil-os.net)
+-- Forum: Snakeoil Topics (https://forums.snakeoil-os.net/Forum-Snakeoil-Topics)
+--- Forum: Feature Requests (https://forums.snakeoil-os.net/Forum-Feature-Requests)
+--- Thread: Extending cpuset to include user processes (/Thread-Extending-cpuset-to-include-user-processes)

Extending cpuset to include user processes - Snoopy8 - 03-Jun-2021

Would like to see the ability for a user to add a process to be included in cpuset. E.g. would like dedicate a processor for CamillaDSP
p/s tried to manualy add cpuset but it appears that cpusets are reset at boot

RE: Extending cpuset to include user processes - agent_kith - 03-Jun-2021

Good idea. Really worth exploring CPUSet to see what it can do..

Havn't had much time to work on the new GUI past week, and unlikely to have the time with the coming long weekend. But will try and incorp this into the new GUI rework. Issue created and tracked here.

RE: Extending cpuset to include user processes - Snoopy8 - 20-Jun-2021

Some more information for next release of cpuset:

cpuset does not stop IRQs from using the user cpuset (CPUs 2,3). Ideally, to reduce latency, IRQs should be on a different CPU to the music player. My workaround is to edit
/etc/default/irqbalance and set IRQBALANCE_BANNED_CPUS=C (hex mask value, run in CPUs 0,1 not in CPUs 2,3)
This may help with setting a new cpuset for a user process. CamillaDSP was started as a system process at boot. Manually shifted using
sudo cset proc --move --toset=user --pid 723 --threads

Confirmed move worked using cset shield --shield -v
Code:
cset: "user" cpuset of CPUSPEC(2-3) with 11 tasks running USER PID PPID SPPr TASK NAME -------- ----- ----- ---- --------- root 723 1 Sf70 [camilladsp] root 738 1 Sf70 [camilladsp] root 746 1 Sf70 [camilladsp] root 747 1 Sf70 [AlsaPlayback] root 748 1 Sf70 [AlsaCapture] root 2125 1 Soth [mpd] root 2174 1 Soth [io] root 2175 1 Sf40 [rtio] root 2178 1 Soth [player] root 2182 1 Soth [decoder] root 2387 1 Sf40 [output] cset: done
Some issues with existing cpuset, a few player processes were not shifted to the user cpuset
LMS + Squeezlite R2 : squeeze2upnp-x8 (using DLNA/UPnP bridge) remained in the system cpuset
LMS + upmpdcli : squeeze2upnp-x8, mpd remained in the system cpuset
Minimserver : uses Java which remained in the system cpuset

Can confirm SQ improved with shifting IRQs to system cpuset and CamillaDSP to user cpuset.

*****
Added 22-Jun

A few more notes

Can move a few more kernel processes from user to system cpuset using
sudo cset shield -k on
While I heard no discernible difference with SQ, for completeness, should include this.
With upmpdcli, it is not possible to shift the underlying mpd process from system to user cpuset. Given that upmpdcli is only a wrapper and the work is done by mpd, cpuset may not provide a SQ benefit for upmpdcli?
Tried dedicating a cpu to LMS and another cpu to Squeezelite R2. No difference in SQ to running both LMS and Squeezelite R2 in same cpu.
Similarly no difference in SQ with mpd and CamillaDSP running on same cpu or on separate individual cpus.
However, with cpu intensive servers like Roon or HQplayer, there may be a benefit in running their players on a seperate cpu? Someone should test this.
If mympd is enabled (coming in next release), should shift it from system to user cpuset. I do not use mympd and will disable it.

RE: Extending cpuset to include user processes - agent_kith - 21-Jun-2021

Have updated the ticket with last post. Hoping to incorp this into the new GUI in the upcoming release.

RE: Extending cpuset to include user processes - Snoopy8 - 04-Jul-2021

Further adventures with cpuset; information may be useful for the further extension of cpuset within SnakeOil.

Have a 4 CPU Intel NUC and used this script for my mpd setup

Code:
#

# set up cpusets

#

cset set -c 0 -s system 

cset set -c 1 -s irq --cpu_exclusive

cset set -c 2 -s player --cpu_exclusive

cset set -c 3 -s server --cpu_exclusive

#

# move music server and player threads

#

cset proc --move --toset=player --pid $(pgrep mpd) --threads

cset proc --move --toset=server --pid $(pgrep camilladsp) --threads

#

# move all unbound kernel threads to system cpuset, and usb irq to irq cpuset

#

cset proc -k -f root -t system

cset proc --move --toset=irq --pid $(pgrep irq/123-xhci_hc) --kthread --force

Mpd and CamiilaDSP were moved to individual CPUs using a command similar to this:
"cset proc --move --toset=player --pid $(pgrep mpd) --threads".

Ideally, to reduce latency, IRQs should be on a different CPU to the music threads. Some IRQs were bound to CPUs, but many of the IRQs were moved using the following:

Edited /etc/default/irqbalance and set IRQBALANCE_BANNED_CPUS=C (hex mask value, run in CPUs 0,1 not in CPUs 2,3).
Alternately, disabled or uninstalled IRQBalance and more IRQs will run on the first CPU (CPU0)
Used "cset proc -k -f root -t system" to move some more IRQs
Used "cset proc --move --toset=irq --pid $(pgrep irq/123-xhci_hc) --kthread --force" to move USB IRQ to its own CPU.

Some music threads could not be moved eg. Minimserver is Java based. These music threads should be started within a cpuset using a command similar to this:
"cset proc -s server -e /bin/bash -- /var/www/players/minimserver/start.sh"

Some useful information on cpusets
https://documentation.suse.com/sle-rt/15-SP1/html/SLE-RT-all/cha-shielding-model.html
https://documentation.suse.com/sle-rt/15-SP1/html/SLE-RT-all/cha-shielding-cpuset.html

RE: Extending cpuset to include user processes - Snoopy8 - 10-Aug-2021

Part III. Had the following CPUsets (Intel NUC has 4 CPUs)

Code:
$cset set -l

cset:

         Name       CPUs-X    MEMs-X Tasks Subs Path

 ------------ ---------- - ------- - ----- ---- ----------

         root        0-3 y       0 y   207    4 /

       player          2 y       0 n     7    0 /player

       system          0 n       0 n    31    0 /system

       server          3 y       0 n     5    0 /server

          irq          1 y       0 n     1    0 /irq

The player CPUset ran the following mpd processes:

Code:
$ cset proc -l -s player

cset: "player" cpuset of CPUSPEC(2) with 7 tasks running

 USER       PID  PPID SPPr TASK NAME

 -------- ----- ----- ---- ---------

 root       401     2 Sr55 [irq/127-eth0]

 root      1082     1 Sr75 [mpd]

 root      1083     1 Sr75 [io]

 root      1084     1 Sr75 [rtio]

 root      1090     1 Sr75 [player]

 root      1091     1 Sr75 [decoder]

 root      1379     1 Sr75 [output]

The server CPUset was running DSP:

Code:
$ cset proc -l -s server

cset: "server" cpuset of CPUSPEC(3) with 5 tasks running

 USER       PID  PPID SPPr TASK NAME

 -------- ----- ----- ---- ---------

 root       589     1 Sr80 [camilladsp]

 root       628     1 Sr80 [camilladsp]

 root       631     1 Sr80 [camilladsp]

 root       632     1 Sr80 [AlsaPlayback]

 root       633     1 Sr80 [AlsaCapture]

The IRQ CPUset was running the USB output:

Code:
$ cset proc -l -s irq

cset: "irq" cpuset of CPUSPEC(1) with 1 task running

 USER       PID  PPID SPPr TASK NAME

 -------- ----- ----- ---- ---------

 root        87     2 Sr85 [irq/124-xhci_hc]

Note that processes had increasing proprities down the music chain. IRQ for Ethernet: 55, feeding Mpd: 75, then processed by CamillaDSP:80 and output to IRQ for USB: 85. This ensured that downstream processes were always ready to accept inputs.

With IRQBalance removed, IRQs ran in CPU0 (System CPUset). The USB IRQ, which was the most CPU intensive task, ran on its own CPU. The Ethernet IRQ, being the input for mpd, ran on the same CPU. Further optimisation was done by moving some kernel threads to CPU0 using "cset proc -k -f root -t system".

In addition, had a 5.10 RT Kernel (built my own), did BIOS tweaks (removed hyperthreading, virtual machine settings, unnecessary ports & features) and ran DSP (convolution filters).

With mpd playing music, ran "cyclictest -t4 -p 90 -N -s -i 10000 -l 10000 -q" twenty times, with the following results.

Filename: Cyclictest chart.jpg Size: 44.09 KB 10-Aug-2021, 07:00 PM

Maximum latencies were typically 5 to 7 µs, which is excellent. CPU 0 had three 8 µs and one 10 µs max latency, which is not an issue because it ran everything not directly associated with music playback. However, CPU 1 (USB IRQ) and CPU 2 (mpd) had one 9 µs max latency, which suggests there could be room for improvement? Will discuss things that require further investigation in a follow up post.

In summary, delighted with the tweaks that delivered SQ improvement for SnakeOil.

RE: Extending cpuset to include user processes - Snoopy8 - 13-Dec-2022

A note for AK when CPUset capabilities are further enhanced. Using Ubuntu 20.04, SnakeOil 1.2.9

With Upmpdcli, used the following to shift it

Code:
cset proc --move --toset=player --pid $(pgrep upmpdcli) --threads

The upmpdcli processes are successfully shifted to the player CPUset

Code:
$ cset proc -l -s player

cset: "player" cpuset of CPUSPEC(2) with 16 tasks running

 USER       PID  PPID SPPr TASK NAME

 -------- ----- ----- ---- ---------

 upmpdcli  1139     1 Sr60 [upmpdcli]

 upmpdcli  1141     1 Sr60 [upmpdcli]

 upmpdcli  1145     1 Sr60 [upmpdcli]

 upmpdcli  1149     1 Sr60 [upmpdcli]

 upmpdcli  1150     1 Sr60 [MHD-listen]

 upmpdcli  1151     1 Sr60 [MHD-listen]

 upmpdcli  1153     1 Sr60 [upmpdcli]

 upmpdcli  1597     1 Sr60 [upmpdcli]

 upmpdcli  1599     1 Sr60 [upmpdcli]

 upmpdcli  1689     1 Sr60 [upmpdcli]

 upmpdcli  5595     1 Sr60 [upmpdcli]

 upmpdcli  5638     1 Sr60 [upmpdcli]

 upmpdcli  5797     1 Sr60 [upmpdcli]

 upmpdcli  5798     1 Sr60 [upmpdcli]

 upmpdcli  5840     1 Sr60 [upmpdcli]

 upmpdcli  5843     1 Sr60 [MHD-connection]

Mpd have the following PIDs and should be in player CPUset (see above)

Code:
$ pgrep mpd

1139

1163

However, Top shows that Mpd can run in all the CPUs

Code:
PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND         P

     87 root     -86   0       0      0      0 S  10.0   0.0   2:18.91 irq/122-xhci_hc 1

   1163 root     -61   0  424452  23240  11960 S   6.0   0.1   1:29.56 mpd             3

   1139 upmpdcli -61   0 1749136  10220   7796 S   0.7   0.0   0:02.04 upmpdcli        2

Processor 2 is for Player CPUset and upmpdcli is correctly running in it. But Mpd is appearing in processor 3 in this snapshot (and different processors in other snapshots).

Note that Mpd running on its own can be assigned to a CPUset. It appears that the embedded Mpd process in upmpdcli cannot be locked into a CPUset. No idea why...