FPM status with OpenMetrics format and FULL parameter

[dmamchyts]

Description

Hi,
at the first - thank for your job.

About issue: we use PHP 8.1.9 in k8s + prometheus.
Now we try to migrate from hipages/php-fpm_exporter:2.2.0-amd64 (https://github.com/hipages/php-fpm_exporter) to https://www.php.net/manual/en/fpm.status.php

https://localhost/fpm-status?full - this route returns all data what we want to track/monitor.
We use prometheus.
So, we try to read fpm-status in prometheus format.

https://localhost/fpm-status?openmetrics - this route returns some.
https://localhost/fpm-status?openmetrics&full - this route returns the same data like from https://localhost/fpm-status?openmetrics. It's mean that we can't see additional (like in https://localhost/fpm-status?full).

Looks like is expected behavior according with https://github.com/php/php-src/blob/master/sapi/fpm/fpm/fpm_status.c#L431 -
full_syntax = "";

Do have any plans about setting/adding correct data for case with prometheus format (like for TEXT - https://github.com/php/php-src/blob/master/sapi/fpm/fpm/fpm_status.c#L456)?

Thank you

[bukka]

Yeah this was left out by the look of it. Seems like the exporter supports following:

• processRequests
• processLastRequestMemory
• processLastRequestCPU
• processRequestDuration

It has got childName label to differentiate those so we should do the same.

[cdaguerre]

I stubled upon this issue for the same reason as @dmamchyts

I just wanted to share my 2 cents:

Labels

Parameter	Description
pid	The system PID of the process.
start time	The date/time at which the process started.

In openmetrics, I think pid and pool are labels that should be present on all per-process metrics.

Per process metrics

Parameter	Description
state	The state of the process - Idle, Running, ...
start since	The number of seconds since the process started.
requests	The total number of requests served.

# HELP phpfpm_process_state The current state of the process (0: Accepting/Idle, 1: Reading headers, 2: Getting request information, 3: Running, 4: Ending, 5: Finishing).
# TYPE phpfpm_process_state gauge
phpfpm_process_state{pool="$pool",pid="$pid"} 1

# HELP phpfpm_process_requests The total number of requests served.
# TYPE phpfpm_process_requests counter
phpfpm_process_requests{pool="$pool",pid="$pid"} 12

# HELP phpfpm_process_start_since_seconds The number of seconds since the process started.
# TYPE phpfpm_process_start_since_seconds counter
# UNIT phpfpm_process_start_since_seconds seconds
phpfpm_process_start_since_seconds{pool="$pool",pid="$pid"} 123

Last/current request metrics

Parameter	Description
request duration	The total time in seconds spent serving requests. Time in seconds serving the current or last request.
request method	The HTTP method of the last served request.
request uri	The URI of the last served request (after webserver processing, it may always be /index.php if you use a front controller pattern redirect).
content length	The length of the request body, in bytes, of the last request.
user	The HTTP user (PHP_AUTH_USER) of the last request.
script	The full path of the script executed by the last request. This will be '-' if not applicable (eg. status page requests).
last request cpu	The %cpu of the last request. This will be 0 if the process is not Idle because the calculation is done when the request processing is complete.
last request memory	The maximum amount of memory consumed by the last request. This will be 0 if the process is not Idle because the calculation is done when the request processing is complete.

Last request cpu, memory and content length could be exposed as gauges.
Given the high cardinality that could come from labels like request url, user i think they should not be exposed at all.
It might make sense to add request method as a label.
Not sure of the cardinality the script label would bring...

# HELP phpfpm_process_request_seconds Time in seconds serving the current or last request.
# TYPE phpfpm_process_request_seconds gauge
# UNIT phpfpm_process_request_seconds seconds
phpfpm_process_request_seconds{pool="$pool",pid="$pid"} 0.1223

# HELP phpfpm_process_request_cpu The %cpu of the last request. This will be 0 if the process is not Idle because the calculation is done when the request processing is complete.
# TYPE phpfpm_process_request_cpu gauge
phpfpm_process_request_cpu{pool="$pool",pid="$pid"} 0.2

# HELP phpfpm_process_request_memory_bytes The maximum amount of memory consumed by the last request. This will be 0 if the process is not Idle because the calculation is done when the request processing is complete.
# TYPE phpfpm_process_request_memory_bytes gauge
# UNIT phpfpm_process_request_memory_bytes bytes
phpfpm_process_request_memory_bytes{pool="$pool",pid="$pid"} 123

# HELP phpfpm_process_request_content_length_bytes The maximum amount of memory consumed by the last request. This will be 0 if the process is not Idle because the calculation is done when the request processing is complete.
# TYPE phpfpm_process_request_content_length_bytes gauge
# UNIT phpfpm_process_request_content_length_bytes bytes
phpfpm_process_request_content_length_bytes{pool="$pool",pid="$pid"} 123

My understanding is that any gauge-like metric you would compute of these over time would be (potentially highly) inaccurate depending on scrape intervals, traffic, the distribution of http request footprint in terms of memory and cpu usage, duration, etc...
Generally, to monitor those metrics, it would make more sense for a user to use log-based metrics instead of real-time fpm status...
I'm not sure if they should be exposed at all, but if so, probably not as histograms but as simple gauges.

[bukka]

Nicely written down. Thanks!

It all looks reasonable to me. The last request might make some sense as you can identify when some request is taking too long but agree that logs might be better here. Specifically we have got slowlog for this purpose in FPM. I guess we can just add it as we provide that info in other formats.

[PixiBixi]

Any update on the topic?

[bukka]

Based on my FPM TODO list, it should hopefully get done for PHP 8.4. It's too late for 8.3

[steinmb]

Looks like this issue, with the MR, stalled. Is anyone working on getting this ready to be merged?

[bukka]

I should hopefully have some time to look into this next month.

[moskitone]

Hi.

Are there any updates on the topic?

[jamespavett]

Any updates on this at all?

We are trying to migrate away from using hipages/php-fpm_exporter, but without the ability to output FULL metrics, this isn't a simple migration.

[bukka]

I was looking through this again and just to give an update where we are.

The last attempt to sort this out was #12759 but there were some issue and it also resulted in quite a bloat of the implementation.

It's just that OpenMetrics did not really fit to the current way how status is generated which was already quite messy. With addition of OpenMetrics it got even worse and with labels, it's just a real mess. The whole thing needs a proper refactoring and there was not time for that. I got it at the top of my priorities but it's not the only thing there. But I really hope to start this work at some point.