I am using the following YAML for my prometheus-adapter installation.

prometheus:
  url: http://prometheus-server.prometheus.svc.cluster.local
  port: 80
 
rules:
  custom:
  - seriesQuery: 'http_duration{kubernetes_namespace!="",kubernetes_pod_name!=""}'
    resources:
      overrides:
        kubernetes_namespace: { resource: "namespace" }
        kubernetes_pod_name: { resource: "pod" }
    name:
      matches: "^(.*)_sum"
      as: "${1}_avg"
    metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)"

This YAML is installed with the following command.

helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter --values=./prometheus-adapter-values.yaml --namespace prometheus

After generating some load with hey, I tried looking for the _avg metric with the following command.

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq -r '.resources[] | select (.name | contains ("pods/hello_http"))'

This is the output.

{
  "name": "pods/hello_http_duration_sum",
  "singularName": "",
  "namespaced": true,
  "kind": "MetricValueList",
  "verbs": [
    "get"
  ]
}
{
  "name": "pods/hello_http_duration_count",
  "singularName": "",
  "namespaced": true,
  "kind": "MetricValueList",
  "verbs": [
    "get"
  ]
}
{
  "name": "pods/hello_http_duration_bucket",
  "singularName": "",
  "namespaced": true,
  "kind": "MetricValueList",
  "verbs": [
    "get"
  ]
}

1. Why is the _avg metric not seen? Note that, at the moment, the accuracy of the metricsQuery is not important. I just want to know, why is the _avg metric not seen?
2. Where do I look for logs? The prometheus-adapter or the prometheus-server logs didn't show anything obvious.
3. Do I need additionalScrapeConfigs as described here?
4. This post similar to mine; however, my configuration matches that of OP. What am I missing?