[v4,0/5] Add LVTS support for mt8192

  From: Balsam CHIHI <bchihi@baylibre.com>

Add full LVTS support (MCU thermal domain + AP thermal domain) to MediaTek MT8192 SoC.
Also, add Suspend and Resume support to LVTS Driver (all SoCs),
and update the documentation that describes the Calibration Data Offsets.

Changelog:
    v4 :
        - Shrink the lvts_ap thermal sensor I/O range to 0xc00 to make
          room for SVS support, pointed out by
          AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>

    v3 : 
        - Rebased :
            base-commit: 6a3d37b4d885129561e1cef361216f00472f7d2e
        - Fix issues in v2 pointed out by Nícolas F. R. A. Prado <nfraprado@collabora.com>:
          Use filtered mode to make sure threshold interrupts are triggered,
          protocol documentation, cosmetics
        - I (bero@baylibre.com) will be taking care of this patchset
          from now on, since Balsam has left BayLibre. Thanks for
          getting it almost ready, Balsam!

    v2 :
        - Based on top of thermal/linux-next :
            base-commit: 7ac82227ee046f8234471de4c12a40b8c2d3ddcc
        - Squash "add thermal zones and thermal nodes" and
            "add temperature mitigation threshold" commits together to form
            "arm64: dts: mediatek: mt8192: Add thermal nodes and thermal zones" commit.
        - Add Suspend and Resume support to LVTS Driver.
        - Update Calibration Data documentation.
        - Fix calibration data offsets for mt8192
            (Thanks to "Chen-Yu Tsai" and "Nícolas F. R. A. Prado").
        https://lore.kernel.org/all/20230425133052.199767-1-bchihi@baylibre.com/
        Tested-by: Chen-Yu Tsai <wenst@chromium.org>

    v1 :
        - The initial series "Add LVTS support for mt8192" :
            "https://lore.kernel.org/all/20230307163413.143334-1-bchihi@baylibre.com/".

Balsam CHIHI (5):
  dt-bindings: thermal: mediatek: Add LVTS thermal controller definition
    for mt8192
  thermal/drivers/mediatek/lvts_thermal: Add suspend and resume
  thermal/drivers/mediatek/lvts_thermal: Add mt8192 support
  arm64: dts: mediatek: mt8192: Add thermal nodes and thermal zones
  thermal/drivers/mediatek/lvts_thermal: Update calibration data
    documentation

 arch/arm64/boot/dts/mediatek/mt8192.dtsi      | 454 ++++++++++++++++++
 drivers/thermal/mediatek/lvts_thermal.c       | 160 +++++-
 .../thermal/mediatek,lvts-thermal.h           |  19 +
 3 files changed, 631 insertions(+), 2 deletions(-)

base-commit: 8c33787278ca8db73ad7d23f932c8c39b9f6e543
  

On Wed, May 31, 2023 at 3:51 AM Bernhard Rosenkränzer <bero@baylibre.com> wrote:
>
> From: Balsam CHIHI <bchihi@baylibre.com>
>
> Add full LVTS support (MCU thermal domain + AP thermal domain) to MediaTek MT8192 SoC.
> Also, add Suspend and Resume support to LVTS Driver (all SoCs),
> and update the documentation that describes the Calibration Data Offsets.
>
> Changelog:
>     v4 :
>         - Shrink the lvts_ap thermal sensor I/O range to 0xc00 to make
>           room for SVS support, pointed out by
>           AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
>
>     v3 :
>         - Rebased :
>             base-commit: 6a3d37b4d885129561e1cef361216f00472f7d2e
>         - Fix issues in v2 pointed out by Nícolas F. R. A. Prado <nfraprado@collabora.com>:
>           Use filtered mode to make sure threshold interrupts are triggered,

I'm seeing sensor readout (either through sysfs/thermal/<x>/temp or hwmon)
fail frequently on MT8192. If I run `sensors` (lm-sensors), at least a couple
of the LVTS sensors would be N/A. Not sure if this is related to this change.


ChenYu

>           protocol documentation, cosmetics
>         - I (bero@baylibre.com) will be taking care of this patchset
>           from now on, since Balsam has left BayLibre. Thanks for
>           getting it almost ready, Balsam!
>
>     v2 :
>         - Based on top of thermal/linux-next :
>             base-commit: 7ac82227ee046f8234471de4c12a40b8c2d3ddcc
>         - Squash "add thermal zones and thermal nodes" and
>             "add temperature mitigation threshold" commits together to form
>             "arm64: dts: mediatek: mt8192: Add thermal nodes and thermal zones" commit.
>         - Add Suspend and Resume support to LVTS Driver.
>         - Update Calibration Data documentation.
>         - Fix calibration data offsets for mt8192
>             (Thanks to "Chen-Yu Tsai" and "Nícolas F. R. A. Prado").
>         https://lore.kernel.org/all/20230425133052.199767-1-bchihi@baylibre.com/
>         Tested-by: Chen-Yu Tsai <wenst@chromium.org>
>
>     v1 :
>         - The initial series "Add LVTS support for mt8192" :
>             "https://lore.kernel.org/all/20230307163413.143334-1-bchihi@baylibre.com/".
>
> Balsam CHIHI (5):
>   dt-bindings: thermal: mediatek: Add LVTS thermal controller definition
>     for mt8192
>   thermal/drivers/mediatek/lvts_thermal: Add suspend and resume
>   thermal/drivers/mediatek/lvts_thermal: Add mt8192 support
>   arm64: dts: mediatek: mt8192: Add thermal nodes and thermal zones
>   thermal/drivers/mediatek/lvts_thermal: Update calibration data
>     documentation
>
>  arch/arm64/boot/dts/mediatek/mt8192.dtsi      | 454 ++++++++++++++++++
>  drivers/thermal/mediatek/lvts_thermal.c       | 160 +++++-
>  .../thermal/mediatek,lvts-thermal.h           |  19 +
>  3 files changed, 631 insertions(+), 2 deletions(-)
>
> base-commit: 8c33787278ca8db73ad7d23f932c8c39b9f6e543
> --
> 2.41.0.rc2
>
  

On Wed, May 31, 2023 at 12:49:43PM +0800, Chen-Yu Tsai wrote:
> On Wed, May 31, 2023 at 3:51 AM Bernhard Rosenkränzer <bero@baylibre.com> wrote:
> >
> > From: Balsam CHIHI <bchihi@baylibre.com>
> >
> > Add full LVTS support (MCU thermal domain + AP thermal domain) to MediaTek MT8192 SoC.
> > Also, add Suspend and Resume support to LVTS Driver (all SoCs),
> > and update the documentation that describes the Calibration Data Offsets.
> >
> > Changelog:
> >     v4 :
> >         - Shrink the lvts_ap thermal sensor I/O range to 0xc00 to make
> >           room for SVS support, pointed out by
> >           AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
> >
> >     v3 :
> >         - Rebased :
> >             base-commit: 6a3d37b4d885129561e1cef361216f00472f7d2e
> >         - Fix issues in v2 pointed out by Nícolas F. R. A. Prado <nfraprado@collabora.com>:
> >           Use filtered mode to make sure threshold interrupts are triggered,
> 
> I'm seeing sensor readout (either through sysfs/thermal/<x>/temp or hwmon)
> fail frequently on MT8192. If I run `sensors` (lm-sensors), at least a couple
> of the LVTS sensors would be N/A. Not sure if this is related to this change.

Yes, it is. Filtered mode has some delay associated with reading, meaning most
of the time the value isn't ready, while immediate mode is, well, pretty much
immediate and the read always succeeds.

For temperature monitoring, filtered mode should be used. It supports triggering
interrupts when crossing the thresholds. Immediate mode is meant for one-off
readings of the temperature. This is why I suggested using filtered mode.

As far as the thermal framework goes, it's ok that filtered mode doesn't always
return a value, as it will keep the old one. But of course, having the
temperature readout always work would be a desired improvement.

As for ways to achieve that, I think the intended way would be to enable the
interrupts that signal data ready on filtered mode (bits 19, 20, 21, 28), read
the temperature and cache it so it is always available when the get_temp()
callback is called. The issue with this is that it would cause *a lot* of
interrupts, which doesn't seem worth it.

Another option that comes to mind would be to enable immediate mode only during
the get_temp() callback, to immediately read a value, and return to filtered
mode at the end. That might work, but I haven't tried yet.

Thanks,
Nícolas
  

Il 01/06/23 19:09, Nícolas F. R. A. Prado ha scritto:
> On Wed, May 31, 2023 at 12:49:43PM +0800, Chen-Yu Tsai wrote:
>> On Wed, May 31, 2023 at 3:51 AM Bernhard Rosenkränzer <bero@baylibre.com> wrote:
>>>
>>> From: Balsam CHIHI <bchihi@baylibre.com>
>>>
>>> Add full LVTS support (MCU thermal domain + AP thermal domain) to MediaTek MT8192 SoC.
>>> Also, add Suspend and Resume support to LVTS Driver (all SoCs),
>>> and update the documentation that describes the Calibration Data Offsets.
>>>
>>> Changelog:
>>>      v4 :
>>>          - Shrink the lvts_ap thermal sensor I/O range to 0xc00 to make
>>>            room for SVS support, pointed out by
>>>            AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
>>>
>>>      v3 :
>>>          - Rebased :
>>>              base-commit: 6a3d37b4d885129561e1cef361216f00472f7d2e
>>>          - Fix issues in v2 pointed out by Nícolas F. R. A. Prado <nfraprado@collabora.com>:
>>>            Use filtered mode to make sure threshold interrupts are triggered,
>>
>> I'm seeing sensor readout (either through sysfs/thermal/<x>/temp or hwmon)
>> fail frequently on MT8192. If I run `sensors` (lm-sensors), at least a couple
>> of the LVTS sensors would be N/A. Not sure if this is related to this change.
> 
> Yes, it is. Filtered mode has some delay associated with reading, meaning most
> of the time the value isn't ready, while immediate mode is, well, pretty much
> immediate and the read always succeeds.
> 
> For temperature monitoring, filtered mode should be used. It supports triggering
> interrupts when crossing the thresholds. Immediate mode is meant for one-off
> readings of the temperature. This is why I suggested using filtered mode.
> 
> As far as the thermal framework goes, it's ok that filtered mode doesn't always
> return a value, as it will keep the old one. But of course, having the
> temperature readout always work would be a desired improvement.
> 
> As for ways to achieve that, I think the intended way would be to enable the
> interrupts that signal data ready on filtered mode (bits 19, 20, 21, 28), read
> the temperature and cache it so it is always available when the get_temp()
> callback is called. The issue with this is that it would cause *a lot* of
> interrupts, which doesn't seem worth it.
> 
> Another option that comes to mind would be to enable immediate mode only during
> the get_temp() callback, to immediately read a value, and return to filtered
> mode at the end. That might work, but I haven't tried yet.
> 

The issue with keeping all as filtered mode comes when we want to add MediaTek
SVS functionality which, on most SoCs, is used only for the GPU (apart from the
MT8183 which has cpu+gpu svs).

It makes sense to cache the readings, but I'm concerned about possible
instabilities that we could get through the SVS voltage adjustment flows, as
that algorithm takes current IP temperature to shape the DVFS "V" curve; please
keep in mind that this concern is valid only if temperature readings get updated
"very slowly" (>100ms would be too slow).

So, point of the situation:
  - Filtered mode, less than 100ms per temperature reading -> cache it, it's ok
  - Filtered mode, more than 100ms per temp reading -> switch GPU to Immediate mode.

Your call.

Keep up the good work!
- Angelo
  

On 01/06/2023 19:09, Nícolas F. R. A. Prado wrote:
> On Wed, May 31, 2023 at 12:49:43PM +0800, Chen-Yu Tsai wrote:
>> On Wed, May 31, 2023 at 3:51 AM Bernhard Rosenkränzer <bero@baylibre.com> wrote:
>>>
>>> From: Balsam CHIHI <bchihi@baylibre.com>
>>>
>>> Add full LVTS support (MCU thermal domain + AP thermal domain) to MediaTek MT8192 SoC.
>>> Also, add Suspend and Resume support to LVTS Driver (all SoCs),
>>> and update the documentation that describes the Calibration Data Offsets.
>>>
>>> Changelog:
>>>      v4 :
>>>          - Shrink the lvts_ap thermal sensor I/O range to 0xc00 to make
>>>            room for SVS support, pointed out by
>>>            AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
>>>
>>>      v3 :
>>>          - Rebased :
>>>              base-commit: 6a3d37b4d885129561e1cef361216f00472f7d2e
>>>          - Fix issues in v2 pointed out by Nícolas F. R. A. Prado <nfraprado@collabora.com>:
>>>            Use filtered mode to make sure threshold interrupts are triggered,
>>
>> I'm seeing sensor readout (either through sysfs/thermal/<x>/temp or hwmon)
>> fail frequently on MT8192. If I run `sensors` (lm-sensors), at least a couple
>> of the LVTS sensors would be N/A. Not sure if this is related to this change.
> 
> Yes, it is. Filtered mode has some delay associated with reading, meaning most
> of the time the value isn't ready, while immediate mode is, well, pretty much
> immediate and the read always succeeds.
> 
> For temperature monitoring, filtered mode should be used. It supports triggering
> interrupts when crossing the thresholds. Immediate mode is meant for one-off
> readings of the temperature. This is why I suggested using filtered mode.
> 
> As far as the thermal framework goes, it's ok that filtered mode doesn't always
> return a value, as it will keep the old one. But of course, having the
> temperature readout always work would be a desired improvement.
> 
> As for ways to achieve that, I think the intended way would be to enable the
> interrupts that signal data ready on filtered mode (bits 19, 20, 21, 28), read
> the temperature and cache it so it is always available when the get_temp()
> callback is called. The issue with this is that it would cause *a lot* of
> interrupts, which doesn't seem worth it.
> 
> Another option that comes to mind would be to enable immediate mode only during
> the get_temp() callback, to immediately read a value, and return to filtered
> mode at the end. That might work, but I haven't tried yet.

Why not understand why the filtered mode is unable to return temperature 
values most of the time?

I tried with the filtered mode and I can see 90% of the time it is not 
possible to read the temperature.

IIUC there are timings which can be setup, may be understand how to set 
them up in order to read the temperature correctly?

Caching values, switching the mode or whatever is hackish :/
  

On 01/06/2023 19:09, Nícolas F. R. A. Prado wrote:
> On Wed, May 31, 2023 at 12:49:43PM +0800, Chen-Yu Tsai wrote:
>> On Wed, May 31, 2023 at 3:51 AM Bernhard Rosenkränzer <bero@baylibre.com> wrote:
>>>
>>> From: Balsam CHIHI <bchihi@baylibre.com>
>>>
>>> Add full LVTS support (MCU thermal domain + AP thermal domain) to MediaTek MT8192 SoC.
>>> Also, add Suspend and Resume support to LVTS Driver (all SoCs),
>>> and update the documentation that describes the Calibration Data Offsets.
>>>
>>> Changelog:
>>>      v4 :
>>>          - Shrink the lvts_ap thermal sensor I/O range to 0xc00 to make
>>>            room for SVS support, pointed out by
>>>            AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
>>>
>>>      v3 :
>>>          - Rebased :
>>>              base-commit: 6a3d37b4d885129561e1cef361216f00472f7d2e
>>>          - Fix issues in v2 pointed out by Nícolas F. R. A. Prado <nfraprado@collabora.com>:
>>>            Use filtered mode to make sure threshold interrupts are triggered,
>>
>> I'm seeing sensor readout (either through sysfs/thermal/<x>/temp or hwmon)
>> fail frequently on MT8192. If I run `sensors` (lm-sensors), at least a couple
>> of the LVTS sensors would be N/A. Not sure if this is related to this change.
> 
> Yes, it is. Filtered mode has some delay associated with reading, meaning most
> of the time the value isn't ready, while immediate mode is, well, pretty much
> immediate and the read always succeeds.
> 
> For temperature monitoring, filtered mode should be used. 

Why?

> As far as the thermal framework goes, it's ok that filtered mode doesn't always
> return a value, as it will keep the old one.

It is not by design but just the code not returning the error when 
updating the thermal zone as it should so the side effect is giving the 
illusion everything is all right.

> But of course, having the
> temperature readout always work would be a desired improvement.

More than a desired improvement, it is mandatory. How can the thermal 
framework protect and mitigate the temperature with a twisted vision of 
the thermal situation?

> As for ways to achieve that, I think the intended way would be to enable the
> interrupts that signal data ready on filtered mode (bits 19, 20, 21, 28), read
> the temperature and cache it so it is always available when the get_temp()
> callback is called.

It sounds very strange how the filtered mode is working. We setup the 
filtered mode, then we shall receive an interrupt telling the data is 
ready so we can read it?

> The issue with this is that it would cause *a lot* of
> interrupts, which doesn't seem worth it.

Why not use the immediate mode + interrupts ?
  

On Thu, Jun 08, 2023 at 11:39:27AM +0200, Daniel Lezcano wrote:
> On 01/06/2023 19:09, Nícolas F. R. A. Prado wrote:
> > On Wed, May 31, 2023 at 12:49:43PM +0800, Chen-Yu Tsai wrote:
> > > On Wed, May 31, 2023 at 3:51 AM Bernhard Rosenkränzer <bero@baylibre.com> wrote:
> > > > 
> > > > From: Balsam CHIHI <bchihi@baylibre.com>
> > > > 
> > > > Add full LVTS support (MCU thermal domain + AP thermal domain) to MediaTek MT8192 SoC.
> > > > Also, add Suspend and Resume support to LVTS Driver (all SoCs),
> > > > and update the documentation that describes the Calibration Data Offsets.
> > > > 
> > > > Changelog:
> > > >      v4 :
> > > >          - Shrink the lvts_ap thermal sensor I/O range to 0xc00 to make
> > > >            room for SVS support, pointed out by
> > > >            AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
> > > > 
> > > >      v3 :
> > > >          - Rebased :
> > > >              base-commit: 6a3d37b4d885129561e1cef361216f00472f7d2e
> > > >          - Fix issues in v2 pointed out by Nícolas F. R. A. Prado <nfraprado@collabora.com>:
> > > >            Use filtered mode to make sure threshold interrupts are triggered,
> > > 
> > > I'm seeing sensor readout (either through sysfs/thermal/<x>/temp or hwmon)
> > > fail frequently on MT8192. If I run `sensors` (lm-sensors), at least a couple
> > > of the LVTS sensors would be N/A. Not sure if this is related to this change.
> > 
> > Yes, it is. Filtered mode has some delay associated with reading, meaning most
> > of the time the value isn't ready, while immediate mode is, well, pretty much
> > immediate and the read always succeeds.
> > 
> > For temperature monitoring, filtered mode should be used. It supports triggering
> > interrupts when crossing the thresholds. Immediate mode is meant for one-off
> > readings of the temperature. This is why I suggested using filtered mode.
> > 
> > As far as the thermal framework goes, it's ok that filtered mode doesn't always
> > return a value, as it will keep the old one. But of course, having the
> > temperature readout always work would be a desired improvement.
> > 
> > As for ways to achieve that, I think the intended way would be to enable the
> > interrupts that signal data ready on filtered mode (bits 19, 20, 21, 28), read
> > the temperature and cache it so it is always available when the get_temp()
> > callback is called. The issue with this is that it would cause *a lot* of
> > interrupts, which doesn't seem worth it.
> > 
> > Another option that comes to mind would be to enable immediate mode only during
> > the get_temp() callback, to immediately read a value, and return to filtered
> > mode at the end. That might work, but I haven't tried yet.
> 
> Why not understand why the filtered mode is unable to return temperature
> values most of the time?
> 
> I tried with the filtered mode and I can see 90% of the time it is not
> possible to read the temperature.
> 
> IIUC there are timings which can be setup, may be understand how to set them
> up in order to read the temperature correctly?
> 
> Caching values, switching the mode or whatever is hackish :/

So this is what I've found after some more testing.

With the current settings, using filtered mode, only about 30% of the
measurement reads return valid results:
rate: 29%   (success: 293, fail: 707)

While, as observed, in immediate mode, the reads always succeed:
rate: 100%   (success: 1000, fail: 0)

Changing the configurations so that the measurements take less time improve the
rate (and analogously increasing the time worsens the rate). That is, with
PERIOD_UNIT = 0, GROUP_INTERVAL = 0, FILTER_INTERVAL = 0, SENSOR_INTERVAL = 0,
HW_FILTER = 0 (ie single sample) the rate is much improved:
rate: 91%   (success: 918, fail: 82)

Though note that even though we're sampling as fast as possible and sampling
only once each time, so supposedly what immediate mode does, it's still not at
100% like in immediate mode.

Enabling the sensor 0 filter IRQ (bit 19) I've observed that it is triggered
about every 3500us (on the controller with all four sensors) with the current
settings, but after changing those timing registers, it happens every 344us.
With that in mind, in addition to those timing changes, if we also read the
register more than once with a timeout longer than that 344, that is,

rc = readl_poll_timeout(msr, value, value & BIT(16), 240, 400);

it's enough to get
rate: 100%   (success: 1000, fail: 0)
and even better:
rate: 100%   (success: 10000, fail: 0)

So it's still not exactly clear what's the relation of the VALID bit with the
timings in the hardware, but this at least gives us a way to get valid reads
without sacrificing interrupts.

Meanwhile, I've also tried reading the measurement during handling of the sensor
0 filter IRQ (bit 19), and while it definitely works much better than the
current 30%, giving a rate of 92%, it's still not 100%, which is intriguing
given this IRQ is supposed to signal the data is ready... I thought this might
be caused by timing issues, but increasing the timing of the measurements (by
setting PERIOD_UNIT = 120), lowered the rate to 84%.
Simply enabling this interrupt (and not reading the data in the IRQ), gives a
drastically worse rate:
rate: 3%   (success: 32, fail: 968)
Which I understand to mean that whenever the IRQ is cleared, the hardware
invalidates the previous measurement. So this IRQ is definitely related to the
VALID bit, but it also is unexpectedly influenced by the timings.

The VALID bit is also updated when read, and it tends to take the same time
between IRQs to be reset, so my understanding is that on every IRQ the VALID
bit is re-set to 1, and reading it clears it. But this does not explain why with
smaller intervals a single read has more chance of succeeding.

At this point, though, I feel like if it is possible to guarantee that readings
in filtered mode will always be valid, it must be some hidden setting in
LVTS_CONFIG. But with what we have access to, the best we can hope for is to
make the invalid reads extremely unlikely, which is what shrinking the intervals
and polling the register as shown above gives us, so it's what I suggest us to
do.

Thanks,
Nícolas
  

Hi Nicolas,

thanks for investigating !

On 15/06/2023 21:17, Nícolas F. R. A. Prado wrote:
> On Thu, Jun 08, 2023 at 11:39:27AM +0200, Daniel Lezcano wrote:
>> On 01/06/2023 19:09, Nícolas F. R. A. Prado wrote:
>>> On Wed, May 31, 2023 at 12:49:43PM +0800, Chen-Yu Tsai wrote:
>>>> On Wed, May 31, 2023 at 3:51 AM Bernhard Rosenkränzer <bero@baylibre.com> wrote:
>>>>>
>>>>> From: Balsam CHIHI <bchihi@baylibre.com>
>>>>>
>>>>> Add full LVTS support (MCU thermal domain + AP thermal domain) to MediaTek MT8192 SoC.
>>>>> Also, add Suspend and Resume support to LVTS Driver (all SoCs),
>>>>> and update the documentation that describes the Calibration Data Offsets.
>>>>>
>>>>> Changelog:
>>>>>       v4 :
>>>>>           - Shrink the lvts_ap thermal sensor I/O range to 0xc00 to make
>>>>>             room for SVS support, pointed out by
>>>>>             AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
>>>>>
>>>>>       v3 :
>>>>>           - Rebased :
>>>>>               base-commit: 6a3d37b4d885129561e1cef361216f00472f7d2e
>>>>>           - Fix issues in v2 pointed out by Nícolas F. R. A. Prado <nfraprado@collabora.com>:
>>>>>             Use filtered mode to make sure threshold interrupts are triggered,
>>>>
>>>> I'm seeing sensor readout (either through sysfs/thermal/<x>/temp or hwmon)
>>>> fail frequently on MT8192. If I run `sensors` (lm-sensors), at least a couple
>>>> of the LVTS sensors would be N/A. Not sure if this is related to this change.
>>>
>>> Yes, it is. Filtered mode has some delay associated with reading, meaning most
>>> of the time the value isn't ready, while immediate mode is, well, pretty much
>>> immediate and the read always succeeds.
>>>
>>> For temperature monitoring, filtered mode should be used. It supports triggering
>>> interrupts when crossing the thresholds. Immediate mode is meant for one-off
>>> readings of the temperature. This is why I suggested using filtered mode.
>>>
>>> As far as the thermal framework goes, it's ok that filtered mode doesn't always
>>> return a value, as it will keep the old one. But of course, having the
>>> temperature readout always work would be a desired improvement.
>>>
>>> As for ways to achieve that, I think the intended way would be to enable the
>>> interrupts that signal data ready on filtered mode (bits 19, 20, 21, 28), read
>>> the temperature and cache it so it is always available when the get_temp()
>>> callback is called. The issue with this is that it would cause *a lot* of
>>> interrupts, which doesn't seem worth it.
>>>
>>> Another option that comes to mind would be to enable immediate mode only during
>>> the get_temp() callback, to immediately read a value, and return to filtered
>>> mode at the end. That might work, but I haven't tried yet.
>>
>> Why not understand why the filtered mode is unable to return temperature
>> values most of the time?
>>
>> I tried with the filtered mode and I can see 90% of the time it is not
>> possible to read the temperature.
>>
>> IIUC there are timings which can be setup, may be understand how to set them
>> up in order to read the temperature correctly?
>>
>> Caching values, switching the mode or whatever is hackish :/
> 
> So this is what I've found after some more testing.
> 
> With the current settings, using filtered mode, only about 30% of the
> measurement reads return valid results:
> rate: 29%   (success: 293, fail: 707)
> 
> While, as observed, in immediate mode, the reads always succeed:
> rate: 100%   (success: 1000, fail: 0)
> 
> Changing the configurations so that the measurements take less time improve the
> rate (and analogously increasing the time worsens the rate). That is, with
> PERIOD_UNIT = 0, GROUP_INTERVAL = 0, FILTER_INTERVAL = 0, SENSOR_INTERVAL = 0,
> HW_FILTER = 0 (ie single sample) the rate is much improved:
> rate: 91%   (success: 918, fail: 82)
> 
> Though note that even though we're sampling as fast as possible and sampling
> only once each time, so supposedly what immediate mode does, it's still not at
> 100% like in immediate mode.
> 
> Enabling the sensor 0 filter IRQ (bit 19) I've observed that it is triggered
> about every 3500us (on the controller with all four sensors) with the current
> settings, but after changing those timing registers, it happens every 344us.
> With that in mind, in addition to those timing changes, if we also read the
> register more than once with a timeout longer than that 344, that is,
> 
> rc = readl_poll_timeout(msr, value, value & BIT(16), 240, 400);
> 
> it's enough to get
> rate: 100%   (success: 1000, fail: 0)
> and even better:
> rate: 100%   (success: 10000, fail: 0)
> 
> So it's still not exactly clear what's the relation of the VALID bit with the
> timings in the hardware, but this at least gives us a way to get valid reads
> without sacrificing interrupts.
> 
> Meanwhile, I've also tried reading the measurement during handling of the sensor
> 0 filter IRQ (bit 19), and while it definitely works much better than the
> current 30%, giving a rate of 92%, it's still not 100%, which is intriguing
> given this IRQ is supposed to signal the data is ready... I thought this might
> be caused by timing issues, but increasing the timing of the measurements (by
> setting PERIOD_UNIT = 120), lowered the rate to 84%.
> Simply enabling this interrupt (and not reading the data in the IRQ), gives a
> drastically worse rate:
> rate: 3%   (success: 32, fail: 968)
> Which I understand to mean that whenever the IRQ is cleared, the hardware
> invalidates the previous measurement. So this IRQ is definitely related to the
> VALID bit, but it also is unexpectedly influenced by the timings.
> 
> The VALID bit is also updated when read, and it tends to take the same time
> between IRQs to be reset, so my understanding is that on every IRQ the VALID
> bit is re-set to 1, and reading it clears it. But this does not explain why with
> smaller intervals a single read has more chance of succeeding.
> 
> At this point, though, I feel like if it is possible to guarantee that readings
> in filtered mode will always be valid, it must be some hidden setting in
> LVTS_CONFIG. But with what we have access to, the best we can hope for is to
> make the invalid reads extremely unlikely, which is what shrinking the intervals
> and polling the register as shown above gives us, so it's what I suggest us to
> do.
Let me summarize and check I'm understanding correctly:

1. Immediate mode

  - 100% successful read, no delay when reading
  - No interrupts when crossing the thresholds (at the first glance)

2. Filtered mode

  - Interrupts when data is ready
  - Interrupts when crossing the thresholds
  - Polling read until TMU valid
	- maximum two register reads
	- minimum delay 240us
	- maximum delay 480us

 From my POV, the filtered mode is not designed for an OSPM, it is for 
real time system for thermal acquisition or similar. It is unthinkable a 
sensor is firing so many interrupts waking up the CPU to tell a 
temperature is ready to be read. And it is strange we have to poll loop 
a register to read a temperature.

The thermal framework is designed to protect the silicon and 
consequently reads with non constant delay and/or high delay can have an 
impact on time sensitive governor. Skipping the temperature because we 
fail to read is also not acceptable, in the case of mitigation, that can 
have an impact.


The normal mode should be:

  - temperature below threshold => no wakeups
  - temperature crosses the threshold => interrupt fires
  - mitigation => wake up every 'passive' delay period

With the filtered mode we have:

  - temperature below threshold => interrupts telling the value is ready 
(we want to ignore that)

  - temperature crosses the threshold => interrupt but not sure we can 
read the temperature correctly

  - mitigation => wake up every 'passive' delay period but not sure we 
can read the temperature correctly

With the immediate mode:

  - temperature below threshold => interrupts is not working, so we have 
to monitor the temperature and wake up every <monitor> delay

  - temperature crosses the threshold => no interrupt, detected by the 
monitoring

  - mitigation => wake up every 'passive' delay period, temperature is 
accurate

It seems not logical to have the immediate mode not working with the 
interrupts when crossing the thresholds. I would say we should stick to 
the immediate mode and double check if the interrupt can work with this 
mode.
  

On Fri, Jun 16, 2023 at 04:14:43PM +0200, Daniel Lezcano wrote:
> 
> Hi Nicolas,
> 
> thanks for investigating !
> 
> On 15/06/2023 21:17, Nícolas F. R. A. Prado wrote:
> > On Thu, Jun 08, 2023 at 11:39:27AM +0200, Daniel Lezcano wrote:
> > > On 01/06/2023 19:09, Nícolas F. R. A. Prado wrote:
> > > > On Wed, May 31, 2023 at 12:49:43PM +0800, Chen-Yu Tsai wrote:
> > > > > On Wed, May 31, 2023 at 3:51 AM Bernhard Rosenkränzer <bero@baylibre.com> wrote:
> > > > > > 
> > > > > > From: Balsam CHIHI <bchihi@baylibre.com>
> > > > > > 
> > > > > > Add full LVTS support (MCU thermal domain + AP thermal domain) to MediaTek MT8192 SoC.
> > > > > > Also, add Suspend and Resume support to LVTS Driver (all SoCs),
> > > > > > and update the documentation that describes the Calibration Data Offsets.
> > > > > > 
> > > > > > Changelog:
> > > > > >       v4 :
> > > > > >           - Shrink the lvts_ap thermal sensor I/O range to 0xc00 to make
> > > > > >             room for SVS support, pointed out by
> > > > > >             AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
> > > > > > 
> > > > > >       v3 :
> > > > > >           - Rebased :
> > > > > >               base-commit: 6a3d37b4d885129561e1cef361216f00472f7d2e
> > > > > >           - Fix issues in v2 pointed out by Nícolas F. R. A. Prado <nfraprado@collabora.com>:
> > > > > >             Use filtered mode to make sure threshold interrupts are triggered,
> > > > > 
> > > > > I'm seeing sensor readout (either through sysfs/thermal/<x>/temp or hwmon)
> > > > > fail frequently on MT8192. If I run `sensors` (lm-sensors), at least a couple
> > > > > of the LVTS sensors would be N/A. Not sure if this is related to this change.
> > > > 
> > > > Yes, it is. Filtered mode has some delay associated with reading, meaning most
> > > > of the time the value isn't ready, while immediate mode is, well, pretty much
> > > > immediate and the read always succeeds.
> > > > 
> > > > For temperature monitoring, filtered mode should be used. It supports triggering
> > > > interrupts when crossing the thresholds. Immediate mode is meant for one-off
> > > > readings of the temperature. This is why I suggested using filtered mode.
> > > > 
> > > > As far as the thermal framework goes, it's ok that filtered mode doesn't always
> > > > return a value, as it will keep the old one. But of course, having the
> > > > temperature readout always work would be a desired improvement.
> > > > 
> > > > As for ways to achieve that, I think the intended way would be to enable the
> > > > interrupts that signal data ready on filtered mode (bits 19, 20, 21, 28), read
> > > > the temperature and cache it so it is always available when the get_temp()
> > > > callback is called. The issue with this is that it would cause *a lot* of
> > > > interrupts, which doesn't seem worth it.
> > > > 
> > > > Another option that comes to mind would be to enable immediate mode only during
> > > > the get_temp() callback, to immediately read a value, and return to filtered
> > > > mode at the end. That might work, but I haven't tried yet.
> > > 
> > > Why not understand why the filtered mode is unable to return temperature
> > > values most of the time?
> > > 
> > > I tried with the filtered mode and I can see 90% of the time it is not
> > > possible to read the temperature.
> > > 
> > > IIUC there are timings which can be setup, may be understand how to set them
> > > up in order to read the temperature correctly?
> > > 
> > > Caching values, switching the mode or whatever is hackish :/
> > 
> > So this is what I've found after some more testing.
> > 
> > With the current settings, using filtered mode, only about 30% of the
> > measurement reads return valid results:
> > rate: 29%   (success: 293, fail: 707)
> > 
> > While, as observed, in immediate mode, the reads always succeed:
> > rate: 100%   (success: 1000, fail: 0)
> > 
> > Changing the configurations so that the measurements take less time improve the
> > rate (and analogously increasing the time worsens the rate). That is, with
> > PERIOD_UNIT = 0, GROUP_INTERVAL = 0, FILTER_INTERVAL = 0, SENSOR_INTERVAL = 0,
> > HW_FILTER = 0 (ie single sample) the rate is much improved:
> > rate: 91%   (success: 918, fail: 82)
> > 
> > Though note that even though we're sampling as fast as possible and sampling
> > only once each time, so supposedly what immediate mode does, it's still not at
> > 100% like in immediate mode.
> > 
> > Enabling the sensor 0 filter IRQ (bit 19) I've observed that it is triggered
> > about every 3500us (on the controller with all four sensors) with the current
> > settings, but after changing those timing registers, it happens every 344us.
> > With that in mind, in addition to those timing changes, if we also read the
> > register more than once with a timeout longer than that 344, that is,
> > 
> > rc = readl_poll_timeout(msr, value, value & BIT(16), 240, 400);
> > 
> > it's enough to get
> > rate: 100%   (success: 1000, fail: 0)
> > and even better:
> > rate: 100%   (success: 10000, fail: 0)
> > 
> > So it's still not exactly clear what's the relation of the VALID bit with the
> > timings in the hardware, but this at least gives us a way to get valid reads
> > without sacrificing interrupts.
> > 
> > Meanwhile, I've also tried reading the measurement during handling of the sensor
> > 0 filter IRQ (bit 19), and while it definitely works much better than the
> > current 30%, giving a rate of 92%, it's still not 100%, which is intriguing
> > given this IRQ is supposed to signal the data is ready... I thought this might
> > be caused by timing issues, but increasing the timing of the measurements (by
> > setting PERIOD_UNIT = 120), lowered the rate to 84%.
> > Simply enabling this interrupt (and not reading the data in the IRQ), gives a
> > drastically worse rate:
> > rate: 3%   (success: 32, fail: 968)
> > Which I understand to mean that whenever the IRQ is cleared, the hardware
> > invalidates the previous measurement. So this IRQ is definitely related to the
> > VALID bit, but it also is unexpectedly influenced by the timings.
> > 
> > The VALID bit is also updated when read, and it tends to take the same time
> > between IRQs to be reset, so my understanding is that on every IRQ the VALID
> > bit is re-set to 1, and reading it clears it. But this does not explain why with
> > smaller intervals a single read has more chance of succeeding.
> > 
> > At this point, though, I feel like if it is possible to guarantee that readings
> > in filtered mode will always be valid, it must be some hidden setting in
> > LVTS_CONFIG. But with what we have access to, the best we can hope for is to
> > make the invalid reads extremely unlikely, which is what shrinking the intervals
> > and polling the register as shown above gives us, so it's what I suggest us to
> > do.
> Let me summarize and check I'm understanding correctly:
> 
> 1. Immediate mode
> 
>  - 100% successful read, no delay when reading
>  - No interrupts when crossing the thresholds (at the first glance)
> 
> 2. Filtered mode
> 
>  - Interrupts when data is ready

In fact, immediate mode also has interrupts for when data is ready (bits 16, 17,
18, 27 in MONINTST), but its frequency is so quick, that the VALID bit is always
set, and keeping these interrupts enabled would just cause unnecessary wakeups.

>  - Interrupts when crossing the thresholds
>  - Polling read until TMU valid
> 	- maximum two register reads
> 	- minimum delay 240us
> 	- maximum delay 480us

It's more like minimum 0us (it might already available) and maximum 344us
(though it will vary with timing settings, sampling setting, and number of
sensors in the controller)

> 
> From my POV, the filtered mode is not designed for an OSPM, it is for real
> time system for thermal acquisition or similar. It is unthinkable a sensor
> is firing so many interrupts waking up the CPU to tell a temperature is
> ready to be read. And it is strange we have to poll loop a register to read
> a temperature.

Indeed, that would explain the way this hardware behaves.

> 
> The thermal framework is designed to protect the silicon and consequently
> reads with non constant delay and/or high delay can have an impact on time
> sensitive governor. Skipping the temperature because we fail to read is also
> not acceptable, in the case of mitigation, that can have an impact.
> 
> 
> The normal mode should be:
> 
>  - temperature below threshold => no wakeups
>  - temperature crosses the threshold => interrupt fires
>  - mitigation => wake up every 'passive' delay period
> 
> With the filtered mode we have:
> 
>  - temperature below threshold => interrupts telling the value is ready (we
> want to ignore that)

And to be clear, we can ignore that.

> 
>  - temperature crosses the threshold => interrupt but not sure we can read
> the temperature correctly
> 
>  - mitigation => wake up every 'passive' delay period but not sure we can
> read the temperature correctly
> 
> With the immediate mode:
> 
>  - temperature below threshold => interrupts is not working, so we have to
> monitor the temperature and wake up every <monitor> delay
> 
>  - temperature crosses the threshold => no interrupt, detected by the
> monitoring
> 
>  - mitigation => wake up every 'passive' delay period, temperature is
> accurate
> 
> It seems not logical to have the immediate mode not working with the
> interrupts when crossing the thresholds. I would say we should stick to the
> immediate mode and double check if the interrupt can work with this mode.

But that is the one thing I'm really really confident about: It's not possible
to have threshold interrupts working in immediate mode. As soon as immediate
mode is enabled for one of the sensors by writing to the MSRCTL1 register, the
threshold interrupts no longer trigger.

I think we'll need to pick between the two non-ideal options we have:
* Immediate mode without working interrupts but reliable reads
* Filtered mode with working interrupts but unreliable reads

Keeping in mind that we can make filtered mode have mostly reliable reads,
with the cost of taking longer to read the measurements. And while I understand
that the governor is time-sensitive like you said, right now I think the numbers
still make it worth it: delaying 400us to get good reads on interrupts in
filtered mode vs waiting for the 1000ms poll in immediate mode.

(Another undiscussed consideration is that immediate mode might return glitch
reads, while filtered mode, by performing multiple samples, averaging, etc,
might actually return measurements that are closer to reality, though of course
if it can't be read, it's no good. And I haven't compared results enough to say
how much the filtering helps).

Thanks,
Nícolas
  

On Thu, Aug 17, 2023 at 4:49 AM Daniel Lezcano
<daniel.lezcano@linaro.org> wrote:
>
>
> Hi,
>
> On 16/08/2023 21:57, Nícolas F. R. A. Prado wrote:
>
> [ ... ]
>
> > Hi Daniel,
> >
> > just a gentle reminder. As you've just applied [1], there are no longer any
> > concerns with this series, and it'll provide both working interrupts and
> > reliable thermal readings on MT8192.
>
> There are still concerns and questions in the series for patch2 and 3.

FWIW the readout errors raised in patch 3 were fixed by

"thermal/drivers/mediatek/lvts_thermal: Make readings valid in filtered mode"

So I guess the remaining concern is on patch 2 about whether the noirq
suspend callback should be used instead.

ChenYu