Quantcast
Channel: Windows Desktop Development for Accessibility and Automation forum
Viewing all articles
Browse latest Browse all 585

UIAutomation method ElementFromPoint() retrieves an incorrect element from Notepad on Windows 10

$
0
0

We are developing a C# Windows Forms App and use UI Automation to record user activity. Our app declares its DPI-awareness as Per-monitor, so the system doesn't lie to it with coordinate virtualization.

On Windows 10 we ran into an issue with Notepad: after a screen scaling performed, an incorrect element is returned when calling UIA methodElementFromPoint() with correct physical coordinates passed. Also, coordinates of its BoundingRectangle returned by aCurrentBoundingRectangle() call are also incorrect: divided by the current screen scale value, i.e. 1.5 for 150%.

It looks like a bug in Per-monitor implementation of Notepad for Windows 10.
Has anyone encountered this problem before and how did you solve it?


Background

Not all UI elements of the Notepad window are affected: only System button and Main menu items. Other elements, like main text area, scroller, window title, dialog buttons, submenu items, are handled properly.

Consider the following test code:

private CUIAutomation automation = new CUIAutomation();
public async Task GetElement(int x, int y)
{
    try
    {
        Debug.WriteLine($"MouseDown received: X={x} Y={y}");
        await Task.Run(() =>
        {
            // Retrieving an UIA element lying on physical coordinates
            tagPOINT point = new tagPOINT { x = x, y = y };
            IUIAutomationElement clickedElement = automation.ElementFromPoint(point);
            var elementName = clickedElement.GetCurrentPropertyValue(30005);
            var elementRect = clickedElement.CurrentBoundingRectangle;

            // Actually retrieved UIA element
            Debug.WriteLine($"UIA element: Name={elementName} " +
                $"Rect=[left={elementRect.left} top={elementRect.top} right={elementRect.right} bot={elementRect.bottom}]");
        });
    }
    catch (Exception ex)
    {
        Debug.WriteLine(ex);
    }
}

On Win 10 this code returns an incorrect element and BoundingRectangle for the "File" main menu item:

MouseDown received: X=735 Y=391
UIA element: Name=Application
Rect=[left=475 top=249 right=822 bot=268]


Just incorrect BoundingRectangle for the System button:

MouseDown received: X=701 Y=282
UIA element: Name=System
Rect=[left=453 top=183 right=475 bot=205]

And correct element and BoundingRectangle for other UI controls (i.e. File -> Save submenu item):

MouseDown received: X=1386 Y=666
UIA element: Name=Save
Rect=[left=1320 top=652 right=1452 bot=691]

These results are not reproducing on older versions of Notepad that declares itself as System-DPI-aware.

For instance, on Windows 7 always correct elements and bounding <g class="gr_ gr_37 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling" data-gr-id="37" id="37">rects</g> are retrieved.

Also, I've tested other apps on Win 10 that implement Per-monitor DPI-awareness mode: Acrobat Reader DC, Edge, Skype, Slack, Explorer. Main menus of these apps are handled properly too: correct elements and bounding <g class="gr_ gr_51 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling" data-gr-id="51" id="51">rects</g> are retrieved.





Viewing all articles
Browse latest Browse all 585

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>