We are developing a C# Windows Forms App and use UI Automation to record user activity. Our app declares its DPI-awareness as Per-monitor, so the system doesn't lie to it with coordinate virtualization.
On Windows 10 we ran into an issue with Notepad: after a screen scaling performed, an incorrect element is returned when calling UIA methodElementFromPoint() with correct physical coordinates passed. Also, coordinates of its BoundingRectangle returned by aCurrentBoundingRectangle() call are also incorrect: divided by the current screen scale value, i.e. 1.5 for 150%.
It looks like a bug in Per-monitor implementation of Notepad for Windows 10.
Has anyone encountered this problem before and how did you solve it?
Background
Not all UI elements of the Notepad window are affected: only System button and Main menu items. Other elements, like main text area, scroller, window title, dialog buttons, submenu items, are handled properly.
Consider the following test code:
private CUIAutomation automation = new CUIAutomation(); public async Task GetElement(int x, int y) { try { Debug.WriteLine($"MouseDown received: X={x} Y={y}"); await Task.Run(() => { // Retrieving an UIA element lying on physical coordinates tagPOINT point = new tagPOINT { x = x, y = y }; IUIAutomationElement clickedElement = automation.ElementFromPoint(point); var elementName = clickedElement.GetCurrentPropertyValue(30005); var elementRect = clickedElement.CurrentBoundingRectangle; // Actually retrieved UIA element Debug.WriteLine($"UIA element: Name={elementName} " + $"Rect=[left={elementRect.left} top={elementRect.top} right={elementRect.right} bot={elementRect.bottom}]"); }); } catch (Exception ex) { Debug.WriteLine(ex); } }
On Win 10 this code returns an incorrect element and BoundingRectangle for the "File" main menu item:
MouseDown received: X=735 Y=391 UIA element: Name=Application Rect=[left=475 top=249 right=822 bot=268]
Just incorrect BoundingRectangle for the System button:
MouseDown received: X=701 Y=282 UIA element: Name=System Rect=[left=453 top=183 right=475 bot=205]
And correct element and BoundingRectangle for other UI controls (i.e. File -> Save submenu item):
MouseDown received: X=1386 Y=666 UIA element: Name=Save Rect=[left=1320 top=652 right=1452 bot=691]
These results are not reproducing on older versions of Notepad that declares itself as System-DPI-aware.
For instance, on Windows 7 always correct elements and bounding <g class="gr_ gr_37 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling" data-gr-id="37" id="37">rects</g> are retrieved.
Also, I've tested other apps on Win 10 that implement Per-monitor DPI-awareness mode: Acrobat Reader DC, Edge, Skype, Slack, Explorer. Main menus of these apps are handled properly too: correct elements and bounding <g class="gr_ gr_51 gr-alert gr_spell
gr_inline_cards gr_run_anim ContextualSpelling" data-gr-id="51" id="51">rects</g> are retrieved.