Quantcast
Channel: Windows Desktop Development for Accessibility and Automation forum
Viewing all 585 articles
Browse latest View live

UI automation cannot find an element. Inspect Object can.

$
0
0

Hi,

I'm attempting to grab a particular control using UI automation.  The Inspect Object tools shows the control in the appropriate tree, and I can even navigate to it and have it be highlighted, but my code can't see it.  Here's the code (where 'root' is the parent of the control):

System.Windows.Automation.

Condition elementPropCondition = newPropertyCondition(AutomationElement.AutomationIdProperty, id, PropertyConditionFlags

.IgnoreCase);

 

AutomationElement theElement = root.FindFirst((TreeScope.Element | TreeScope.Descendants), elementPropCondition);

Any ideas?

 

Tablet keyboard problem in WPF app due to automation.

$
0
0

Redirected from wpf forum to here.

With deleted automation for window via OnCreateAutomationPeer() keyboard on tablet is auto hiding after showing in half cases.

In msdn: UI Automation is the mechanism through which developers communicate whether or not a particular UI element can receive text input. For Windows-provided controls, this will be done automatically because proper accessibility properties are set by default, but for custom controls and experiences you must do additional work to set the accessibility properties correctly; remember that the touch keyboard reacts to these properties. 

The touch keyboard determines whether a focused UI element is intended for text entry by checking to see whether it has a UIA TextPattern (or a TextChildPattern) control pattern and is editable. If a focused element fails to satisfy either of those conditions, the touch keyboard does not show itself and hides itself if it was showing.

So how delete automation (is requirement) without creating this bug?

Thanks.

How to upload a new voice that will work with Eye Control in Windows 10 text to speech option

$
0
0

Hi,

I am trying to integrate the Windows 10 Beta Eye Control featurewith a Tobii Eye Tracker 4C and ModelTalker2 voice banking software.  The eye control is working perfectly. We are able to use the eye control software with the eye tracker 4C device, as expected. But when we use the conversation mode and it speaks the text aloud, it is not using the correct voice.

 It seems we are forced to use the voices in the section:

Settings  > Time & language  > Speech

However, I can see the ModelTalker2 voice in the following accessibility sections:

Settings-> Ease of Access-> Narrator

Control Panel-> Ease of Access-> Speech Recognition-> Text-to-speech

Is there a way to upload a new voice that could work with the eye control text to speech options?

I would appreciate any advice as we are at a complete standstill with the project at this point.

Regards, 

Dave Menousek

can't get elements inside a pane

$
0
0

hi Guys,

i am trying to Automate the client application. In that i can able to get all the elements list with UI Automation. but inside a particular pane i can't able to get any elements. i am Using Access VBA even if i get solution C# also is better. i am using Unmanaged API that is UIAutomationcore.dll.Kindly help me with this. 

similar to the same pane you can see the immediate window.

[Advanced Topic] Understanding UI Automation Navigation and Performance

$
0
0
While there are several interesting threads out there on the various problem's people have had in developing automation, there aren't many good resources for understanding how to make UI Automation fast.

UI Automation should never take longer to perform any UI interaction then a human user.  If this is ever the case, it is more then likely a case of your implementation and this thread should help.

Performance with navigation is particularly painful with .Net controls over their win32 equivalents due to the inherent weight of .Net which makes doing things efficiently incredibly important.

What follows are some lessons we learned developing our internal UI Automation testing framework based on .Net 3.0 over the last year and a half.  I invite anyone with other perspectives to form a discussion in this thread and Microsoft to correct us all if we misrepresent something.

-----------------------------------------------------------

You can't build fast automation without optimizing the most fundamental aspect of UI Automation - navigation.  Navigation is the straight forward need to "find" something based on criteria and a starting point.

Microsoft provides a basic find tool in the form of FindFirst off of the base AutomationElement class.  It allows the user to specify the search criteria and the scope of the search.  Scope allows automation to determine where to look when trying to find your element and the most common ones needed are Element, Children, Descendants, and SubTree.  Element is just your current element, Children are just your immediate child elements, Descendants are any nodes which are below the current element in the UI tree, and SubTree is the current element plus all of the descendants.

The number one inefficiency I have seen people do in writing UI Automation code is the abuse of the scope field.  To understand why, you first need to consider what Microsoft is doing when you specify a scope.

Try a simple experiment on a .Net form.  Using a known element in the middle of a complex tree (lots of elements above and below), perform a search using FindFirst of an immediate child using the scopes of Children and Descendants.  The result is that the search with a scope of Children isn't just faster, it is ridiculously faster depending on how many descendants the node you are testing on has.

The reason is not that Microsoft wrote a terrible search tool, it is that Microsoft is assuming you know what you are talking about when you specify scope.  Microsoft generates a cache of the entire scope of the search before applying the criteria.   The reason for the cache is that the round trip to Win32 is expensive and the search is expensive, so by doing everything upfront they are optimizing the marginal per node cost in the search domain.  This becomes particularly important because Microsoft can't assume that the element you are looking for exists and thus has to optimize both cases in a balanced way.

So what does all that mean?  Well, use the smallest scope you possibly can.  If you need to drill down multiple levels using Microsoft FindFirst, it is faster to find each level and perform the search piecemeal.

-----------------------------------------------------------

In addition to FindFirst, Microsoft also provides a TreeWalker class for basic navigation the UI tree.  TreeWalker essentially allows you to navigate from any node to its immediate parent, child, or siblings.  By leveraging this tool, it is possible to build a vastly improved Find library, including support for Regular Expressions against AutomationIds, Names, etc.

The simple use of recursion and a copy of the Depth First Search algorithm would get you a basic Find function, but unless you consider the performance reasons why Microsoft's FindFirst wasn't fast enough, the new tool will be at least as inefficient since you don't have access to the optimizations under the hood Microsoft does when they perform their searching.

The golden rule when using TreeWalker is to minimize the number of times it is used to achieve a desired result.  Work smarter, not harder.

FindFirst is a great starting point for developing automation, but to truly write fast automation you will require something more.  The problem isn't with the way FindFirst was written but with the assumptions it makes aren't necessarily the assumption you want your Find Algorithm to make.

The alternative I suggest using is one which takes advantage of more contextual information when searching then simple scope.

1) As the author of Automation which needs to find a control, you are in the perfect position to assert whether the control you are looking for should exist or not.  In most cases, if you are looking for something you are probably doing so because you want to interact with it.  If you can assume that the common case is that your element should exist and you develop your find tool to use short circuit logic to return as soon as you find your element, then you are already theoretically twice as fast as Microsoft since you will only need to access statistically 1/2 of the nodes in the scope instead of all of them.  (In practice you'll see a performance improvement but only to a lesser degree because the cost of you searching every node using TreeWalker is not equivalent to the cost of Microsoft searching every node due to under the hood optimizations.)

2) Windows forms are collections of organization and interactive controls.  Organization controls are found in the form of Windows and Panes while interactive controls are the Edit Controls, Combo Boxes, Buttons, etc.  95% of the controls you will ever need find are on a window and the immediate child of either that window or a descendant pane.  A simple optimization then would to narrow the scope to just those elements which are the immediate descendants of either windows or panes.  This eliminates the need to uselessly walk into list item groups, grids, tree views, etc which while interesting - should be approached as a two step find for the sake of performance.  Find the control of interest, then manipulate that control or its children as appropriate.

3) When automating a page, reconsider not just how to Find a single element but how to perform the collective Find actions required on a specific window.  The common case is not that only one control is desired.  Consider the case you have a data form with 20 fields and need to set the values for all 20.

The least efficient way to approach this problem is iteratively by "finding a field, setting a field, and repeated."  The reason this approach is bad is that you are asking the Framework to repeatedly load the same controls as the find algorithm explores the tree.

A slightly better approach, but also inefficient would be to perform a FindAll of the scope (such as Descendants) to return a collection of all AutomationElements which would then be searched for the desired elements.  The benefit of the approach is the simplicity of reducing the number of loads required but it still requires us to navigate more of the tree then absolutely necessary.

The optimum approach would be to accept a list of criteria, and as you navigate the tree you set aside those objects so that once you've found all of the elements on the list you can return - eliminating the need to visit all of the nodes in the scope.  It is important that when you build your list of automation elements in the recursive structure that you don't constantly create new list objects and combine them.  By simply passing the list by reference between levels of recursion, you eliminate a great deal of extra work for .Net.

An everyday example would be to ask is it faster to go to the grocery store every time you need something, go to the store and bring the entire contents of the store home, or build a list ahead of time and just bring those items you think you'll need.  If you're wrong, you can always go back - but you'll have saved yourself all those trips you were able to consolidate.  (If you really get into the act you can accept the new values for the fields to go along with those criteria so automation can set all of the fields as it walks the tree without ever needing to even return AutomationElements to the caller.)

4) When you build your own Find function you have the luxury of determining how criteria are specified.  If you want to use the Microsoft Condition objects, simply doing myElement.FindFirst(Element, myCondition) would be sufficient for comparing if the condition matches the current element.  This is the point were you can decide that regular expressions are important when searching for an element by name of automation id and add the support to do so.  (The lack of Regex support can be really annoying for dynamic automation ids).

One approach could be to have a function which takes an Interface requiring the function IsMatch, and then any class which implements that interface would be compatible with your Find algorithm.  There are a lot of different choices.

-----------------------------------------------------------

Those are the lessons we learned when working to optimize navigation code in UI Automation.  I'm not adding code because I don't want people to skim, copy, and paste and hope magically that performance will be improved.  True performance comes from a greater understanding of how things work and how to leverage tools for the specific projects you are faced with.

I'm also curious how everyone is working to address the performance issues of UI Automation - and if there is the interest out there we can definitely create additional threads to talk about how to optimize other processes.

Deleted

Exception in oleacc.dll

$
0
0

The app does not support Active Accessibility. The WM_GETOBJECT event is passed to DefWindowProc, and the NotifyWinEvent function is not called. When start Screen Reader on Windows 7, everything works fine. But on Windows Vista and higher, an exception occurs in oleacc.dll. Call stack:

oleacc.dll!AccWrap_Base::ProcessIDispatch()

oleacc.dll!AccWrap_Base::ProcessVariant()

oleacc.dll!AccWrap_Base::get_accFocus(struct tagVARIANT *)

oleacc.dll!AccWrap_Annotate::get_accFocus(struct tagVARIANT *)

UIAutomationCore.dll!AccNavUtils::HitTestOrFocus()

UIAutomationCore.dll!MsaaProxy::GetFocus()

UIAutomationCore.dll!GetFocusedChild(struct IRawElementProviderSimple *)

UIAutomationCore.dll!HookBasedServerConnection::ClaimPendingObject()

UIAutomationCore.dll!ProcessStaticMethod()

UIAutomationCore.dll!ProcessIncomingRequest()

UIAutomationCore.dll!HookBasedServerConnectionManager::HookCallback()

UIAutomationCore.dll!HandleHookMessage()

UIAutomationCore.dll!HookUtil<&HookBasedClientConnection::HookCallback(void *,unsigned long,void * *,unsigned long *,void * *),0>::CallWndProc(int,unsigned int,long)

[…]

The problem is in the VARIANT structure. The "vt" field specifies the value VT_DISPATCH, but the structure stores a value of the VT_I4 type. Because of this, the AccWrap_Base::ProcessVariant function calls AccWrap_Base:: ProcessIDispatch, which tries to access the address 0x14.

Where to find the problem?

Microsoft mouse and keyboard center - How can I ignore specific keys (Ms Natural Ergonomic keyboard 4000)?

$
0
0
How can I set some keys, specifically the favorite keys 1-5, to completely be ignored by the software? Is that possible?

Bluetooth pairing with SSP (Secure Simple Pairing) Just Works and no MITM (Man In The Middle)

$
0
0

Hello All,
I have problem with pairing on a Bluetooth Device using SSP (Secure Simple Pairing) Just Works (NoInputNoOutput) and no MITM protection.
From a c++ program I have to discover and pair the bluetooth device.
I can discover the device and I do the pair.

But if I research the device the attribute fAuthenticated on the BLUETOOTH_DEVICE_INFO_STRUCT is FALSE.

This is my code:

BOOL CALLBACK auth_callback_ex_ssp(LPVOID pvParam, PBLUETOOTH_AUTHENTICATION_CALLBACK_PARAMS authParams)
{
	BLUETOOTH_AUTHENTICATE_RESPONSE response = { sizeof(BLUETOOTH_AUTHENTICATE_RESPONSE) };

	response.authMethod = authParams->authenticationMethod; // BLUETOOTH_AUTHENTICATION_METHOD_NUMERIC_COMPARISON
	response.bthAddressRemote = authParams->deviceInfo.Address;
	response.negativeResponse = FALSE;

	// Respond with numerical value for Just Works pairing
	response.numericCompInfo.NumericValue = 1;

	DWORD retVal = BluetoothSendAuthenticationResponseEx(0, &response);

	if (ERROR_SUCCESS != retVal)
	{
		TRACE(_T("BluetoothSendAuthenticationResponseEx() failed! %s"), MsBt7_ErrorDisp(retVal));

		return (FALSE);
	}

	return (TRUE);
}

HBLUETOOTH_AUTHENTICATION_REGISTRATION authCallbackHandleSSP = NULL;

BOOL MsBt7_ExecAuthenticateDeviceEx(PBYTE address)
{
	DWORD dwRes;
	BLUETOOTH_DEVICE_INFO_STRUCT deviceInfo = { sizeof(BLUETOOTH_DEVICE_INFO_STRUCT) };
	HBLUETOOTH_RADIO_FIND hFind;
	HANDLE hRadio;

	hFind = BluetoothFindFirstRadio(&btfrp, &hRadio);

	if (NULL == hFind)
	{
		return (FALSE);
	}

	BLUETOOTH_RADIO_INFO radioInfo = { sizeof(BLUETOOTH_RADIO_INFO) };
	if (ERROR_SUCCESS != BluetoothGetRadioInfo(hRadio, &radioInfo))
	{
		return FALSE;
	}

	CopyMemory(&deviceInfo.Address.rgBytes, address, sizeof(deviceInfo.Address.rgBytes));

	dwRes = BluetoothRegisterForAuthenticationEx(&deviceInfo, &authCallbackHandleSSP, (PFN_AUTHENTICATION_CALLBACK_EX)auth_callback_ex_ssp, hRadio);
	if (ERROR_SUCCESS != dwRes)
	{
		CloseHandle(hRadio);
		BluetoothFindRadioClose(hFind);
		return FALSE;
	}

	deviceInfo.fAuthenticated = 0;
	dwRes = BluetoothAuthenticateDeviceEx(NULL, hRadio, &deviceInfo, NULL, MITMProtectionNotRequired);

	if (ERROR_SUCCESS != dwRes)
	{
		CloseHandle(hRadio);
		BluetoothFindRadioClose(hFind);
		return (FALSE);
	}

	GUID pServiceGuid = SerialPortServiceClass_UUID;
	dwRes = BluetoothSetServiceState(hRadio, &deviceInfo, &pServiceGuid, BLUETOOTH_SERVICE_ENABLE);

	if (ERROR_SUCCESS != dwRes)
	{
		CloseHandle(hRadio);
		BluetoothFindRadioClose(hFind);
		return (FALSE);
	}

	// ... some code to get the com port

	CloseHandle(hRadio);
	BluetoothFindRadioClose(hFind);
	return TRUE;
}

The BluetoothAuthenticatedDeviceEx result is ERROR_SUCCESS and after that the fAuthenticated is TRUE.
I can also get the com port.
But if i try another search the fAuthenticated is FALSE again.

Instead if I do the pairing from the Windows user interface (add bluetooth device ...), I get fAuthenticated = TRUE.

Can someone help me?


Unable to Show Tree in Inspect

$
0
0

Hello,

I'm using Windows 8.1 and Inspect.exe to view UI control information.  However, I'm unable to see the option to show the UI tree in inspect.  I can only see the currently focused item.

Is there an issue with Inspect in Windows 8.1 that prevents this?  I've tried running Inspect in admin mode as well with no luck.


The problem of displaying list items on Windows 10 x64 for Windows Defender Firewall

$
0
0

Hello,

Inspect does not find list items from 'Allowed apps list'.

Path: Control Panel\All Control Panel Items\Windows Defender Firewall\Allowed apps

Can you help me understand why?

Searching for items in Windows 10 x86 does not find items

$
0
0

Hello,

On Windows 10 x64 for the Services window (services.msc), I can find list items from list (SysListView32).
But for Windows 10 x86 - no.

I used FindAll and TreeWalker to both cases.

How can I make it work for Windows 10 x86?


Using TreeWalker.RawViewWalker.GetLastChild in single thread leads to an crash with ntdll.dll

$
0
0

Hello,

I get crash only if use threads to search.
If I use TreeWalker.RawViewWalker.GetLastChild without the threads then all are Okay.

What can be reason?

OS: Windows 10 x64.

Eventvwr log:

Log Name:      Application
Source:        Application Error
Date:          4/28/2020 19:55:41
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MY-COMPUTER
Description:
Faulting application name: MyApplication.exe, version: 1.0.0.0, time stamp: 0x5ea70824
Faulting module name: ntdll.dll, version: 10.0.17763.1131, time stamp: 0x4dc06dfc
Exception code: 0xc0000374
Fault offset: 0x00000000000fb049
Faulting process id: 0xffc
Faulting application start time: 0x01d61dcbbd3544a0
Faulting application path: D:\Project\MyApplication.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 7fe808da-dad9-4d30-a6c7-df6e96b4fc9d
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2020-04-29T02:55:41.147045900Z" />
<EventRecordID>1677</EventRecordID>
<Channel>Application</Channel>
<Computer>MY-COMPUTER</Computer>
<Security />
  </System>
  <EventData>
<Data>MyApplication.exe</Data>
<Data>1.0.0.0</Data>
<Data>5ea70824</Data>
<Data>ntdll.dll</Data>
<Data>10.0.17763.1131</Data>
<Data>4dc06dfc</Data>
<Data>c0000374</Data>
<Data>00000000000fb049</Data>
<Data>ffc</Data>
<Data>01d61dcbbd3544a0</Data>
<Data>D:\Project\MyApplication.exe</Data>
<Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
<Data>7fe808da-dad9-4d30-a6c7-df6e96b4fc9d</Data>
<Data>
</Data>
<Data>
</Data>
  </EventData>
</Event>

Desktop icons are not in Program Manager tree after Win + D "Hide Desktop"

$
0
0

Hi,

I would like to be sure, that this behavior is "by Design" in Windows 7 and 10. 

Reproducing steps for Windows 7, 10 :

  1. Open notepad, for instance, to be sure that Win+D works.
  2. Win + D. Notepad is minimized.
  3. Get Program Manager tree via UI automation COM interface: there are no desktop icons list.
  4. Win + D. Notepad is open.
  5. Get Program Manager tree via UI automation COM interface: desktop icons list present.

UI automation: Desktop icons are not in Program Manager tree after Win + D "Hide Desktop"

$
0
0

Hi,

I would like to be sure, that this behavior is "by Design" in Windows 7 and 10. 

Reproducing steps for Windows 7, 10 :

  1. Open notepad, for instance, to be sure that Win+D works.
  2. Win + D. Notepad is minimized.
  3. Get Program Manager tree via UI automation COM interface: there are no desktop icons list.
  4. Win + D. Notepad is open.
  5. Get Program Manager tree via UI automation COM interface: desktop icons list present.



How to capture the event of an app that "requests attention"?

$
0
0

For e.g., on a windows 10 machine, if apps such as skype (running in the background) receives an IM, the windows narrator app is able detect this state is say "Skype is requesting attention". During this "request attention" phase the icon in the taskbar flashes, with an orange background, and, slowly the orange background persists until we have interacted with the app in the foreground. 

In my program I want to respond to such an event. Any pointers?

The ultimate goal is to have a powershell script operate a c# assembly (created on-the-fly) call a remote web service that can send a notification (of that event) to my android phone. 


I am a bundle of mistakes intertwined together with good intentions

Windows Accessibility API (UIAutomation) not working in Windows 10

$
0
0

Hi Team,

We have used MS UIAutomation (Windows Accessibility API) for window application automation. We are successfully able to perform actions on the screen in Windows Server 2012 R2 & Windows Server 2008.

Whereas, we are unable to do on Window 10. It able to open the application but not performing actions on the screen throwing below exception

MSAutomationDriverException: Application not activated after the timeout of 10000 [AddInfo:No AddInfo].

Help us to in understand incompatibility of dll used.

Is there an expert team on accessibility on Windows Desktop Apps?

$
0
0

Hi,   

I am looking for a dedicated forum or team on accessibility for Windows Desktop Apps.

Up till now I tried several things to improve the apps so Narrator or NVDA speak all required text field.

For insstance most controls have some Accessibility properties but the effect is not always as expected.

Tab order control is also a puzzle using Visual Studio View Tab Order, especially with a group boxes on a tab control 

Thank you

#NVDA #Screen reader #Narrator #No mouse #Blind


WM_HTML_GETOBJECT returns developers console instead of the main window (IE)

$
0
0

I'm accessing my Internet Explorer window with a valid handler (hWnd) like this:

lngMsg = RegisterWindowMessage("WM_HTML_GETOBJECT");
                if (lngMsg != 0)
                {
                    SendMessageTimeout(hWnd, lngMsg, 0, 0, SMTO_ABORTIFHUNG, 1000, out lRes);
                    if (!(bool)(lRes == 0))
                    {
                        int hr = ObjectFromLresult(lRes, ref IID_IHTMLDocument2, 0, ref document);
                    }
                }

While accessing the window as is it works as expected and I get an object of type IHTMLDocument2 with the window document. But if I open the developers console, it seems that the console overlaps the main window. 

For instance, the url of the document will be "res://C:\\WINDOWS\\SYSTEM32\\F12\\F12Script.dll/23/console/console.html", the title will match the title of the selected tab, etc. 

Is there a way to access the window itself and not the debugger?

Elements in tab control are not detected sometimes

$
0
0

Hello Everyone,

I am trying to automate a WPF app. This app has multiple modules and each module has multiple tabs that can be docked. But sometimes when I switch tabs, the elements in the tab control do not appear in the UI automation tree. This behavior is random. As you can see in the screen shots, initially I can see the elements in the 'Shipping Schedule' tab. But after I switch between a few tabs and come back to 'Shipping Schedule' again, inspect cannot detect the elements in the tab. When I float and dock the tab again, the elements reappear in the tree sometimes, thisdoesn't work always. Could anybody possibly help me with this issue?

https://social.msdn.microsoft.com/Forums/getfile/1600376

https://social.msdn.microsoft.com/Forums/getfile/1600375


Viewing all 585 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>