UiBot : Robots for Everyone!

Your Location: Training Center> Documents > 19.Direct UI-Based Software
19.Direct UI-Based Software

Previously, the Windows software interface development framework had been provided by Microsoft, including MFC, WTL, WinForm, WPF, etc. Microsoft was kind enough to provide an automated interface to the interface of these frameworks. In recent years, to make the software interface more attractive and easier to use, many vendors and development teams have come up with their own Windows software interface development framework. These frameworks are collectively referred to as DirectUI. The interface elements of these frameworks are all drawn, and although the human eye can see them, neither the operating system nor any other program knows where the interface elements are. Some DirectUI frameworks provide external interfaces to determine whether or not an interface element exists. Other programs, including UiBot, cannot 46 naturally find these interface elements.


As a matter of fact, the UiBot Creator and UiBot Worker's own interface was developed using a DirectUI framework called [electron] (https://electronjs.org/). Electron provides a lookup interface for the interface elements, but by default, all releases are closed. So a careful reader may find that the interface elements in UiBot are not found on any RPA platform in the market, including UiBot itself.


There is common DirectUI-based software, WeChat's Windows client. Since Tencent did not reveal the WeChat Windows client, we were unable to find out what kind of DirectUI framework WeChat uses. However, it turns out that this framework does not provide an interface for interface elements and is therefore not available on any of the RPA platforms in the market. Additionally, WeChat provides a browser-based version of the web, which can retrieve interface elements and is recommended for use in the RPA process.


Because the game's interface emphasizes beauty and individuality, the general game interface elements are "drawn" out in principle similar to DirectUI. This interface also does not typically provide an interface that 47 informs us of the location for different interface elements. In contrast to the DirectUI-based software, the game interface changes quickly and requires more timeliness. Generally speaking, the RPA platform is not optimized for the game, so it won't work well on it.


If you want to use auto-operation on the game, it is recommended to use the keystrokes. Keystrokes are specially designed for the game with builtin, game-specific interface-lookup such as single point color alignment, multi-point color alignment, image lookup, and so on.


In [the previous chapter] [the interface element] we introduced used the command with the target, whereas UiBot used the command without the target. As shown in the figure below, the red box represents the command with the target, and the blue box represents the command without the target.


! [Objective and Goalless Command] (imgsTarget/1.png){width="40%"}


If you encounter Windows software without a target, the Target Command cannot be used, but you can still use the “No Target” command. Of these commands without a target in the diagram, the most important is "Simulated Movement". Since Simulated Movement requires that we specify a coordinate point within the command, the mouse pointer will 48 move to the same point when the command is executed. After moving, you can use the "Simulated Click" command to simulate the left-click button to press a button properly. You also use it to set the focus correctly on an input box and then use the “Enter Text” command to enter a paragraph of text into the input box where the focus resides. For example, there is an input box in which the coordinates are x: 200 and y: 300. Then we need to simulate the move and set the coordinates to x: 200, y: 300. Next, we must press the left button with "Simulated Click" to set the focus. Lastly, we reenter the text to type normally. Otherwise, if you use "input text" directly, there is a high probability that it is entered into the other input box.


Here, we need to explain the screen coordinates of the Windows operating system. If you already know the screen coordinates, this section can be skipped.


[press skip here] (allow it to jump to the next appropriate section so the user doesn’t need to search for themselves)


In the Windows operating system, each point on the screen has a unique coordinate which is made up of two integers: one called x and the other called y. For example, the coordinates x:200, y:300 means that the x value of this point is 200 and the y value is 300. X is calculated as 0 on the left 49 side of the screen (0, 1, 2, 3...so on from left to right) and Y is calculated from the top to bottom of the screen (0, 1, 2, 3...so on from top to bottom). Therefore, the point at coordinates x: 200, y: 300, is roughly the position of the red circle shown in the figure below:


You can determine the position of a point on the screen whenever you have two integer values, x and y. In UiBot, there are commands to get the location of a point on the screen and output it into a variable. How do you save both x and y values into one variable? As we learned in [the] [language reference] of the Webspherecript language used by UiBot, Webspherecript has a "Dictionary" data type that can hold multiple values, so when UiBot outputs the position of a point, it outputs to a variable of a specific dictionary type. Suppose this variable is called “pnt,” then the values x and y can be obtained using “pnt[ "x" ]”and “pnt[ "y" ].”


If we are looking for an interface element in a fixed position on the screen with no Target commands, we can normalize the simulation operation. This is often the case because Windows is a multi-Windows system and the location of each Windows can be dragged, causing the location of the interface elements in the Windows to change. Moreover, in software such as WeChat, the location of a contact is not fixed but is sorted according to 50 the most recent contact time, and the location can change at any time.


Therefore, in UiBot, it is generally not recommended to write fixed coordinates directly because there are so many variations which make it hard to think through all of them. Usually, if you use the command without the target, you need to use it with other commands so that other commands can find the interface element coordinates by using its specific characteristics. Then you can take the coordinates found as variables and pass them on to the command without the target.


In UiBot, the best partner for a nonobject command is the image command.


Are You Ready to Explore the RPA World?
GET STARTED