PythonNET custom module successfully created

In my quest to find a good solution for clicking images (besides AutoHotkey, which few will probably use but works great), I discovered pythonnet and a great package for including it in a C# application, Python.Included. I have been able to create a test module for pythonnet using this links, with a few fits and starts. Python.Included calls for .NET Standard, but including it via NuGet resolved all dependencies.

Some things you’ll want to do to make this work:

  • Set PYTHONHOME (for me, %userprofile%\Appdata\Local\Programs\python\Python37
  • Set PYTHONPATH (for me, %userprofile%\Appdata\Local\Programs\python\Python37\Lib\site-packages)

Once the modules is compiled, your bin\Release directory will be jam packed. You don’t need to include everything in there. Copy at least the following files to your Custom Modules directory:

  • Modules.PythonNet.dll;
  • Python.Included.dll;
  • Python.Runtime.NETStandard.dll;
  • System.Value.Tuple.dll (may not be needed more than once or evan at all; I’ve since removed it);

If you don’t do this, you won’t be able to import Python modules without some workarounds (which I didn’t care to try).

For my Modules.PythonNet, I decided to try some simple string manipulation; the ‘numpy’ package, which is incredibly useful; and for images, the ‘pyautogui’ package.

Given the issues I’ve been having with screenshots, I will provide code listings instead. Please be aware that this is a work in heavy development for me, and I have not cleaned up the code nor removed superfluous "using"s. The resources file still needs to be properly constructed.

Some things to remember (see the code):

Always initialize the PythonEngine, and put your Python code in a ‘using (Py.GIL())’ block. See the documents on the pythonnet github page (you’ll need them).

public override void Execute(ActionContext context)
{
    try
    {
        PythonEngine.Initialize();

        using (Py.GIL())
        {
            dynamic pag = Py.Import("pyautogui");
            var coords = pag.locateCenterOnScreen(ImageFullPath);
            pag.doubleClick(coords);
            PyOutX = coords[0];
            PyOutY = coords[1];
        }
    }

Testing.cs (simple string manipulation):

using System;
using Robin.Core;
using Robin.Core.Attributes;
using Python.Included;
using Python.Runtime;
using System.Collections.Generic;
using System.CodeDom.Compiler;

namespace Modules.PythonNet
{
    [Action(Order = 1)]
    [Throws("ActionError")] // TODO: change error name (or delete if not needed)
    public class Testing : ActionBase
    {
        #region Properties

        // NOTE: You can find sample description and friendly name entries in Resources

        [InputArgument]
        public string StringIn { get; set; }

        [OutputArgument]
        public string StringOut { get; set; }

        #endregion

        #region Methods Overrides

        public override void Execute(ActionContext context)
        {
            try
            {
                //TODO: add action execution code here
                PythonEngine.Initialize();
                // PythonEngine.ImportModule("numpy");

                using (Py.GIL())
                {
                    var tempString = StringIn;
                    StringOut = StringIn + " is a bad monkey.";
                }
            }
            catch (Exception e)
            {
                if (e is ActionException) throw;

                throw new ActionException("ActionError", e.Message, e.InnerException);
            }

            // TODO: set values to Output Arguments here
        }

        #endregion
    }
}

Testing2.cs (using numpy):

using System;
using Python.Runtime;
using Robin.Core;
using Robin.Core.Attributes;

namespace Modules.PythonNet
{
    [Action(Order = 1)]
    [Throws("ActionError")] // TODO: change error name (or delete if not needed)
    public class Testing2 : ActionBase
    {
        #region Properties

        // NOTE: You can find sample description and friendly name entries in Resources

        /*
        public bool Result { get; private set; }

        [InputArgument]
        public string InputArgument1 { get; set; }
        */
        [OutputArgument]
        public string PyOut { get; set; }


        #endregion

        #region Methods Overrides

        public override void Execute(ActionContext context)
        {
            try
            {
                PythonEngine.Initialize();

                using (Py.GIL())
                {
                    dynamic np = Py.Import("numpy");
                    PyOut = np.cos(np.pi * 2).ToString();
                }
            }
            catch (Exception e)
            {
                throw new ActionException("ActionError", e.Message, e.InnerException); // TODO: change error name (or delete if not needed)
            }

            // TODO: set values to Output Arguments and Result here
            // Result = ...
            // OutputArgument1 = ...
        }

        #endregion
    }
}

Testing3.cs (locate an image on screen, double-click the center of the image, and return the coordinates of the found image - no appmask!):

using System;
using Python.Runtime;
using Robin.Core;
using Robin.Core.Attributes;
using System.Drawing;

namespace Modules.PythonNet
{
    [Action(Order = 1)]
    [Throws("ActionError")] // TODO: change error name (or delete if not needed)
    public class Testing3 : ActionBase
    {
        #region Properties

        // NOTE: You can find sample description and friendly name entries in Resources

        // public bool Result { get; private set; }

        [InputArgument]
        public string ImageFullPath { get; set; }

        [OutputArgument]
        public Int64 PyOutX { get; set; }

        [OutputArgument]
        public Int64 PyOutY { get; set; }


        #endregion

        #region Methods Overrides

        public override void Execute(ActionContext context)
        {
            try
            {
                PythonEngine.Initialize();

                using (Py.GIL())
                {
                    dynamic pag = Py.Import("pyautogui");
                    var coords = pag.locateCenterOnScreen(ImageFullPath);
                    pag.doubleClick(coords);
                    PyOutX = coords[0];
                    PyOutY = coords[1];
                }
            }
            catch (Exception e)
            {
                throw new ActionException("ActionError", e.Message, e.InnerException); // TODO: change error name (or delete if not needed)
            }

            // TODO: set values to Output Arguments and Result here
            // Result = ...
            // OutputArgument1 = ...
        }

        #endregion
    }
}

A small screenshot of my references (some not needed) of 7.8kb could not be shared, sorry! I understand the issue is being worked on.

A Robin test script of the three classes available so far. Ouput is:

  • A string to which more text is appended by Python;

  • A numpy evaluation of cos(pi * 2);

  • And finally, the script locates the PNG image of a program on my desktop called KillBill.exe, and double clicks it, then returning the coordinates of the image found.

    PythonNet.Testing StringIn: “Cheetah” StringOut=> StringOut

    Console.Write Message: StringOut

    PythonNet.Testing2 PyOut=> PyOut

    Console.Write Message: PyOut

    PythonNet.Testing3 ImageFullPath: “C:\Images\killbill.png” PyOutX=> PyOutX PyOutY=> PyOutY

    Console.Write Message: "X: " + PyOutX + " Y: " + PyOutY

Its output (no images available, sorry!)

Checking script...
Loading robot...
Running script...
Cheetah is a bad monkey.
1.0
X: 1190 Y: 578
Execution completed successfully.
4 Likes

Changed name to PyNet.
Amazon sign in animated gif courtest of imgur:
Imgur

Screenshot of script:
Imgur

PyNet is cloned from PythonNet. Only the names have been changed to protect the guilty. :grinning:

For image clicking based on Pyautogui, especially for the web, and especially for sites like Amazon that actively defeat automation, I’ve found it useful to run an interactive Python session in the console and make sure the image I’m using (grabbed with the Windows stock snipping tool for now) is found by the script. If so, then I can be fairly sure it will be found in a Robin script as well.

Code for ImageClick class:

using System;
using Python.Runtime;
using Robin.Core;
using Robin.Core.Attributes;
using System.Drawing;

namespace Modules.PyNet
{
    [Action(Order = 1)]
    [Throws("ActionError")] // TODO: change error name (or delete if not needed)
    public class ImageClick : ActionBase
    {
        #region Properties

        // NOTE: You can find sample description and friendly name entries in Resources

        // public bool Result { get; private set; }

        [InputArgument]
        public string ImageFullPath { get; set; }

        [InputArgument]
        public string ClickType { get; set; }

        [OutputArgument]
        public Int64 PyOutX { get; set; }

        [OutputArgument]
        public Int64 PyOutY { get; set; }


        #endregion

        #region Methods Overrides

        public override void Execute(ActionContext context)
        {
            try
            {
                PythonEngine.Initialize();

                using (Py.GIL())
                {
                    dynamic pag = Py.Import("pyautogui");
                    var coords = pag.locateCenterOnScreen(ImageFullPath);
                    if (ClickType == "DoubleClick")
                    {
                        pag.doubleClick(coords);
                    }
                    if (ClickType == "RightClick")
                    {
                        pag.rightClick(coords);
                    }
                    if (ClickType == "MiddleClick")
                    {
                        pag.middleClick(coords);
                    }
                    if (ClickType == "TripleClick")
                    {
                        pag.tripleClick(coords);
                    }
                    if (ClickType == "LeftClick")
                    {
                        pag.click(coords[0], coords[1]);
                    }
                    PyOutX = coords[0];
                    PyOutY = coords[1];
                }
            }
            catch (Exception e)
            {
                throw new ActionException("ActionError", e.Message, e.InnerException); // TODO: change error name (or delete if not needed)
            }

            // TODO: set values to Output Arguments and Result here
            // Result = ...
            // OutputArgument1 = ...
        }

        #endregion
    }
}

Added to “DoubleClick” were “LeftClick”, “RightClick”, “MiddleClick”, and “TripleClick”. There’s no error checking yet so you have to enter the click type as shown in the screenshot above.

Regards,
burque505

2 Likes

@burque505, this was an excellent and very interesting guide.
Looking forward to the next great post!
Keep it up!

Best regards,
J.

1 Like

Thank you, @jpap, very kind of you. @Murali, @Suniel, @jokoum, thanks for the encouragement!

Regards,
burque505

Still not able to edit the original post, so I’m adding some caveats here:

  • I had to remove the PYTHONPATH environment variable on my system because it trashed pip, although python itself and the Robin extension continued to work fine.
  • Also, the PYTHONHOME variable is probably wrong, but works with the Robin extension and doesn’t hurt python or pip.

Sorry if this has caused confusion (it certainly did for me)!

Regards,
burque505