Selenium Unable To Locate Element Only When Using Headless Chrome (python)

I just started learning Selenium and need to verify a login web-page using a jenkins machine in the cloud, which doesn't have a GUI. I managed to run the script successfully on my

Solution 1:

If the script is working perfectly fine without headless mode, probably there is issue with the window size. Along with specifying --no-sandbox option, try changing the window size passed to the webdriver

chrome_options.add_argument('--window-size=1920,1080')

This window size worked in my case.

Even if this dosen't work you might need to add wait timers as answered before as rendering in headless mode works in a different way as compared to a browser in UI mode.

Ref for rendering in headless mode - https://www.toolsqa.com/selenium-webdriver/selenium-headless-browser-testing/

Solution 2:

I had the same issue, it was initially working, though after an update from a website that we were using Selenium on, it stopped working in headless mode, though kept on working in non headless. After 2 days of researching the deepest and darkest depths of the web and a lot of trial and error, finally found what the issue was.

I tried all the methods outlined on the web and more, though nothing worked until I found this.

In headless chrome mode, the user agent is something like this: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/60.0.3112.50 Safari/537.36

The service provider updated their code to identify the HeadlessChrome part and would cause the tab to crash and in turn, destroying the Selenium user session.

Which lead to the issue firing above in one of the exceptions.

To solve this, I used a plugin called fake_headers (https://github.com/diwu1989/Fake-Headers):

from fake_headers import Headers

header = Headers(
    browser="chrome",  # Generate only Chrome UA
    os="win",  # Generate only Windows platform
    headers=False# generate misc headers
)
customUserAgent = header.generate()['User-Agent']

options.add_argument(f"user-agent={customUserAgent}")

Though this was only half the solution as I only wanted Windows and Chrome headers and the fake_headers module has not not include the latest Chrome browsers and has a lot of older versions of Chrome in the list as can be seen in this file https://github.com/diwu1989/Fake-Headers/blob/master/fake_headers/browsers.py. The particular site I was running Selenium had certain features that only worked on newer versions of Chrome, so when an older version of Chrome was passed through the user-agent header, the certain features on the site would actually stop working. So I needed to update the browsers.py file in the fake_headers module to include only the versions of Chrome that I wanted to include. So I deleted all the older versions of Chrome and created a select list of versions (each one individually tested to work on the site in question and deleted the ones that did not). Which ended up with the following list, which I could expand on though have not for the time being.

chrome_ver = [
    '90.0.4430', '84.0.4147', '85.0.4183', '85.0.4183', '87.0.4280', '86.0.4240', '88.0.4324', '89.0.4389', '92.0.4515', '91.0.4472', '93.0.4577', '93.0.4577'
]

Hope this helps spare someone two days of stress and messing around.

Some more useful info on headless chrome detectability: https://intoli.com/blog/making-chrome-headless-undetectable/

Solution 3:

I would refactor code in a way to wait until elements will be present on a web page:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import Byfrom selenium.webdriver.support.ui import WebDriverWait

WebDriverWait(wd, 10).until(EC.presence_of_element_located((By.ID, 'username'))).send_keys(args.usr)
WebDriverWait(wd, 10).until(EC.presence_of_element_located((By.ID,'password'))).send_keys(args.pwd)
WebDriverWait(wd, 10).until(EC.presence_of_element_located((By.ID, 'login-btn'))).click()

Generally usage of WebDriverWait in combination with some condition should be preferred to implicit waits or time.sleep(). Here is explained in details why.

Other thing to double check are whether elements have the IDs used for search and that these elements aren't located within an iframe.