Managing Browser Cookies with Selenium: Get and Add Methods
When utilizing Selenium for web scraping, maintaining login sessions through cookie management becomes essential. Selenium provides built-in methods to retrieve and apply cookies, streamlining the automation process. The get_cookies() function captures all cookies as a list of dictionaries, where each dictionary represents an individual cookie.
python
captured_cookies = browser.get_cookies()
serialized_cookies = .dumps(captured_cookies)
with open('session_cookies.txt', 'w') as file:
file.write(serialized_cookies)
Note that serialized_cookies represents a complete cookie list in JSON format.
python
{'name': 'SESSION_ID', 'value': 'jKlPmNoFgHqRsTuVwXyZ1234567890', 'path': '/', 'domain': 'example.com', 'secure': False, 'httpOnly': True}
{'name': 'USER_PREFERENCES', 'value': 'theme=dark&lang=en', 'path': '/', 'domain': 'example.com', 'secure': False, 'httpOnly': False}
{'name': 'TRACKING_TOKEN', 'value': 'aBcDeFgHiJkLmNoPqRsTuVwXyZ987654321', 'path': '/', 'domain': 'example.com', 'secure': True, 'httpOnly': False}
To utilize these captured cookies, the add_cookie() method enables injecting them into the browser session.
python
with open('session_cookies.txt', 'r', encoding='utf8') as file:
cookie_list = .loads(file.read())
for cookie_data in cookie_list:
browser.add_cookie(cookie_data)
browser.get('https://example.com/dashboard')
browser.refresh()
This approach facilitates session persistence without repeated authentication. Several scenarios benefit from this technique: applications with unreliable CAPTCHA solving, environments with monitoring systems that detect frequent login attempts, or scheduled crawlers requiring frequent updates at short intervals (e.g., every 2-5 minutes). A practical implementation involves a dedicated script that authenticates and updates cookies periodically (e.g., every 30 minutes), while other scripts simply load these pre-existing cookies for their operations.