Web Scraping Yahoo Finance Python

In this post, we are going to scrape Yahoo Finance using python. This is a great source for stock-market data. We will code a scraper for that. Using that scraper you would be able to scrape stock data of any company from yahoo finance. As you know I like to make things pretty simple, for that, I will also be using a web scraper which will increase your scraping efficiency. Yahoo Financeから株価情報を取得し、グラフの描画とデータの保存を行う。 Googleで検索条件を入力し、複数のページから検索結果を取得し保存する。インスタグラムにログインし、画面スクロールしながら全ての画像をダウンロードする。. What is yahoofin? Yahoofin is a Python 3 package designed to scrape historical stock price data, as well as to provide current information on market caps, dividend yields, and which stocks comprise the major exchanges. Download JSON file from Yahoo Finance using Python's requests library. This is probably the difficult part, you will need to get cookies before actually downloading the CSV file, more info here,here and here. Once you create the proper URL with the cookie, you can download it with.

Web Scraping Yahoo Finance Python Download

What is yahoo_fin?

Yahoo_fin is a Python 3 package designed to scrape historical stock price data, as well as to provide current information on market caps, dividend yields, and which stocks comprise the major exchanges. Additional functionality includes scraping income statements, balance sheets, cash flows, holder information, and analyst data. The package includes the ability to scrape live (real-time) stock prices, capture cryptocurrency data, and get the most actively traded stocks on a current trading day. Yahoo_fin also contains a module for retrieving option prices and expiration dates.

The latest version of yahoo_fin can also scrape earnings calendar history and has an additional module for scraping financial news RSS feeds.

If you like yahoo_fin and / or this blog, consider making a contribution here to support the project.

For navigating yahoo_fin’s documentation, click on any of the links below.

To see in-depth examples, check out the my video series on YouTube or the following posts:

Two intro videos in the series are below.

Installation & Getting historical / real-time stock prices

Easily scraping ticker lists

Updates

Update: March 2021

yahoo_fin 0.8.8 was released in March 2021. This release contains a patch for the tickers_dow method.

Update: Feb. 2021

yahoo_fin 0.8.7 was released in Feb. 2021. This version adds a collection of new features.

Added several functions to extract historical earnings calendar data. These functions allow you to get the dates of previous earnings releases along with the expected and actual EPS. You can also get the next earnings date for a stock ticker or search for stocks that have earnings in particular date range or individual date. See get_earnings_history, get_earnings_for_date, and get_earnings_in_date_range in the stock_info module.

A new (experimental) module was added to scrape financial news data. Currently, this module can download the Yahoo Finance RSS news feeds for an input ticker.

Added functions to get currencies and futures data (get_currencies and get_futures, respectively)

Added a function, get_quote_data, which returns a collection of useful data on a stock ticker. It returns a dictionary containing over 70 elements, including current real-time price, company name, book value, P/E, current market state (Open / Closed), pre-market price (if applicable), post-market price (if applicable), and more.

You can also now get the current pre-market and post-market (after hours) prices directly (when available). See get_premarket_price and get_postmarket_price in the stock_info module.

New ticker lists can be pulled for several other exchanges: (NIFTY 50, NIFTYBANK, FTSE 100, FTSE 250, and Ibovespa)

A couple issues around specific tickers were fixed. For example, get_quote_table would throw an error for some tickers due to the different structure for those tickers (e.g. “CL=F”). This issue has been resolved.

Update: July 11, 2020

Version 0.8.6 of yahoo_fin made the following changes:

By popular demand, quarterly data (in addition to previously available yearly data) can now be downloaded for balance sheets, cash flow statements, and income statements!

Earnings information can be pulled with the get_earnings function.

A bug affecting ticker extraction was fixed (thanks to Alvaro Ritter!)

The code extracting fundamentals data was rewritten to have greater stability.

The get_financials function was added, which allows you to more efficiently extract balance sheets, cash flows, and income statements for the same ticker all at once

The get_dividends and get_splits functions were added to easily download historical dividends and splits information. Thank you to Daniel Catlin for this contribution!

Update: April 24, 2020

This update to yahoo_fin occurred on April 24, 2020 (version 0.8.5). This version updated the get_stats function, as well as added the get_stats_valuation function. Follow the guidance in the installation section below to upgrade yahoo_fin to the latest version.

Update: December 15, 2019

An update to this package was pushed on December 15, 2019. This update fixes the issues caused by a recent change in Yahoo Finance’s website. If you have a previously installed version of yahoo_fin, please follow the guidance below to upgrade your installation using pip.

Recommended Python Version

A few methods in yahoo_fin require a package called requests_html as a dependency. Since requests_html requires Python 3.6+, you’ll need Python 3.6+ when installing yahoo_fin.

yahoo_fin Installation

Yahoo_fin can be installed using pip:

If you have a previously installed version, you can upgrade like this:

Requirements

Yahoo_fin requires the following packages to be installed:

With the exception of requests_html, these dependencies come pre-installed with Anaconda. requests_html requires Python 3.6+ and is needed for several of the functions in yahoo_fin, as described above. To install requests_html, you can use pip:

Methods

The yahoo_fin package has three modules. These are called stock_info, options, and news. stock_info has the below primary methods.

The methods for options are listed below:

The news module currently contains one method:

stock_info module

Any method from yahoo_fin’s stock_info module can be imported by running the follow line, with get_analysts_info replaced with the method of choice.

Alternatively, all methods can be imported at once like so:

get_analysts_info(ticker)

Scrapes data from the Analysts page for the input ticker from Yahoo Finance (e.g. https://finance.yahoo.com/quote/NFLX/analysts?p=NFLX. This includes information on earnings estimates, EPS trends / revisions etc.

Returns a dictionary containing the tables visible on the ‘Analysts’ page.

Possible parameters

get_balance_sheet(ticker, yearly = True)

Scrapes the balance sheet for the input ticker from Yahoo Finance (e.g. https://finance.yahoo.com/quote/NFLX/balance-sheet?p=NFLX.

Possible parameters

get_cash_flow(ticker, yearly = True)

Scrapes the cash flow statement for the input ticker from Yahoo Finance (e.g. https://finance.yahoo.com/quote/NFLX/cash-flow?p=NFLX.

Possible parameters

get_currencies()

Scrapes the currencies table Yahoo Finance: https://finance.yahoo.com/currencies

get_data(ticker, start_date = None, end_date = None, index_as_date = True, interval = “1d”)

Downloads historical price data of a stock into a pandas data frame. Offers the functionality to pull daily, weekly, or monthly data.

Possible parameters

If you want to filter by a date range, you can just add a value for the start_date and / or end_date parameters, like below:

Get weekly or monthly historical price data:

get_day_gainers()

Scrapes the top 100 (at most) stocks with the largest gains (on the given trading day) from Yahoo Finance (see https://finance.yahoo.com/gainers).

get_day_losers()

Scrapes the top 100 (at most) worst performing stocks (on the given trading day) from Yahoo Finance (see https://finance.yahoo.com/losers).

get_day_most_active()

Scrapes the top 100 most active stocks (on the given trading day) from Yahoo Finance (see https://finance.yahoo.com/most-active).

get_dividends(ticker, start_date = None, end_date = None, index_as_date = True)

Downloads historical dividend data of a stock into a pandas data frame.

Possible parameters

get_earnings(ticker)

Scrapes earnings information from Yahoo Finance’s financials page for a given ticker (see https://finance.yahoo.com/quote/NFLX/financials?p=NFLX). Returns a dictionary with quarterly actual vs. estimated earnings per share, quarterly revenue / earnings data, and yearly revenue / earnings data.

Possible parameters

get_earnings_for_date(ticker)

Returns a list of dictionaries. Each dictionary contains a ticker, its corresponding EPS estimate, and the time of the earnings release.
Possible parameters

get_earnings_history(ticker)

Scrapes earnings history information from Yahoo Finance’s financials page for a given ticker. Returns a list of dictionaries with quarterly actual vs. estimated earnings per share along with dates of previous earnings releases. Currently, this method can pull back data for over 20 years.

Possible parameters

get_earnings_in_date_range(ticker)

Returns a list of dictionaries. Each dictionary contains a ticker, its corresponding EPS estimate, and the time of the earnings release. The data is returned based upon what earnings occur in the input date range. The date range is inclusive of the start_date and end_date inputs.
Possible parameters

get_financials(ticker, yearly = True, quarterly = True)

Efficient method to scrape balance sheets, cash flow statements, and income statements in a single call from Yahoo Finance’s financials page for a given ticker (see https://finance.yahoo.com/quote/NFLX/financials?p=NFLX).

If you’re looking to get all of this information for a given ticker, or set of tickers, this function will be 3x faster than running get_balance_sheet, get_cash_flow, and get_income_statement separately. Yearly, quarterly, or both time-periods can be pulled.

Returns a dictionary with the following keys:

If yearly = True:

yearly_income_statement

yearly_balance_sheet

yearly_cash_flow

If quarterly = True:

quarterly_income_statement

quarterly_balance_sheet

quarterly_cash_flow

If yearly and quarterly are both set to be True, all six key-value pairs are returned.

Possible parameters

get_futures()

Returns the table of futures prices from Yahoo Finance here: https://finance.yahoo.com/commodities

get_holders(ticker)

Scrapes data from the Holders tab from Yahoo Finance (e.g. https://finance.yahoo.com/quote/NFLX/holders?p=NFLX for an input ticker.

Possible parameters

get_income_statement(ticker, yearly = True)

Scrapes the income statement for the input ticker, which includes information on Price / Sales, P/E, and moving averages (e.g. https://finance.yahoo.com/quote/NFLX/financials?p=NFLX.

Possible parameters

get_live_price(ticker)

Scrapes the live quote price for the input ticker.

Possible parameters

get_market_status()

Returns a status specifying whether the market is currently pre-market (“PRE”), open (“OPEN”), post-market (“POST”), or closed (“CLOSED”).

get_next_earnings_date(ticker)

Returns the next upcoming earnings date for a given ticker.

Possible parameters

get_premarket_price(ticker)

Returns the premarket price for a given ticker if available / applicable.

Possible parameters

get_postmarket_price(ticker)

Returns the postmarket price for a given ticker if available / applicable.

Possible parameters

get_quote_data(ticker)

Scrapes a collection of over 70 data points for an input ticker from Yahoo Finance (e.g. https://query1.finance.yahoo.com/v7/finance/quote?symbols=NFLX), including current real-time price, company name, book value, 50-day average, 200-day average, pre-market price / post-market price (if available), shares outstanding, and more. The results are returned as a dictionary.

Possible parameters

get_quote_table(ticker , dict_result = True)

Scrapes the primary table found on the quote page of an input ticker from Yahoo Finance (e.g. https://finance.yahoo.com/quote/AAPL?p=AAPL)

Possible parameters

The following fields with their corresponding values are returned:

get_splits(ticker, start_date = None, end_date = None, index_as_date = True)

Downloads historical stock splits data of a stock into a pandas data frame.

Possible parameters

get_stats(ticker)

Scrapes data off the statistics page for the input ticker, which includes information on moving averages, return on equity, shares outstanding, etc. (e.g. https://finance.yahoo.com/quote/NFLX/key-statistics?p=NFLX.

Possible parameters

get_stats_valuation(ticker)

Scrapes the “Valuation Measures” data off the statistics page for the input ticker, which includes information on Price / Sales, P/E, and market cap (e.g. https://finance.yahoo.com/quote/NFLX/key-statistics?p=NFLX.

Possible parameters

get_top_crypto(ticker)

Scrapes data for top 100 cryptocurrencies by market cap (see https://finance.yahoo.com/cryptocurrencies).

Possible parameters

get_undervalued_large_caps

Returns the table of the top 100 undervalued large caps from Yahoo Finance here: https://finance.yahoo.com/screener/predefined/undervalued_large_caps?offset=0&count=100

tickers_dow(include_company_data = False)

If no parameters are passed, returns a list of tickers currently listed on the Dow Jones. The tickers are scraped from Wikipedia (see https://en.wikipedia.org/wiki/Dow_Jones_Industrial_Average. If you set include_company_data = True, it will return the full table on this webpage.

tickers_ftse100(include_company_data = False)

If no parameters are passed, returns a list of tickers currently listed on the FTSE 100 index. Otherwise, setting include_company_data = True will return a table with ticker, sector, and company name. The tickers are scraped from here: https://en.wikipedia.org/wiki/FTSE_100_Index.

tickers_ftse250(include_company_data = False)

If no parameters are passed, returns a list of tickers currently listed on the FTSE 250 index. Otherwise, setting include_company_data = True will return a table with ticker and company name. The tickers are scraped from here: https://en.wikipedia.org/wiki/FTSE_250_Index.

tickers_nasdaq(include_company_data = False)

Returns a list of tickers currently listed on the NASDAQ. If you specify include_company_data = True, it will return a table containing the tickers, their corresponding company names, and several other attributes. This method, along with tickers_other, works by scraping text files from ftp://ftp.nasdaqtrader.com/SymbolDirectory/.

tickers_nasdaq scrapes the nasdaqlisted.txt file from the link above, while tickers_other scrapes the otherlisted.txt file.

tickers_nifty50(include_company_data = False)

Returns a list of tickers currently listed on the NIFTY50. This method scrapes the tickers from here: https://en.wikipedia.org/wiki/NIFTY_50. If include_company_data is set to True, a table containing the tickers and company names is returned.

tickers_niftybank()

Returns a list of tickers currently listed on the NIFTYBANK. No parameters need to be passed.

tickers_other(include_company_data = False)

See above description for tickers_nasdaq.

tickers_sp500(include_company_data = False)

Returns a list of tickers currently listed in the S&P 500. The data for this is scraped from Wikipedia:

If include_company_data is set to True, the tickers, company names, and sector information is returned as a data frame.

options module

We can import any method from options module like this:

Just replace get_options_chain with any other method. Also, we can import all methods at once like so:

get_calls(ticker, date = None)

Scrapes call options data for the input ticker from Yahoo Finance (e.g. https://finance.yahoo.com/quote/NFLX/options?p=NFLX.

Returns a pandas data frame containing the call options data for the given ticker and expiration date.

Possible parameters

get_expiration_dates(ticker)

Scrapes expiration dates for the input ticker from Yahoo Finance (e.g. https://finance.yahoo.com/quote/NFLX/options?p=NFLX.

Returns a list of expiration dates for the input ticker. This list is based off the drop-down selection box on the options data webpage for the input ticker.

Possible parameters

get_options_chain(ticker, date)

Scrapes calls and puts tables for the input ticker from Yahoo Finance (e.g. https://finance.yahoo.com/quote/NFLX/options?p=NFLX.

Returns a dictionary with two data frames. The keys of the dictionary are labeled calls (which maps to the calls data table) and puts (which maps to the puts data table).

Possible parameters

get_puts(ticker, date = None)

Scrapes put options data for the input ticker from Yahoo Finance (e.g. https://finance.yahoo.com/quote/NFLX/options?p=NFLX.

Returns a pandas data frame containing the put options data for the given ticker and expiration date.

Possible parameters

yahoo_fin news module

Currently the news module contains a single function, get_yf_rss, which retrieves the Yahoo Finance news RSS feeds for an input ticker.

To learn more about Python and / or open source coding, check out a new online Python course I co-created with 365 Data Science! You’ll learn all about web scraping, how to use APIs in Python, how to scrape JavaScript pages, and how to deal with other modern challenges like logging into websites! Check it out on Udemy here!