Discussion:
HTML/Javascript question
(too old to reply)
root
2018-05-25 19:42:58 UTC
Permalink
I posted this question to alt.os.linux group with no response.

I regularly fetch financial and economic data from various web
sites. Increasingly these sites require javascript which cripples
automatic data gathering. I can look at the html source of
the page and see the <script> code and frequently these scripts
are pulled down from another web site and are many screenfulls
of code.

Is there an engine I can use to fetch a web page using wget
or curl and pipe that page to the engine to process the
javascript? I don't care about any script whose purpose
it formatting the page display, I can format the output
myself.

Google recently shut down access to financial reports
for US companies which prompts my request.

Thanks.
Rich
2018-05-25 20:49:47 UTC
Permalink
Post by root
I posted this question to alt.os.linux group with no response.
I regularly fetch financial and economic data from various web
sites. Increasingly these sites require javascript which cripples
automatic data gathering. I can look at the html source of
the page and see the <script> code and frequently these scripts
are pulled down from another web site and are many screenfulls
of code.
Is there an engine I can use to fetch a web page using wget
or curl and pipe that page to the engine to process the
javascript? I don't care about any script whose purpose
it formatting the page display, I can format the output
myself.
Google recently shut down access to financial reports
for US companies which prompts my request.
Don't bother.

Sign up for a free Alpha Vantage sign-on, and download in either CSV or
JSON:

https://www.alphavantage.co/
root
2018-05-25 23:04:51 UTC
Permalink
Post by Rich
Don't bother.
Sign up for a free Alpha Vantage sign-on, and download in either CSV or
https://www.alphavantage.co/
Once again thanks Rich. When Yahoo Finance switched over to
Verizon the old ways of getting price data failed. I posted
here software to once again fetch the price data from the
new Yahoo.

In this case I'm not after price data but corporate financial
reports: statement of income, balance sheet, and statement
of cash flows. finance.google.com used to provide these
but not since some time in March. Yahoo does provide some
of these data but only going back 3 years.


I signed up at the alphavantage site even though it seems
more directed toward day-traders [ a suicidal venture].

The alphavantage site is very user friendly and the html source code
is probably the clearest example I have ever seen. Among the
hrefs is:

https://www.alphavantage.co/query?function=HT_DCPHASE&symbol=MSFT&interval=30min&series_type=close&apikey=demo

where I change demo to my apikey.

In this case I get json format, but I want csv. I can change the slot for MSFT
into whatever I want. I tried sticking &datatype=csv in the string, but
it always came out json. It's easy enough to convert to csv.

Thanks Rich.
Rich
2018-05-26 05:16:39 UTC
Permalink
Post by root
Post by Rich
Don't bother.
Sign up for a free Alpha Vantage sign-on, and download in either CSV or
https://www.alphavantage.co/
Once again thanks Rich.
You are welcome.
Post by root
When Yahoo Finance switched over to Verizon the old ways of getting
price data failed. I posted here software to once again fetch the
price data from the new Yahoo.
In this case I'm not after price data but corporate financial
reports: statement of income, balance sheet, and statement of cash
flows. finance.google.com used to provide these but not since some
time in March. Yahoo does provide some of these data but only going
back 3 years.
I did not realize this was your request. I am unaware of any API
method of obtaining these. Web scraping (or manual grabbing) may be
the two available options. For web scraping with JS, you might look
into one of the "run a headless browser" systems.
Post by root
I signed up at the alphavantage site even though it seems more
directed toward day-traders [ a suicidal venture].
That may very well be the case, but that does not in and of itself make
it useless to those of us who want to obtain current price quotes to
see where we stand at the moment.
Post by root
The alphavantage site is very user friendly and the html source code
is probably the clearest example I have ever seen. Among the hrefs
https://www.alphavantage.co/query?function=HT_DCPHASE&symbol=MSFT&interval=30min&series_type=close&apikey=demo
where I change demo to my apikey.
In this case I get json format, but I want csv. I can change the slot
for MSFT into whatever I want. I tried sticking &datatype=csv in the
string, but it always came out json. It's easy enough to convert to
csv.
Yeah, I tried datatype=csv and got JSON as well. Maybe that particular
function simply only returns JSON. But as you say, it is easy enough
to convert to csv in the end.

Loading...