Post New

The source that Elan mentioned for retrieving pricing (Jason Strimpel http://finance.jasonstrimpel.com/bulk-stock-downlo...) has not been working for the last few weeks so I am trying Wisesheets that was also mentioned. For Excel, Wisesheets does not work on Excel 2019 for retrieving pricing, it does work for getting other data. They advised that you need Microsoft 365. On Goooglesheets it works fine. The only issue I have noticed is that although there is a formula for adjusted prices, the most recent dividend is not included (at least in the three stocks I checked, XOM, APPL and GOOG). I have written to them and received a reply that they would check with their vendor.

Aussi

(Jason Strimpel http://finance.jasonstrimpel.com/bulk-stock-downlo...) has not been working for the last few weeks

Yeah, that's the downside of depending on the kindness of strangers. ;-(

Yahoo changed the download features once again a few weeks ago. I guess they finally realized that everybody who cared has long ago figured out their cookie/crumb trick.

Anyway, people who are really interested and into this stuff pretty quickly figured out the new side door. But this time, nobody seems to be posting all over the internet the way to download that data now. (I'm one of those.) You have to get kinda down and dirty with Linux, although I guess you could also manage to do it on Windows if you had the proper tools.

Years (many) ago when I was into Mechanical Investing I grabbed data from all kinds of free sources (Multex, Yahoo and many others whose names I forgot) with a technique that always works: WSH, the "Windows Scripting Host", and therein the "SendKeys" method, programmatically simulating keystrokes.

All that is required is to have the data you want to grab on screen in a webpage. Then your program sends the keystrokes you would manually use to select, copy and paste into an Excel sheet one after another the data blocks you are interested in.

Downside: It works only until the layout of one of those webpages is changed. Then you have to adapt your program to the new design = if lucky only to the new location of the same data. That's regularly highly frustrating and a lot of work, but this technique works with every webpage.

WSH, the "Windows Scripting Host", and therein the "SendKeys" method, programmatically simulating keystrokes.

All that is required is to have the data you want to grab on screen in a webpage. Then your program sends the keystrokes you would manually use to select, copy and paste into an Excel sheet

I don't know if even that would work with some of the sources I previously used.

My internet is a slow DSl (the downside of being in the mountains), so I can sometimes see how a web page is painted.
One of the sources (WSJ, I think) used to put all the data up in one go. Then they changed it and only the static text is painted at first. Then, probably by some java scripting, the data fields are populated. Slow enough on my connection that I can see the data values get painted one at a time. It takes 1-2 seconds for the page to stabilize.

The 99% of the rest of the world are on faster internet, 10 times faster than mine, so they wouldn't see it.

I suspect WSH and the like would have trouble with this.

I am aware of various methods of sending keystrokes, but didn't feel the work was worth the effort, so I gave up on the screen(s) that needed that data.

For automation

- WSH can be good for automating native windows applications. It has been a while since I've used either, but I thought Powershell was the direction Microsoft was leaning for the future.
- BeautifulSoup can be good for automating websites and browsers. If native browsers are needed, Playwright and Selenium, which interact with browsers through management protocols. So something like "If a div element with specific test is present and visible, issue a scroll down event" is easily expressible.
- For more exotic cases, I've used to used Sikuli (https://www.sikuli.org/) which was based on OpenCV.

The latter one will use computer vision techniques capture a screenshot and recognize elements you want to interact with visually, and determine their screen coordinates dynamically. It is a little more robust for differing screen geometries or cases where simply sending an event at a particular screen coordinate is difficult for timing or precision reasons.

It has been more than a decade since I've used Sikuli and I assume better options are now available, but it represents a powerful and generic automation technique.

The source that Elan mentioned for retrieving pricing (Jason Strimpel http://finance.jasonstrimpel.com/bulk-stock-downlo...) has not been working for the last few weeks so I am trying Wisesheets that was also mentioned. For Excel, Wisesheets does not work on Excel 2019 for retrieving pricing, it does work for getting other data. They advised that you need Microsoft 365. On Goooglesheets it works fine. The only issue I have noticed is that although there is a formula for adjusted prices, the most recent dividend is not included (at least in the three stocks I checked, XOM, APPL and GOOG). I have written to them and received a reply that they would check with their vendor.

Jason Strimpel may not know that his bulk stock price download has stopped working. I haven't found a way to contact him directly. Someone with a verified X account or a LinkedIn premium account may be able to do so, based on the links on his website.

In any case, can you provide more info on using Wisesheets? I may want to use it next week.

Thanks,
Elan

Elan

When the retrieving site went down a few months ago I contacted Jason at J S <strimp101@gmail.com and he immediately fixed it. I have recently sent a note again but no response yet.

As for Wisesheets

If you don't want to bother in the short term, you can send me a list of stocks and the period when you want pricing, and I can return a csv file.

If you want to set up Wisesheets, here is a link.

https://www.wisesheets.io/?gad_source=1

It is $60/year, but I found a $15 coupon fairly quickly. The free version is not sufficient for pricing data.

It is an add on that you have to go through the Microsoft store, if you have 365, or through google if you use googlesheets. Once you have the add on, it is quite easy to use. Put a list of tickers in a column A2 downwards, put the command =WISEPRICE(A2:A387, "AdjClose",350)in cell B1. A 387 is the last cell for my ticker list, use what is the last for you. 350 is the number of calendar days I want retrieved. It only retrieves trading days so there are no blanks to worry about.

There are various prices that can be retrieved. High, low, close and other data.

As mentioned, I think adjusted close is not correct as it is missing an adjustment for the most recent dividend.

I have communicated with the help desk a few times. Not real responsive but I did get answers, or a reply that they would check with their vendor.

Aussi

If you want to set up Wisesheets, here is a link.

Thank you!

Elan

I assume better options are now available, Some sites are simply member gating everything.

An example: BC's extensive data tables used to be simply scrapable using googlesheets importhtml command. As were Stockcharts' charts; they were even accessible in Excel integrations.

Those sites have put their table content and chart content behind javascript objects that aren't reachable by any of the basic tools I'm aware of.

The only way I can get BC PAMA data is with a basic account; I literally hit the link to get the table response and then click in one very specific location, drag down through the 3 months of data rows (only and no further), copy & paste into a Sheet (which doesn't understand it's in columns), and wrote a Gscript (macro lite) to copy / transpose paste every 7 rows in to one row with 6 columns. I do this once per PAMA period (10,20, etc.), once a week. If I paid for a premium account I could download & import these but...

Yes I'm cheap and still working fulltime and not interested in buying Pinnacle data.

Thank the "charity" of etfscreen.com for leaving their tables open in html, which is why I still give them a donation every year.

FC

It takes 1-2 seconds for the page to stabilize.
I suspect WSH and the like would have trouble with this.

WSH and other scripting languages have a Sleep (or equivalent) command. Send the URL request, followed by a sleep for 10 seconds before continuing. It's slow but just let it run overnight. Just set the number of seconds high enough that it always works. (If it fails, restart the download at the stock that failed, then merge the two files.)

Another option for certain data - one that I use & so may be of interest to others - is Financials-Extension:

https://github.com/cmallwitz/Financials-Extension

Some provisos ...

1) It's for LibreOffice - which I use as a free, easy to access & continually updated alternative to old-school Excel.

2) It works by scraping Yahoo! & the FT and so can be (a little) slow. I typically use Yahoo! with the FT as a fall-back during the usual periods of Yahoo! shenanigans (I handle this in the various formulas I use to access the data) & so IMO it works pretty reliably (due to the fallback). It also has Coinbase data if that's of interest. If I was to use it a lot I'd create a cron (or similar say Power Automate for Windows) job to pull the data I needed automatically overnight in conjunction with 3) below.

3)It's open-source & Python so could be deconstructed to offer the data in another format - databases etc. - if that's what you prefer. I haven't taken a close look but there's (likely) a subroutine or library to manage the Yahoo! / FT / Coinbase scraping that could be wrapped in your own output code. That way you could update the Financials-Extension core in-line with the Github update for ease-of-use (when Yahoo! makes a change).

Just a suggestion. Might be worth a quick look given it's Python.

The Strimpel website is up and running at

https://bulkstockdatadownloader.app/

I am not sure but there may be a limit on the number of stocks that can be downloaded at one time. I could not get my SP400 stock pricing but I could get all of the stocks in QQQ.

In the donation section he mentions that it costs $499 to run the server.

Aussi

The Strimpel website is up and running at

https://bulkstockdatadownloader.app/

That's awesome. Thanks for the update. I've been using it regularly to identify my 6/3 option picks, although right now I'm inclined to skip next week's reinvesting cycle. The option returns have been brutal in the past month. No surprise. I may keep my powder dry for a while.

Elan

Post New

Berkshire Hathaway FAQ

---

Contact Shrewd'm
Contact the developer of these message boards.