Can you use incremental refresh on SharePoint file sources in Power BI?

YES you can use incremental refresh on SharePoint files in Power BI! The trick is, the date column you use needs to be something OUTSIDE the file, such as the “created date” column in the SharePoint library. What that means, is, this is only really useful for SharePoint folder sources, where Power BI is grabbing files in a specific folder and combining them. An example would be where someone adds a file to a folder once a month, and the older data is not being modified. That happens to be what people tend to want most of the time when they use the folder connector anyways, so it works out.

I think some of the confusion here comes around people trying to use date fields inside their file, and those will not “fold”, so they won’t work for incremental refresh, and that gets repeated without context.

Tips for setting up incremental refresh on SharePoint folder sources

Make sure to use a column that won’t be changing, hence I recommend the created date (not modified date). If you have an added custom column, that works fine too. The field type needs to be set to date/time before the parameter filtering happens (usually this is automatic, but I saw it give me a date/time/timezone type field today so watch out).

Also make sure that the step that does the filtering on RangeStart and Range End is BEFORE the expansion step and BEFORE all of your non-foldable query steps. This works out, because the created date column tends to disappear after you expand, but I just want to be super clear about that.

Yes, I tested it

Just to make sure, I tested it on a SharePoint folder query with > 1 million rows. I ran a refresh, then I added a file and refreshed again.

The first refresh that ran on the full list of CSVs took a minute and a half to run. The second refresh, where I added one extra file to the folder, took 8 seconds!

incremental refresh screenshot

This is a partner post to a video about using Power BI with Excel sources here.

Leave a Comment