Author Archives: Matt

Friendly Reminder: Don’t use platform defaults

tl;dr Don’t use platform defaults as they may not be the same across all machines, especially when sending data in between them. Microsoft has a big warning section in the Encoding.Default property docs about this. The story A few years … Continue reading

Posted in Azure, C# | Leave a comment

Get Azure Functions 2 Preview Up And Running On OS X With C# Functions

** This is what I encountered as of April 16, 2018 so this may no longer be accurate. Also, note that this is for the preview version. tl;dr Finally after much frustration and a bit of luck, I got something … Continue reading

Posted in Azure Functions, C# | Leave a comment

Disappearing Azure Data Lake Files?

tl;dr If you upload files from the Azure Data Lake Explorer in Visual Studio, it will set an expiration time on the uploaded files. That expiration time looks to be about one week from the upload time. Once that time … Continue reading

Posted in Azure, Data Lake, U-SQL | Leave a comment

Friendly Reminder: GetHashCode Isn’t an Identifier

tl;dr Don’t use GetHashCode to identify an object. Microsoft has a big warning section in the GetHashCode docs about this. The story We store around 70 million unique objects in Azure Table Storage and we access them through an API … Continue reading

Posted in C# | Leave a comment

U-SQL – Extracting Information From a Column Containing JSON

There are plenty of examples on how to parse JSON files using U-SQL, but what if one of the columns of a TSV contains JSON like this example: ID Name Vehicle 1 Joe “{“”Year””:””2017″”,””Make””:””Ford””,””Model””:””Fusion””}” 2 Bob “{“”Year””:””2016″”,””Make””:””Kia””,””Model””:””Sorento””}” We can adapt … Continue reading

Posted in Azure, Big Data, JSON, U-SQL | 2 Comments

Using Azure Functions to Replace Cloud Workers

When looking at options for replacing Worker Roles in Azure Cloud Services, more and more, Azure Functions look to fit the bill. I like the idea of just writing code that focuses on what needs to be done and not … Continue reading

Posted in Azure, Azure Functions | Leave a comment

Repost: Filtering columns out of data using Azure Data Factory

At work, every night we ingest a couple of large comma separated files (CSVs) that are 45 to 50 GB each. These files are generated externally, so we don’t control what information is in them. For our purposes, we really … Continue reading

Posted in Data Factory | Leave a comment

Whoops, starting over (or why you should really keep your contact info up to date)

Last year, I started blogging again. Nothing major, just tidbits from things I struggled with at work or found interesting. Back in September, my hosting at dreamhost expired, they sent out warnings and I never got them. It’s not their … Continue reading

Leave a comment