Keboola rocks

It’s been a while when I wrote my first Keboola component (in Czech). I improved it a bit during the time (like supporting incremental extracts), but didn’t have time for a big changes (like BULK API 2.0). Just lately got another request for change – the increments work nicely for „normal“ objects but not for historical. Reason being, that historical objects don’t have LastModifiedDate field which I brutally add to the WHERE clause.

Ok, that would be nice change, to allow for incremental upload to Keboola and still support custom WHERE clause. Would be great to have a dynamic UI of the component, so it shows only at the right time and so on. So I asked, whether we can make a dynamic UI. And after a bit of back and forth I found out, that I don’t have to update my code and can just use what Keboola offers.

3rd party components

Keboola has a library of components. Some of them are „real“ components, which connects to other systems, while others just work in combination – like zip files, add row numbers and so on.

I have no idea how to use them, but Najlos has, so he updated my normal configuration, to support incremental load any time. So now, my extractor just extract data which I specify (and I can use something like LAST_N_DAYS:5 to load just part of the data) and this processor will do the incremental load to Keboola.

{
  „parameters“: {
    „sinceLast“: false,
    „objects“: [
      {
        „name“: „Contact“,
        „soql“: „SELECT FirstName,RedNoses__Last_Donation_Amount__c FROM Contact WHERE RedNoses__MaxDonation__c > 10″
      }
    ]
  },
  „processors“: {
    „after“: [
      {
        „definition“: {
          „component“: „keboola.processor-create-manifest“
        },
        „parameters“: {
          „incremental“: true,
          „primary_key“: [
            „Id“
          ]
        }
      }
    ]
  }
}
The first part defines from which object I want to extract data and SOQL statement (so I want to extract all who donated more than specific amount) and the second part says that after extracting data the manifest file should be updated and set the incremental flag to true and primary key to Id. So it will take all extracted data and add them to existing table.
Nice, I didn’t have to change anything and it works exactly as needed.

Leave a Reply