Posted on by admin

About APIs and SDKs

API vs. SDK

Early on in the life of Pathomation, it became clear that in order to tackle the variety of use cases out there for digital pathology, we needed to build of piece of veritable digital chameleon software.

Luckily there is a way to do exactly that in engineering, and that is through the establishment of an Application Programming Interface (or API for short). It all starts with an API.

Hugo Bowne-Anderson of Datacamp explains what that means: An API is a set of protocols and routines for building and interacting with software applications.

Why are APIs important? An API is a bunch of code that allows two software programs to communicate with each other. You can connect to an API, pull data from them, and subsequently parse that data.

Using APIs has become the standard way to interact with applications ranging from Wikipedia to Twitter. PMA.core (and its little brother PMA.start) has an API, and our own product suite makes heavy use of it.

So while PMA.core isn’t our main product, it is definitely where everything starts, and today I see how many of our success stories can be attributed to the API that PMA.core offers.

Version 2

Currently we’re in the works of wrapping up version 2 of PMA.core. As it should be, we learned a lot in the last couple of years about how (not) to use our own interfaces.

Some good things:

Some not so good things:

  • We limited ourselves in terms of granularity. Here’s an example: PMA.core offers the possibility to store slide meta-data in an audit-trailed, 21CFR part 11-compliant manner. However, the present interface only allows to get data for one slide, one form at the time. So when you work on courseware projects like PathoTrainer, you need to put in additional progress bars, while the underlying data is retrieved in an atomic fashion. In contrast, of course, you’d much rather be able to retrieve the n metadata sets for m slides in a single call. In version 2 we will be able to do just that, and more.
  • We need much more documentation and more sample code. Pointing people to your WSDL manifest is NOT sufficient.

SDK

We’ve always had the idea to bring out a complete Software Development Kit (or SDK for short). The idea for an SDK is simple enough: take everything your API can do and build content every call and every combination of parameters imaginable.

But what exactly do you put in it? What languages do you support? Looking around our own environment, we chose Microsoft.Net, Java and PHP as prime candidates. We know there are more of course, but you have to pick your battles.

We thought we were doing pretty well, having Java and Microsoft.Net desktop application sample code, until one partner told us they were using Delphi. What’s next? Windev https://fr.wikipedia.org/wiki/WinDev?

Two problems then exist with the SDK the way we currently have it:

  • When we make a matrix with documented features and supported languages, we end up with a rather sparse matrix. This means that some things are documented in one language, but not in another. There are a couple of essential tasks that we have in any environment of course, like establishing a connection, but it’s inconsistent at best. We thought actually that we could do this demand-driven, meaning that when someone asks us how to do task t in language l, we just fill up cell (t, l) in the foresaid matrix. This works, to some extend; you can respond to the client, and be confident that the code you contribute is actually something application develops want. But the end-result is messy and doesn’t look very professional.
  • Our code examples that we’ve been adding were mostly wrappers around API calls. In Microsoft.Net, we’d a WebRequest call and interpret the returned JSON or XML stream. In PHP, we’d do a file_get_contents of whatever API call we needed to get the job done, and again interpret the results. This got the job done, but as a result much of the code that we delivered was more a tutorial in how to read webcontent and interpret the returned structured data. Ideally however, these should focus more on what can  actually be done with the software (instead of how to do something).

Wrapper libraries

For the current version 2 then, I want to be more up-front with our SDK offering. I want to be better prepared. I don’t think we should try to convince people anymore to “just” go and use our API. It’s too abstract, too inconsistent even at places (sometimes for historical reason; who thought you could grow legacy so quickly?), and frankly too steep a learning curve probably for a great many people.

Pathomation offers a platform for the development of digital pathology software. I want to make sure people like to stand on our platform to begin with. It’s not necessarily what I want to do w/ it… it’s what others would want to with it.

We have already have code in python that fetches images from PMA.core and does “something” with them. But there’s nothing fancy about your sample Python script; you DL a /region URL, then you process it like any other image. This is at API level.

A wrapper library can now add more functionality; repetitive basic tasks. What do developers not like to do? Write plumbing code. We can encapsulate all of that under the hood of a PyPI package or Java namespace. We’re already doing that for the front-end handling and representation of whole slide imaging content with our Javascript PMA.UI framework.

Here’s another idea: I can’t do a nuclear cell count on an (reduced resolution) overview image; I need at least 20x resolution for that. But I can do a cell count on an individual tile at high resolution, and then put on a dot on the overview image to at least indicate where the cell was found. What I would want to do instead is create an overview image of e.g. 1000 x 2000 pixels, then loop through x * y tiles at a zoomlevel that in reality represents 5000 x 10000 pixels (but which is too bulky to process in one time via /regio; we’re not VIPS); process the individual tiles, scale and imprint the result from each  tile back onto the 1000 x 2000 pixel overview image.

I can imagine a class representing a slide that has indexer logic that retrieves tiles in real time (sort of like a programmer’s server-side version of PMA.UI), but then destroys them again once the object is not needed anymore (so the memory usage doesn’t explode).

from pathomation import pma

dir = pma.get_first_non_empty_directory()
slide = pma.get_slides(dir)[0]

max_zoomlevel = pma.get_max_zoomlevel(slide)
print ("Max. Zoomlevel: " + str(max_zoomlevel))
print ("Size in pixels: " + str(pma.get_pixel_dimensions(slide)))
print ("Resolution (PPM): " + str(pma.get_pixels_per_micrometer(slide)))
print ("Physical resolution (µm x µm): " + str(pma.get_physical_dimensions(slide)))
print ("Number of channels: " + str(pma.get_number_of_channels(slide)))
print ("Slide is fluorescent? " + str(pma.is_fluorescent(slide)))
print ("Number of tiles: " + str(pma.get_number_of_tiles(slide)))

selectedZl = 1 # do the following on zoomlevel 1 for demo purposes
tileSz = pma.get_number_of_tiles(slide, selectedZl) # zoomlevel 1
for tile in pma.get_tiles(slide, toX = tileSz[0], toY = tileSz[1], zoomlevel = selectedZl):
     tile.show()
     # do something with the tile

In closing

There’s a tremendous problem today with scientific software: published methods are described at a very high level; and not at all easy to replicate. And when they are detailed enough, or source code is made available, it’s not in an accessible language like Python (or Java for that matter; or it has sooooo many dependencies…). Too many research papers can be concluded with “now good luck finding the one former postdoc that actually knew how to do this…”

Remember “Developers developers developers”? Yes, make fun of Steve Balmer all you want, but he got it right. You offer people basic services, and then get those people to develop software on top of your infrastructure.

So how do we intend on addressing this? By continuing to make our own software as versatile and kick-ass as possible (duh 😊), but also by going just one step further and reaching out to the legion of developers and researchers out there that currently still just have to make do with what they can get their hands on. We claim to have a better mousetrap for you, and we’ll prove it to you.