How to look at Pittsburgh arrest data using an API - Beginner-friendly

 Find the data you want to access

The Western PA Regional Data Center (WPRDC) is an awesome resource with lots of Pittsburgh-related data. I wanted to access Pittsburgh city arrest data. 

After you find your data file, you'll need to find the file's resource id. It's a hash, a long string of letters & numbers with four hyphens. There's a few places this is located. If you download a sample file, the name of that file is the id. The resource id is also the last section of the page's URL. If you're visiting this webpage:

https://data.wprdc.org/datastore/dump/e03a89dd-134a-4ee8-a2bd-62c40aeebc6f

then your resource ID is  

e03a89dd-134a-4ee8-a2bd-62c40aeebc6f 

 

Build your request URL

WPRDC uses DataStore to house its data. You can ask WPRDC to send you data by requesting it in a URL format the database understands, an API. To make a request, we'll use the datastore_search call. Here's the base of the URL:

https://data.wprdc.org/api/3/action/datastore_search?

Now we need to tell the database which file we're looking for. Add your request id:

https://data.wprdc.org/api/3/action/datastore_search?resource_id=e03a89dd-134a-4ee8-a2bd-62c40aeebc6f

Since we're just getting started still, let's add a limit on the amount of data WPRDC will send us at once. That way we don't accidentally request huge data files, wasting resources for both us and WPRDC. I'll start with a limit of 5 rows of data. Once we know our program is working, we can come back later and remove the limit. 

https://data.wprdc.org/api/3/action/datastore_search?resource_id=e03a89dd-134a-4ee8-a2bd-62c40aeebc6f&limit=5  

There's lots of other helpful tricks you can use when building your URL. For example, you can add filters to reduce the amount of irrelevant data you receive. The datastore_search documentation explains more. For now, just hold on to that URL.

Install Python and a few libraries

Install Python 3. 

Install requests and pandas. The easiest way is to use pip. 

pip install requests

pip install pandas 

(Optional) I like to use Jupyter notebooks to mess around with tabular data. Consider installing Jupyter Lab and running your code in a notebook. 

pip install notebook 

jupyter notebook 

Request your data using Python

Open a new Python file or notebook, and import your libraries:

import requests

import pandas as pd

 

Paste in the request URL we built earlier:

url = https://data.wprdc.org/api/3/action/datastore_search?resource_id=e03a89dd-134a-4ee8-a2bd-62c40aeebc6f&limit=5 

Ask (request) WPRDC to send you that data, then turn it into a readable format (JSON). 

resp = requests.get(url).json()

The response contains a bunch of extra metadata that we don't need right now. So let's grab the meaty part of the response ("result") and pull out the actual data ("records").  We'll use pandas to turn that into a nice spreadsheet table (a "DataFrame"). 

data = pd.DataFrame(resp['result']['records'])

data (if you're using a notebook)

display(data.to_string()) (if you're not using a notebook) 

 

A note on API politeness

Every time you run requests.get(url), you're connecting to the web, asking WPRDC to send you data, and downloading the response. When you're using an API, try to be conscientious about the frequency and size of the requests you're making. Many databases will enforce a limit on the number of requests you make in a day, and some will even ban your IP address if they think you're trying to abuse their servers with tons of spammy requests. I didn't find any documentation about the API limits for WPRDC, but it's still best-practice to be intentional about how you design your code to only request new data when you actually need it.  

 

This tutorial was written in August 2023. 

A hack to make it easier for QA Testers in Twine (Sugarcube 2)

For the game OnlyBans, we got a wonderful accessibility audit done by the Disability Sexuality Access Network. We also periodically have non-technical QA volunteers and playtesters give feedback on the game. I don't really want them to have to learn a bunch of new tools to give effective feedback and bug reports. 

Goals:

• Testers can easily jump to/from/back to passages
• Testers can easily reference the title of the passage they're looking at
• Testers don't have to learn Twine or have overwhelming debug tools
• Don't insert new content on the screen. (e.g., I can't just add the title to the top of every page
• Make "QA Mode" something I can easily toggle on/off

To achieve these goals, I relied on adding content through the UI Bar. If your game doesn't have the Sugarcube UI bar, or if your game has complex path logic or a nonlinear history, these tactics might not be helpful for you.

1. Make a QA Passage

This passage will be a list of all the passage titles.

As far as I can tell*, there's no Sugarcube API to directly reference all the passage objects. So I instead searched all the Story objects to find objects with a non-empty passage title. Then, print that array as a link. 

!! All Pages

<<silently>>

  <<set $allPassages to []>>

  <<script>>

    var $anyRegExp = new RegExp('');

    var $passageTitles = Story.lookupWith(function (_p) {

        return $anyRegExp.test(_p.title);

        }).map(_psg => _psg.title);

    state.variables.allPassages = $passageTitles;

  <</script>>

<</silently>>

<<for _i to 0; _i lt $allPassages.length; _i++>>

<<link $allPassages[_i] $allPassages[_i]>><</link>>

<</for>>

QA Passage example 

With a few more lines of code, you could probably use the Story APIs to filter out the behind-the-scenes passages with Special Names. For example, play testers probably don't need to see the StoryInit passage listed here. 

Sometimes I'll manually add a few passages of special note. For example, during accessibility testing, I listed out a few specific passages that had complex interactions and might need special attention.

2. Make a QA mode boolean flag

In your game javascript, give yourself the nice gift of having a global variable that controls the logic of whether you want to play in the game in QA mode. 

var $isQATestMode = true;

State.setVar("$isQATestMode", $isQATestMode);

3. Add your new QA Passage to the UI Sidebar

In your StoryMenu passage (make one if you don't have one already), add a handy link to your QA Test passage. Here's what ours looks like:

[[How to Play]]

[[About]]

[[Credits]]

<<if $isQATestMode >>[[QA Test]]<</if>>

4. Display the passage name in the sidebar

In your StorySubtitle passage (make one if you don't have one already), display the name of the passage the player is currently on. That way, they can reference the correct passage if they're making a bug report. 

<<if $isQATestMode>>Current Passage: <<print passage()>><</if>>

5. Enable jumping backward & forward in time

In your game javascript file, use the handy Sugarcube Config API to make it so playtesters can go back in time. Usually I don't want my players to be able to do that.  This setting automagically adds a back button and history lightning bolt in the UI Sidebar. 

if($isQATestMode){
console.log("QA Test Mode Enabled");

Config.passages.descriptions = true;  //sets descriptions for Jump To

Config.history.controls = true; //enables back button

}
else{
console.log("NOT QA");
Config.history.controls = false; //disable backward and forward buttons

}

Screenshot with QA Mode OFF

Screenshot with QA Mode ON

What are other tricks you've used to make it easier to collaborate on Twine games? I'm especially curious for those of you with non-technical collaborators. 

*I'm usually wrong about this sort of thing, to be fair. 

How to display media with relative links in Twine play testing mode

Twine is great, and the Twine 2 UI is a fantastic little tool for developing nonlinear stories. But as of this writing, it's great for text and not-so-great for other media. You can easily embed media using html, but it's not easy to work with as you're developing or testing your game. One frustrating note from the Twine documentation:

If you are using relative links for your multimedia, you will need to publish your story to the correct location on your computer so that the references point to the right place. If you have an image tag whose source property is “myimage.jpeg”, for example, then your published file must be placed in the same folder as the file myimage.jpeg. Learn more about relative links

If you are using relative links, testing or playing your story inside Twine will unfortunately not work.

I was using many local, relative links for images and video as I was working on OnlyBans and got annoyed enough by this to make a temporary solution for myself. To use this, you need a file syncing tool. 

1. Make a media/ directory in the same location that your Twine story lives. This is likely your Twine/Stories directory. Drop your images here: Twine/Stories/images/myimage.jpg. (This step is largely for convenience.)
2. Open your story and add an image to a story passage using an HTML tag such as <img> or <video>
3. Test your story. You'll notice that instead of images, you'll get a little default 'missing image' icon. 
4. Open the developer view tool for your browser. Find your image in the HTML source panel. You can often hover over the icon to locate it. 
5. Right-click on the image name, hover over it, or open the image in a new tab to find an expanded file path. The specific filepath will vary by browser and operating system. The image below shows one way to view the expected path in Chrome. For example, on my Windows machine my browser expected to find the file at C://Users/username/AppData/Temp/Images/myimage.jpg. Our goal now is make sure that there's a copy of our image in this location. 
   
   
   
6. Open or download a file syncing tool. For Linux or people comfortable with the command line, I recommend Rsync. Right now I'm using Windows, so I used SyncFolder. 
7. Sync these two folders. The source directory should be Twine/Stories/images/ and the destination directory should be the one from your browser, C://username/AppData/Temp/Images/. If syncing feels too complicated to you, you can also just manually copy and paste your images to the destination directory.
8. Every time you add an image, alter media, change the name of a file, or close/re-open your browser, you may need to Sync that folder again. You can set up your syncing tool to run frequently, or you can manually run the sync job by pressing 'Sync' in your tool.

You might be wondering: Why not just store my images in the destination folder in the first place? That's not not ideal for two reasons. First, these temporary directories are sometimes automatically deleted by your operating system to clear up space. If you only store your images there, you may lose them to deletion! Second, when you finish your story and are ready to publish it using Twine, Twine expects to find those images in a "nearby" folder with relative paths. If you upload your media online or want to send your game to others, you will need to send them the media as well. 

Let me know if this helped you on your Twine journey, or if you have a different trick you used instead. 

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

List of Machine Learning Projects That Will Inevitably Be Used to Identify Your Nudes

Anus print detection

A mountable toilet system for personalized health monitoring via the analysis of excreta. 6 April 2020. https://www.nature.com/articles/s41551-020-0534-9

Each user of the toilet is identified through their fingerprint and the distinctive features of their anoderm, and the data are securely stored and analysed in an encrypted cloud server.

Areola and Nipple Detection "For Criminal Identification"

https://arxiv.org/pdf/1905.11651.pdf

Log In To Cam Sites with Penis Detection

(This project didn't go anywhere)

https://www.camsoda.com/labs/dickometrics/

Ear print detection

https://dl.acm.org/doi/abs/10.1145/2431211.2431221?

Blurring doesn't always work

https://ui.adsabs.harvard.edu/abs/2019arXiv190210885P/abstract

(Will be updated as more arise.)

Public Comment on AI for the Defense Innovation Board

On March 14, 2019, the Defense Innovation Board held a listening session on Artificial Intelligence Principles at Carnegie Mellon. The DIB is an independent committee made up mainly of tech execs and academics that advises the DOD. I wasn't planning on giving a comment at the session, but I got so angry listening to the pre-written publicity statements read aloud by defense contracting tech companies, that I couldn't stay seated.

Here's my very last-minute statement:

Hello, my name is Maggie Oates. I'm a PhD student here in Societal Computing and I work in Cylab, which is the cybersecurity lab here.

Let me first say that I disagree with the very enterprise and existence of the Defense Innovation Board, as it seems like a tool to lend credibility to the project of advancing military efficiency and further escalating the baseline of 'defense.' Second, I disagree with CMU's continued involvement with the military and am hard-pressed to think of an ethical and responsible use of AI at all [in defense].
That said, I would like to focus on something else today and that is what I view as an externality of the project of AI in the DOD. And that is the growth of civil surveillance, both domestically and abroad. The development of machine learning algorithms relies on massive amounts of data, of course. And while methods are being developed to reduce the amount of data required, or to reduce the amount of labeled data required, these methods often correlate with having the downside of being hard to explain and more difficult to verify, making them an unlikely use in the DOD's context. Beyond that, the project of labeling data often rests on exploitative labor practices. So I stand here to assert that any responsible principles must address the effects that DOD AI will have on surveillance, not only from the state, but also from the tech companies that will be the first line in building that AI. This topic is absolutely not out of scope. Thank you.

You can view the video and comments on the DIB site.

JSON and & "Good, not Evil"

The JSON license is apparently infamous in the FOSS community for its inclusion of a moral clause. The license was nearly identical to the classic MIT License, except the one line,

The Software shall be used for Good, not Evil.

This line has persisted in the license since 2002 when Douglas Crockford originally penned it, and not without controversy. This clause prevents the license from being considered a free license, since it places restrictions on use. That has consequences for artifacts licensed under JSON. Below are some highlights of its fascinating, contentious past and present.

Google Code stops hosting

Around 2009, Google Code (at the time a host for open-source projects) reportedly stopped hosting projects that used the license. Ryan Grove described how he chose to migrate his project, JSMin, to Github. 

In 2010, it would appear that the Google Android team replaced JSON.org implementations in core Android by writing them from scratch, with the commit "Removing the non-free org.json implementation."

Debian rejects use

In 2010, the debian-legal listserv re-visited the topic of whether JSON.org software can be included in Debian's free distributions. Thomas Koch reported he emailed Crockford about the line and received the response,

If you cannot tolerate the license, then do not use the software.

The opinion of the listserv, summarized by Josselin Mouette, was

Definitely non-free, and the author’s clarification removes any doubt.

Crockford & "Evil" IBM

In a fascinating talk for Yahoo on the history of JSON (Trivia: the first JSON 'Hello world' message failed), Crockford discussed granting a license change. In a short section on the history of the license he says, 

this was late in 2002, you know, we just started the war on terror and you know we were going after the evildoers with the president and the vice president and I felt like I need to do my part. So I added one more line to my license was that the software shall be used for good not evil.

 However, he also describes how he's frequently approached by individuals and companies wanting to use his license. After IBM asked for a separate license, he allowed IBM to use his software under the JSON license, writing,

I give permission to IBM, its customers, partners, and minions, to use JSLint for evil.

Apparently that was enough for IBM lawyers in 2011. So Crockford has, in fact, granted adjusted licenses to some large organizations like IBM.

Crockford & Actual Evil

In 2012, compelled by a conference talk that covered how "the regime in Syria was using open source software in its efforts to track down and murder anyone with the courage to oppose the evil government," Crockford posted on Google+ ,

My very, very, very small part in this is that I include The software shall be used for good, not evil in my licenses. It is not effective at all, but it at least states my intention. It is quite controversial in the open source community because it is claimed that it restricts a specific field of use (specifically, evil), and that software cannot be considered free unless its license permits it to be used in the investigation, torture, and murder of patriots who dare to resist tyrants.

 The line, "It is not effective at all, but it at least states my intention," brings forth questions about the tensions between intention and impact. I'd be curious what Kant thinks about this case study.

 Github & project bug reports

Across Github and other software hosts, you can find references and arguments to the license in bug reports and feature requests. 

In 2013, @blickly opened the issue "json.org is not free software #187" on the popular jshint project for Javascript linting and hints. This issue has 79 comments and remains unresolved as of this writing. In September 2017, user @eclipseo commented that the issue was "preventing us from including it in Fedora."

In contrast, the open project management system Trac resolved their use of JSON software in less than two months in 2012. They simply replaced the parser they were using. 

Change.org petition

2014, Thomas Koch (yes, the same Koch from Debian) begins a Change.org petition to get Crockford to remove that line from its license with the reasoning

Many free software activists are also active for good causes in other areas than software. The non-evil clause costs them time and nerves and distracts them from doing good.

The petition got 35 signatures.

Apache doesn't get the joke

In 2016, Apache reversed (archived text) its prior 8-year decision to use JSON-licensed software in its products. Originally, circa 2008, the legal-discuss listserv's thread indicated that "Good, not Evil" was, for the most part, a joke rather than a legal concern. Henri Yandell wrote, 

It's a joke clause in my opinion.

Ethical Licensing

Soon after I first learned about open source and free software license, I started wondering about what sort of clauses could be added to licenses. Recently, I've become progressively interested in licenses with morality clauses in them.

I'm starting a list of the licenses (and their relatives such as Terms of Service) I come across that contain apparent ethical clauses. You can find these under the ethical-license label.

Here is a general Stack Overflow question on this topic; another on the relationship to OSS. Looking forward to exploring this topic with you. 

On Being A Nice Hacker Princess

Spring break. I was sitting in the airport, lamenting that three of my professors had assigned work due the week after break, but delighted that there was enough WiFi to squeeze in a half hour of work. I was focused, writing something in C++ for my Computer Vision class when I noticed a girl peering over my shoulder, maybe 7 or 8 years old & with a ponytail.

Me: "Hi"

Girl: "Um, are you a hacker?"

I was outwardly very amused and inwardly very pleased at the question.

Me: "No, not exactly. I'm in college & I study Computer Science. I write code to ask my computer to make things like websites or Candy Crush. So it's kind of like being a nice hacker."

The girl was thoughtful for a second, ignoring her mom's call that it was time to board. Then she said "OK" & ran off to her mom, who smiled apologetically to me as her daughter jumped up and down in front of her & informed her of important news.

Girl: "Mom, when I grow up I want to be a hacker"

Mom: "A hacker? This morning you wanted to be a princess!"

Girl: "Yeah! Both! I mean, I want- I want to be a ummmmm nice hacker princess"

A Nice. Hacker. Princess.

As I watched them leave, my heart swelled full of hope. There was something so pure, so quietly radical about her statement. She could be a hacker even though she was a princess, dammit. The two did not involve an XOR* in her book.

I used to want to be a software engineer or a researcher or a data scientist, but my aspirations changed in that moment: I, too, wanna be a nice hacker princess. I want to revel in my CS femmeness, I want to break things & fix them & get elbow-deep in code in a hella cute dress. I want to taste the satisfaction of functional code, admire it's resemblance to magic. I want to hack my IDE's into having glitter font. (jk, mostly) I want to do all of this while being nice- contributing my hacker-princess talents to something meaningful, something kind & impactful & clever.

I found a role model in a 7yr old, a small blonde creature who saw no limit in her potential, no paradox in her newfound dream. I refuse to use XOR's in my vernacular, to limit my princessliness with real or self-imposed expectations and inhibitions. I will forever find power in what femmetech means to me. And whether you are woman, man, fluid, trans, all of the above, or none of the above, I invite you to find power in whatever that means to you.

XOR XOXO 
Yrs truly,

a nice hacker princesses

*XOR = "exclusive or". Either this or that, but not both. 

Testing & Pooping at Google

"Testers are paid to break stuff."

I've heard this line a dozen times from a dozen different people, most recently during a presentation by IU alum and current Googler Mark Meiss. Breaking things is fun, so I decided to look into Testing more, specifically Test at Google.

Google has a great blog on testing there and has an especially nice overview series of Google testing, but I'd like to share a few interesting or quirky things I've learned in the past week about Test at Google.

Learn & Poop

Don't you just love sitting down on that porcelain throne and taking a good...look at the testing tips on the walls? Google has a program called Testing on the Toilet (TotT). Bathroom stalls are plastered with best practices and tricks. Sure, it's old news (the program started back in 2007), but it's still interesting and effective.* I'm also struck that the program is a good reflection of testing in general- like a bathroom, Test isn't exactly sexy, but without it things at scale would get real shitty real fast.

The topics range from simple things like effectively naming tests to more technical specifics for Web Toolkit, but nearly all are very accessible concepts if you know basic coding. So next time you're "backing the bus outta the garage" (ew/lol), I recommend scrolling through the TotT tag on your phone to pick up a few tips. 

SET vs TE

(Software Engineer in Test vs Test Engineer)

A nuance I wasn't previously aware of was the fact that Software Engineer in Test and Test Engineer are not synonyms. From what I understand, the two roles can be very similar, but do have different focuses. Pulled from this 2012 interview with a Google TE;

As a Test Engineer, you’re more focused on the overall quality of the product and speed of releases, while a Software Engineer in Test might focus more on test frameworks, automation, and refactoring code for testability. I think of the difference as more of a shift in focus and not capabilities, since both roles at Google need to be able to write production quality code. Example test engineering tasks I worked on are introducing an automated release process, identifying areas for the team to improve code coverage, and reducing the manual steps needed to validate data correctness.

If it matters to you, looking at the oh-so-scientific Glassdoor pages gives salary estimates for each. SET's and TE's make roughly the same $111,000 and $108,700 respectively at the time of this writing. And according to this Google Testing post, the opportunities for career advancement were similar.

Orbs

In the early 2000s, different Google teams tried out different ways of monitoring their test builds to make sure everything was running smoothly. Each team has a centralized, web dashboard for their builds. The centralization is nice but is not a particularly vigorous. As Mike Bland pointed out in a really interesting post on the development of builds at Google,

"...a build system’s value is proportional to the rapidity with which a team can identify breakages and fix them, and having every engineer keep a web page open and refresh it all the time doesn’t scale."

"Statue of LORBerty" taken from
the Google Testing Blog

So a few teams turned to casual, at-a-glance methods of identifying build fail moments of "Oh shoot, we done messed up." Bland talks about teams using a literal traffic light, browser plugins, and (his project) Ambient orbs. 

Ambient builds devices that center around relaying information in an effortless way. Bland threw together some scripts that used one of their products, Orb, to change color based on the status of a build. Hence the birth of the Statue of LORBerty, providing liberty and justice and peace of mind to testers everywhere.

*[Funnily enough, we use the same technique for event advertising or  awareness campaigns pretty frequently in the residence halls at IU. Communal bathrooms are places that receive high traffic with pretty captive audiences. Other places I like are inside elevators or next to clocks in lecture halls. Flyering bulletin boards just doesn't cut it.]