MAX BERGGREN

Linguistic analysis of who wrote the op-ed

Fri, 07 Sep 2018 22:39:50 +0200

David Robinson did a nice writeup of using his R package to analyze who wrote the “I Am Part of the Resistance Inside the Trump Administration” op-ed in NYTimes. His approach was with TF-IDF of the words.

I wanted to try this with different text statsistics of the linguistic features instead, since I’m guessing word usage will not give the author away. And in Python of course.

import spacy
import re

import pandas as pd
import numpy as np
from textstat.textstat import textstat
from twitter_scraper import get_tweets
from sklearn.metrics.pairwise import cosine_similarity
import seaborn as sns
import matplotlib.pyplot as plt
plt.rcParams['figure.dpi'] = 180

from spacy.attrs import ORTH

pd.options.display.max_columns = 999
pd.options.display.max_rows = 999
nlp = spacy.load('en')

article_text = "President Trump is facing a test to his presidency unlike any faced by a modern American leader. It’s not just that the special counsel looms large. Or that the country is bitterly divided over Mr. Trump’s leadership. Or even that his party might well lose the House to an opposition hellbent on his downfall. The dilemma — which he does not fully grasp — is that many of the senior officials in his own administration are working diligently from within to frustrate parts of his agenda and his worst inclinations. I would know. I am one of them. To be clear, ours is not the popular “resistance” of the left. We want the administration to succeed and think that many of its policies have already made America safer and more prosperous. But we believe our first duty is to this country, and the president continues to act in a manner that is detrimental to the health of our republic. That is why many Trump appointees have vowed to do what we can to preserve our democratic institutions while thwarting Mr. Trump’s more misguided impulses until he is out of office. The root of the problem is the president’s amorality. Anyone who works with him knows he is not moored to any discernible first principles that guide his decision making. Although he was elected as a Republican, the president shows little affinity for ideals long espoused by conservatives: free minds, free markets and free people. At best, he has invoked these ideals in scripted settings. At worst, he has attacked them outright. In addition to his mass-marketing of the notion that the press is the “enemy of the people,” President Trump’s impulses are generally anti-trade and anti-democratic. Don’t get me wrong. There are bright spots that the near-ceaseless negative coverage of the administration fails to capture: effective deregulation, historic tax reform, a more robust military and more. But these successes have come despite — not because of — the president’s leadership style, which is impetuous, adversarial, petty and ineffective. From the White House to executive branch departments and agencies, senior officials will privately admit their daily disbelief at the commander in chief’s comments and actions. Most are working to insulate their operations from his whims. Meetings with him veer off topic and off the rails, he engages in repetitive rants, and his impulsiveness results in half-baked, ill-informed and occasionally reckless decisions that have to be walked back. “There is literally no telling whether he might change his mind from one minute to the next,” a top official complained to me recently, exasperated by an Oval Office meeting at which the president flip-flopped on a major policy decision he’d made only a week earlier. EDITORS’ PICKS The Flourishing Business of Fake YouTube Views The Iraqi Spy Who Infiltrated ISIS What Happens to #MeToo When a Feminist Is the Accused? The erratic behavior would be more concerning if it weren’t for unsung heroes in and around the White House. Some of his aides have been cast as villains by the media. But in private, they have gone to great lengths to keep bad decisions contained to the West Wing, though they are clearly not always successful. It may be cold comfort in this chaotic era, but Americans should know that there are adults in the room. We fully recognize what is happening. And we are trying to do what’s right even when Donald Trump won’t. The result is a two-track presidency. Take foreign policy: In public and in private, President Trump shows a preference for autocrats and dictators, such as President Vladimir Putin of Russia and North Korea’s leader, Kim Jong-un, and displays little genuine appreciation for the ties that bind us to allied, like-minded nations. Astute observers have noted, though, that the rest of the administration is operating on another track, one where countries like Russia are called out for meddling and punished accordingly, and where allies around the world are engaged as peers rather than ridiculed as rivals. On Russia, for instance, the president was reluctant to expel so many of Mr. Putin’s spies as punishment for the poisoning of a former Russian spy in Britain. He complained for weeks about senior staff members letting him get boxed into further confrontation with Russia, and he expressed frustration that the United States continued to impose sanctions on the country for its malign behavior. But his national security team knew better — such actions had to be taken, to hold Moscow accountable. This isn’t the work of the so-called deep state. It’s the work of the steady state. Given the instability many witnessed, there were early whispers within the cabinet of invoking the 25th Amendment, which would start a complex process for removing the president. But no one wanted to precipitate a constitutional crisis. So we will do what we can to steer the administration in the right direction until — one way or another — it’s over. The bigger concern is not what Mr. Trump has done to the presidency but rather what we as a nation have allowed him to do to us. We have sunk low with him and allowed our discourse to be stripped of civility. Senator John McCain put it best in his farewell letter. All Americans should heed his words and break free of the tribalism trap, with the high aim of uniting through our shared values and love of this great nation."

Here’s a function that takes a arbritary text, and calculates various text statistics. Stuff like mean sentence lenght, variance in sentence lenght, different text difficulty measures from the textstat package. And of course the usage of how much different part of speech is present through the package Spacy.

def linguistic_features(text):
    
    """ Takes a text as input and returns various linguistic features of it """
    
    doc = nlp(text)      
    
    # Some standard statistics
    sent_lens = [len(s) for s in doc.sents]
    mean_sentence_length = np.mean(sent_lens)
    sentence_std = np.std(sent_lens)
    counts = doc.count_by(ORTH)
    n_sentences = len(list(doc.sents))
    
    def per_sentence(char):
        """ Count occurances and divide by number of sentences """
        try:
            return counts[doc.vocab.strings[char]] / n_sentences
        except KeyError:
            return 0.0
    
    all_caps_count = 0
    pos_counts = {}
    for token in doc:
        # Count capitalized words
        if token.text.isupper():
            all_caps_count += 1
        # Collect POS statistics
        try:
            pos_counts[token.pos_] += 1
        except KeyError:
            pos_counts[token.pos_] = 1
    pos_per_sent = {f'{k}_per_sent': v / n_sentences for k, v in pos_counts.items()}
    
    return {
        'flesch_reading_ease': textstat.flesch_reading_ease(text),
        'smog_index': textstat.smog_index(text),
        'flesch_kincaid_grade': textstat.flesch_kincaid_grade(text),
        'coleman_liau_index': textstat.coleman_liau_index(text),
        'dale_chall_readability_score': textstat.dale_chall_readability_score(text),
        'difficult_words_per_sent': textstat.difficult_words(text) / n_sentences,
        'linsear_write_formula': textstat.linsear_write_formula(text),
        'dots_per_sent': per_sentence('.'),
        'commas_per_sent': per_sentence(','),
        'colons_per_sent': per_sentence(':'),
        'bindings_per_sent': per_sentence('-'),
        'long_bindings_per_sent': per_sentence('–'),
        'quotes_per_sent': per_sentence('"'),
        'questions_per_sent': per_sentence('?'),
        'exclamations_per_sent': per_sentence('!'),
        'mean_sentence_length': mean_sentence_length,
        'sentence_std': sentence_std,
        'uppercase_words_per_sent': all_caps_count / n_sentences,
        **pos_per_sent
        
    }

In lack of time to find good text samples of the White House officials I’ve used their Twitter accounts. This is of course a stretch since there’s no guarrantees that the style you use on Twitter would match the style you use when writing an op-ed. But maybe, just maybe it matches enough?

# List of White House officials twitter accounts to examine
wh_officials = [
    'TomBossert45',
    'jdgreenblatt45',
    'VPComDir',
    'SecPompeo',
    'RajShah45',
    'SecAzar',
    'SecNielsen',
    'USTradeRep',
    'SecretarySonny',
    'SecretaryRoss',
    'OMBPress',
    'EPAAWheeler',
    'SecShulkin',
    'SecretaryPerry',
    'SecPriceMD',
    'BetsyDeVosED',
    'SecretaryCarson',
    'SecretaryZinke',
    'SecElaineChao',
    'POTUS',
    'SBALinda',
    'SecretaryAcosta',
    'Cabinet',
    'VP',
    'stevenmnuchin1',
    'nikkihaley',
    'realDonaldTrump',
    'mike_pence',
    'sendancoats',
    'PressSec',
    'GeneralJohnK',
    'KellyannePolls',
    'StephenMoore'
]

We don’t need the urls.

def remove_urls(t):
    return re.sub(r'https?:\/\/.*[\r\n]*', '', t, flags=re.MULTILINE)

Let’s take a twitter accounts collected text and chunk it. That way we can see if the similiarity is consistent by boostrapping some sort of confidence of the similarity.

def collect(user):
    """ Collect text from twitter account """
    dump = ""
    for tweet in get_tweets(user, pages=50):
        try:
            dump += ' ' + remove_urls(tweet['text'])
        except:
            continue
            
    return dump

def chunk(in_string, num_chunks):
    """ Chunk a string onto `num_chunks` of equal size """
    chunk_size = len(in_string) // num_chunks
    if len(in_string) % num_chunks: chunk_size += 1
    iterator = iter(in_string)
    for _ in range(num_chunks):
        accumulator = list()
        for _ in range(chunk_size):
            try: accumulator.append(next(iterator))
            except StopIteration: break
        yield ''.join(accumulator)

Now we are ready to collect and calculate all the statistics.

data = {}
for o in wh_officials:
    print(o)
    text = collect(o)
    parts = list(chunk(text, 10))
    
    data[o + '_1'] = linguistic_features(parts[0])
    data[o + '_2'] = linguistic_features(parts[1])
    data[o + '_3'] = linguistic_features(parts[2])
    data[o + '_4'] = linguistic_features(parts[3])
    
data['article'] = linguistic_features(article_text)

df = pd.DataFrame.from_dict(data, orient='index').fillna(0.0)

There are many similarity measures but here, after min-max normalizing the features – I use cosine distance.

similarities = (
        pd.DataFrame(
        # Cosine similarity of min-max normalized df
        cosine_similarity((df - df.min()) / (df.max() - df.min())),
        index=df.index,
        columns=df.index
    )[['article']]
    .query('index != "article"')
)

ax = sns.barplot(
    y="index", x="article",
    data=(
        similarities.reset_index()
        .assign(index=lambda r: r['index'].str.split('_').str[0:-1].str.join('_'))
    ),
    order=(
        similarities.reset_index()
        .assign(index=lambda r: r['index'].str.split('_').str[0:-1].str.join('_'))
        .groupby('index').median()
        .reset_index()
        .sort_values('article')
    )['index']
)
ax.set(xlabel='Similarity to article style', ylabel='')
sns.despine()

Dropping out of this is that most similar are BetsyDeVosED, SecretaryCarson, SecAzar, SecNielsen, SecretarySonny, and VP Pence. I have no idea if this anywhere near the truth of course. As far as I’ve read, the only name of those that have been speculated is Pence.

Mapping clustered websites

Thu, 01 Feb 2018 16:39:50 +0100

TriggerScrape is a python script to for example map out the cluster of Swedish sites containing highly anti-immigrant content.

It does this by the following procedure:

Start at some entry point, with many outgoing links
Collecting all outgoing links
Randomly choosing a subsample of them and visiting them
Looking at how many trigger words are found on those links
Visiting them again by probability set by previous step
If the percentage trigger words by the number of visited links is high – use that site as next starting point and restart at (1)

It looks something like this:

In the end it produces a list such as:

domain	ratio	triggered	n_links
http://avpixlat.info	7.774193548387097	210	31
http://petterssonsblogg.se	4.835680751173709	817	213
http://gruvmor.wordpress.com	3.8	28	10
http://thoralfalfsson.webblogg.se	3.6484375	339	128
http://tobbesmedieblogg.blogspot.se	2.583333333333333	19	12
http://galnegunnarsblogg.wordpress.com	2.388888888888889	250	180
http://samnytt.se	2.193548387096774	74	62
http://imittsverige.blogspot.se	1.98	49	50

and if you give it enough time, it will map out the most of the sites in that cluster.

The script is build on top of the exellent Python library Grab, and can be found on my github if you are interested.

Using ANNs on small data – Deep Learning vs. Xgboost

Sun, 18 Jun 2017 21:39:50 +0200

Andrew Beam does a great job showing that small datasets are not off limits for current neural net methods. If you use the regularisation methods at hand – ANNs is entirely possible to use instead of classic methods.

Let’s see how this holds up on up on some benchmark datasets.

%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('ggplot')

seed = 123456
np.random.seed(seed)

Let’s start with the iris dataset that you nicely can pull with the pandas read_csv function right of the internets.

target_variable = 'species'
df = (
    pd.read_csv('https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/d546eaee765268bf2f487608c537c05e22e4b221/iris.csv')

    # Rename columns to lowercase and underscores
    .pipe(lambda d: d.rename(columns={
        k: v for k, v in zip(
            d.columns,
            [c.lower().replace(' ', '_') for c in d.columns]
        )
    }))
    # Switch categorical classes to integers
    .assign(**{target_variable: lambda r: r[target_variable].astype('category').cat.codes})
)

There’s three classes and 150 datapoints. Not really B̫͕̟̱I̜̼͈̖̫G͉ d̙͕a͇͍͕̝̟t̪̝̹̻͉̭ͅa.

df[target_variable].value_counts().sort_index().plot.bar()

We create a feature matrix X and a target y from the Pandas dataframe. And since an ANN needs the features to be normalized, let’s do some min-max-scaling before anything else.

y = df[target_variable].values
X = (
    # Drop target variable
    df.drop(target_variable, axis=1)
    # Min-max-scaling (only needed for the DL model)
    .pipe(lambda d: (d-d.min())/d.max()).fillna(0)
    .as_matrix()
)

We split into a training set and a test set.

from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split, cross_val_score

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=seed
)

Import some keras goodness (and perhaps run pip install keras first if you need it).

from keras.models import Sequential
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.layers import Dense, Activation, Dropout
from keras import optimizers

And set up a thee layer deep and 128 wide net. Nothing fancy, not even sure this would qualify as deep learning – but throw in some dropout between them to help it to not overfit.

Learning rate for the optimization method Adam might be something to tune on other datasets but here 0.001 seems to work nicely.

m = Sequential()
m.add(Dense(128, activation='relu', input_shape=(X.shape[1],)))
m.add(Dropout(0.5))
m.add(Dense(128, activation='relu'))
m.add(Dropout(0.5))
m.add(Dense(128, activation='relu'))
m.add(Dropout(0.5))
m.add(Dense(len(np.unique(y)), activation='softmax'))

m.compile(
    optimizer=optimizers.Adam(lr=0.001),
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

EarlyStopping helps us stop the training when the validation set is not improving any more – which helps us avoid overfitting. And to keep the checkpoint just before overfitting occurs, ModelCheckpoints let’s us save the best model before decline in validation set performance.

m.fit(
    # Feature matrix
    X_train,
    # Target class one-hot-encoded
    pd.get_dummies(pd.DataFrame(y_train), columns=[0]).as_matrix(),
    # Iterations to be run if not stopped by EarlyStopping
    epochs=200,
    callbacks=[
        # Stop iterations when validation loss has not improved
        EarlyStopping(monitor='val_loss', patience=25),
        # Nice for keeping the last model before overfitting occurs
        ModelCheckpoint(
            'best.model',
            monitor='val_loss',
            save_best_only=True,
            verbose=1
        )
    ],
    verbose=2,
    validation_split=0.1,
    batch_size=256,
)

# Load the best model
m.load_weights("best.model")

# Keep track of what class corresponds to what index
mapping = (
    pd.get_dummies(pd.DataFrame(y_train), columns=[0], prefix='', prefix_sep='')
    .columns.astype(int).values
)
y_test_preds = [mapping[pred] for pred in m.predict(X_test).argmax(axis=1)]

Now we can assess the performance on the test set. Below is a confusion matrix showing all predictions held up to reality.

pd.crosstab(
    pd.Series(y_test, name='Actual'),
    pd.Series(y_test_preds, name='Predicted'),
    margins=True
)

Predicted	0	1	2	All
Actual
0	18	0	0	18
1	0	16	0	16
2	0	0	16	16
All	18	16	16	50

print 'Accuracy: {0:.3f}'.format(accuracy_score(y_test, y_test_preds))

Accuracy: 1.000

Actually a perfect score. We will now do the same with an good old xgboost (conda install xgboost) with the nice sklearn api.

Finding the right hyperparameters is a task well suited for an Bayesian approach that can test the alternatives in an effective way without any gradient. GridSearch and such takes alot of time – this way we instead give it a parameter space and a “budget”. It will then in it’s most cost effective way test the hyperparameters of xgboost under those constraints.

from xgboost.sklearn import XGBClassifier

params_fixed = {
    'objective': 'binary:logistic',
    'silent': 1,
    'seed': seed,
}

# The space to search
space = {
    'max_depth': (1, 5),
    'learning_rate': (10**-4, 10**-1),
    'n_estimators': (10, 200),
    'min_child_weight': (1, 20),
    'subsample': (0, 1),
    'colsample_bytree': (0.3, 1)
}

reg = XGBClassifier(**params_fixed)

def objective(params):
    """ Wrap a cross validated inverted `accuracy` as objective func """
    reg.set_params(**{k: p for k, p in zip(space.keys(), params)})
    return 1-np.mean(cross_val_score(
        reg, X_train, y_train, cv=5, n_jobs=-1,
        scoring='accuracy')
    )

For this we use skopt (pip install scikit-optimize). I’ve given it 50 iterations to explore the hyperparameter space. Might be some more performance to squeeze out, but probably not.

from skopt import gp_minimize
res_gp = gp_minimize(objective, space.values(), n_calls=50, random_state=seed)
best_hyper_params = {k: v for k, v in zip(space.keys(), res_gp.x)}

print "Best accuracy score =", 1-res_gp.fun
print "Best parameters =", best_hyper_params

Best accuracy score = 0.96
Best parameters = {'colsample_bytree': 1.0, 'learning_rate': 0.10000000000000001, 'min_child_weight': 5, 'n_estimators': 45, 'subsample': 1, 'max_depth': 5}

from skopt.plots import plot_convergence
plot_convergence(res_gp)

Now let’s fix these hyperparameters and evaluate on the test set. This is exactly the same test set as Keras got to be clear.

params = best_hyper_params.copy()
params.update(params_fixed)

clf = XGBClassifier(**params)

clf.fit(X_train, y_train)
y_test_preds = clf.predict(X_test)

pd.crosstab(
    pd.Series(y_test, name='Actual'),
    pd.Series(y_test_preds, name='Predicted'),
    margins=True
)

Predicted	0	1	2	All
Actual
0	18	0	0	18
1	0	15	1	16
2	0	2	14	16
All	18	17	15	50

print 'Accuracy: {0:.3f}'.format(accuracy_score(y_test, y_test_preds))

Accuracy: 0.940

The deep (?) net got all datapoints right while xgboost missed three of them. On the other hand if you change the seed and rerun the code it might as well be xgboost coming up on top so I wouldn’t read to much into it.

Let’s generalize the code above so that we can plug in any dataset of choosing and see if this holds for harder problems. While we’re at it, I’ve added a boostrap on the accuracy statistic to give a feel about the uncertanty. I’ve put the code in this gist since it’s more or less just a repetition of the code above.

Telecom churn dataset (n=2325)

compare_on_dataset(
    'https://community.watsonanalytics.com/wp-content/uploads/2015/03/WA_Fn-UseC_-Telco-Customer-Churn.csv?cm_mc_uid=06267660176214972094054&cm_mc_sid_50200000=1497209405&cm_mc_sid_52640000=1497209405',
    target_variable='churn',
    lr=0.0005,
    patience=5
)

ANN

Predicted	0	1	All
Actual
0	1478	270	1748
1	218	359	577
All	1696	629	2325

Accuracy: 0.790
Boostrapped accuracy 95 % interval 0.770223752151 0.809810671256

Xgboost

Predicted	0	1	All
Actual
0	1563	185	1748
1	265	312	577
All	1828	497	2325

Accuracy: 0.806
Boostrapped accuracy 95 % interval 0.78743545611 - 0.825301204819

Churn is a bit harder problem, but both methods do well though.

Three class wine dataset (n=59)

compare_on_dataset(
    'https://gist.githubusercontent.com/tijptjik/9408623/raw/b237fa5848349a14a14e5d4107dc7897c21951f5/wine.csv',
    target_variable='wine',
    lr=0.001
)

ANN

Predicted	0	1	2	All
Actual
0	18	1	0	19
1	0	19	0	19
2	0	0	21	21
All	18	20	21	59

Accuracy: 0.983
Boostrapped accuracy 95 % interval 0.931034482759 1.0

Xgboost

Predicted	0	1	2	All
Actual
0	19	0	0	19
1	0	19	0	19
2	0	0	21	21
All	19	19	21	59

Accuracy: 1.000
Boostrapped accuracy 95 % interval 1.0 1.0

A very easy dataset that both methods have no problem with. Since the sample size is so small the boostrap will not be much of use to us here.

German Credit Data (n=1000)

compare_on_dataset(
    'https://onlinecourses.science.psu.edu/stat857/sites/onlinecourses.science.psu.edu.stat857/files/german_credit.csv',
    target_variable='creditability',
    lr=0.001,
    patience=5
)

ANN

Predicted	0	1	All
Actual
0	43	45	88
1	28	214	242
All	71	259	330

Accuracy: 0.779
Boostrapped accuracy 95 % interval 0.727272727273 0.830303030303

Xgboost

Predicted	0	1	All
Actual
0	37	51	88
1	25	217	242
All	62	268	330

Accuracy: 0.770
Boostrapped accuracy 95 % interval 0.715151515152 0.824242424242

So sometimes the ANN comes out on top, and sometime it’s xgboost. I think it’s fair to say that ANNs, controlled for overfitting/overtraining works kinda great even small data. At least pair with xgboost.

And the necessary hyperparameter tuning of xgboost is a pain since it really takes time. On these datasets, training the ANN takes no time at all. So let’s see if ANNs will start to eat small data also anytime soon.

Gender bias in language – analysis of bigrams in a news text corpus

Tue, 02 May 2017 01:39:50 +0200

Julia Silge recently wrote a blog post about co-occurances of words together with gendered pronouns. This made me dig up some old code that did the same, but with the difference that it also extends to gendered names besides pronouns. The data is a corpus of Swedish news texts, and I’ve used name statistics from Sweden Statistics (SCB) to parse out what names are male and female.

So let’s hack up a naive bookkeeping code to count all bigrams containing he/she/male_name/female_name. My corpus has already replaced all names with female_name/male_name since that’s what my gender monitor does. I’ll use textacy for tokenization that uses the really fast spaCy behind the hoods. That said, this still takes some time to run. I did start to redo it in dask, but I realized that would take more time than to just let it run (see appropiate xkcd).

# For my unorthodox csv
import sys, csv
csv.field_size_limit(sys.maxsize)

# A counter for female and male connections
from collections import Counter
cnt = {'he': Counter(), 'she': Counter()}

import textacy
for d in textacy.fileio.read.read_csv('news_corpus.csv'):

    # Extract tokens
    doc = textacy.Doc(d[0].lower(), lang=u'sv')

    # Extract bigrams
    ngrams = textacy.extract.ngrams(
        doc, n=2,
        filter_stops=False,
        filter_punct=False,
        filter_nums=False)

    # Bookkeeping for all bigram findings
    for gram in ngrams:

        t0 = str(gram[0]) # First token in bigram
        t1 = str(gram[1]) # Second token in bigram

        if t0 == 'han':
            cnt['he'][t1] += 1
        elif t1 == 'han':
            cnt['he'][t0] += 1

        elif str(t0) == 'male_name':
            cnt['he'][t1] += 1
        elif str(t1) == 'male_name':
            cnt['he'][t0] += 1

        elif str(t0) == 'hon':
            cnt['she'][t1] += 1
        elif str(t1) == 'hon':
            cnt['she'][t0] += 1

        elif str(t0) == 'female_name':
            cnt['she'][t1] += 1
        elif str(t1) == 'female_name':
            cnt['she'][t0] += 1

Now cnt['he'] and cnt['she'] holds the frequencies for co-occurances of he and vice versa for she. But we want it to be in percent so we can compare. About 75 % of all mentions in news media are of males, so we need to remove this bias to answer the question “How more often is word X used together with he rather than she”.

So let’s transform to percent and throw away some of the long tail to remove noise (keeping everything under percentile 95).

import pandas as pd
import numpy as np

he = (
    pd.DataFrame()
    .from_dict(cnt['he'], orient='index')
    .rename(columns={0: 'he'})
    .pipe(lambda d: d[d.he > d.quantile(.95)['he']])
    .pipe(lambda d: d/d.sum())
    .sort_values('he', ascending=False)
)

she = (
    pd.DataFrame()
    .from_dict(cnt['she'], orient='index')
    .rename(columns={0: 'she'})
    .pipe(lambda d: d[d.she > d.quantile(.95)['she']])
    .pipe(lambda d: d/d.sum())
    .sort_values('she', ascending=False)
)

For the male connected words that’s,

he.head(5)

	he
.	0.487862
har	0.066698
är	0.054389
var	0.033257
–	0.020974

and for female connected words,

she.head(5)

	she
.	0.499746
har	0.073374
är	0.061925
var	0.027128
>	0.019119

holding what we are interested in. Except the punctuation, and also some emails.

odds = (
    # Joining
    he.merge(she, left_index=True, right_index=True)
    # Calc logodds
    .assign(logodds=lambda r: np.log2(r['she'] / r['he']))
    # Removing punctuation
    .pipe(lambda d: d[d.index.str.len() > 1])
    # Removing emails
    .pipe(lambda d: d[~d.index.str.contains('@')])
    .pipe(lambda d: d[~d.index.str.contains('kundservice')])
    .sort_values('logodds', ascending=True)
)

Now let’s compare the ratios to find out what words are more used in connection to she and female names. We do this by joining/merging the he and she DataFrames. Taking the log of this ratio conviniently centers the bias around zero – positive meaning skewed towards females and negative towards male.

Most male connected

odds.query('logodds < 0').head(50)

	he	she	logodds
index
could	0.002774	0.001636	-0.761978
speaks	0.002140	0.001354	-0.660601
argues	0.003989	0.002594	-0.620557
explains	0.002985	0.002030	-0.555968
said	0.004411	0.003046	-0.534531
adds	0.002325	0.001636	-0.507164
wanted	0.004015	0.002876	-0.481216
gave	0.001691	0.001354	-0.320751
knows	0.002087	0.001692	-0.302604
continues	0.003408	0.002764	-0.302231
was	0.033257	0.027128	-0.293883
says	0.005442	0.004455	-0.288434
saying	0.017143	0.014100	-0.282004
saw	0.001875	0.001579	-0.248106
can	0.005521	0.004681	-0.238034
wrote	0.003091	0.002651	-0.221490
–	0.020974	0.018668	-0.168022
mentions	0.002008	0.001805	-0.153641
started	0.001796	0.001636	-0.135196
considers	0.004675	0.004286	-0.125392
had	0.016668	0.015961	-0.062552
sees	0.005943	0.005753	-0.047070
belives	0.006102	0.006035	-0.015996
wants	0.011728	0.011618	-0.013629

And positive gives skew towards females.

Most female connected

odds.query('logodds > 0').tail(50)

	he	she	logodds
index
takes	0.003249	0.003384	0.058662
come	0.003830	0.004004	0.064124
calls	0.001453	0.001523	0.067814
comes	0.004253	0.004512	0.085297
did	0.002298	0.002482	0.110774
means	0.012019	0.013141	0.128750
ska	0.003698	0.004061	0.134928
has	0.066698	0.073374	0.137624
notes	0.003117	0.003440	0.142380
goes	0.001664	0.001861	0.161400
pekar	0.003196	0.003609	0.175423
must	0.001796	0.002030	0.176748
became	0.007370	0.008347	0.179618
is	0.054389	0.061925	0.187220
holds	0.001189	0.001354	0.187396
got	0.006551	0.007896	0.269373
tells	0.010170	0.012351	0.280359
would	0.003328	0.004061	0.286931
took	0.003011	0.003722	0.305790
gets	0.004042	0.005019	0.312632
shows	0.001902	0.002369	0.316679
went	0.002589	0.003271	0.337557
”	0.013815	0.018047	0.385547
write	0.005415	0.007163	0.403491
does	0.002853	0.003779	0.405488
thinks	0.004834	0.006429	0.411476
as	0.002430	0.003271	0.428705
dies	0.001030	0.001410	0.452740
describes	0.003698	0.005076	0.456856
told	0.001294	0.001805	0.479576
born	0.001479	0.002087	0.496385
laughs	0.000898	0.001297	0.530385
lets	0.000925	0.001354	0.549966
feels	0.001347	0.001974	0.551144
and	0.003381	0.005132	0.602081
sits	0.001875	0.002876	0.616964
gives	0.001691	0.002764	0.708996
asked	0.000766	0.001297	0.759867
remembers	0.000819	0.001410	0.783946
lifts	0.001242	0.002200	0.825100
lives	0.001294	0.002425	0.905841
becomes	0.001426	0.004230	1.568217

Pick out the most skewed words for plotting

headsntails = pd.concat([
    odds.head(30).query('logodds < 0'),
    odds.tail(30).query('logodds > 0')
], axis=0)

# Some styling
%matplotlib inline
%config InlineBackend.figure_formats = {'png', 'retina'}
import seaborn as sns
import matplotlib.pyplot as plt
sns.set_palette("Paired", 15, .75)
sns.set_context("notebook", font_scale=1.2, rc={"lines.linewidth": 1.2})
sns.set_style("whitegrid")
custom_style = {
            'grid.color': '0.7',
            'grid.linestyle': '--',
            'grid.linewidth': 0.5,
}
sns.set_style(custom_style)

f, ax = plt.subplots(figsize=(8, 12))
x = [s.decode('utf-8') for s in headsntails['logodds'].index]
y = headsntails['logodds'].values

sns.barplot(y, x, palette="RdBu_r", ax=ax)

labels = [
    '{}x'.format(round(2**item, 1))
    for item in ax.get_xticks()
]
ax.set_xticklabels(labels)
ax.set_xlabel("Female usage")

for bar in ax.patches:
    smaller = 0.3
    height = bar.get_height()
    bar.set_height(height*smaller)
    move = height*(1-smaller)
    y = bar.get_y()
    bar.set_y(y+move)

# Finalize the plot
sns.despine(offset=5)

So men speaks, explains, says, thinks, [speaking punctuation] and statues. While women tells, describes, remembers, feels, live, recieves and sits. It’s clear that language is a mirror of what kind of values our society holds. Men are in news texts, other than just in real numbers more present – more in focus. More active and in the role as an expert. SAD!

Engagement as a KPI in journalism

Sat, 14 Jan 2017 00:39:50 +0100

How media houses measure what is successful journalism and what is not, is central in what kind if journalism we get tomorrow.

If you measure clicks, you get clickbait. If you measure money, you get content promising something it can’t hold.

So now everybody is talking about engagement, but what is it?

Is engagement how long a visitor is engaging with your content? Well, maybe – but short content can be engaging but will never see long time spent.
Is it how much social interaction an article is getting? It might as well be negative interaction. Or worse, shares done to build personal brand without even reading the content.
Is it what percentage of an article the users have reached (i.e. scroll depth)? Scrolling quickly though an article should probably not count as engaging.

I’d say engagement is perceived value. And I propose that it might be calculated from how much better than expected an article is performing. Wait, what? Let’s look at a graph.

Here’s every article published last year by Swedish news site Dagens Nyheter:

Some articles are short and users tend to read most of them. Because they are short. But with long articles, the median active time spent is seldom close to times near what you’d expect if people read approx. 250 words per minute.

That’s fine. People don’t have time to engage in long content. But if the content is engaging more than you’d expect for a 4 page in-depth-whatever. That’s probably a sign of engaging content.

If you like math. That’s,

$ E = \frac{activetime}{f(w)} $

Where $E$ is engagement, $f(w)$ is the function of expected active time given the length $w$ words (red line).

In words it roughly answers: Given X words, how much better than expected did an article perform? Which is simply how far above the red line an article positions itself.

So give your journalists the feedback they deserve: “Your content did (relatively) great, even though it’s long and filled with the complexities of the world”.

If we do that, we can value great content and steer away from shortsighted measures that’s threatening the trust in journalism.

Ok, so since you got this far. Let me expand on the above. Words are actually perhaps not a perfect measure on article length. We’ve seen that some articles have long infographics not so easily counted as words. Or just a lot of images. What we could do then is actually to scrape our content for article height in pixels. This is fairer to content of different forms than just words.

Speaking of different forms. Engagement in videos is obviously a topic for another time.

Plotting time series in Python with labels aligned to data

Mon, 21 Nov 2016 00:39:50 +0100

Randal Olson did a nice example of how to instead of a classic legend, put every label by their corresponding line. This is a nice technique when there are many labels. Having to always look at the legend makes interpretation hard.

Decided to extend upon his work making the alignment of the labels automatic, without them overlaping – and by leveraging Seaborn and Pandas for the plotting. Putting it here for future use and for others.

Results will be something like this:

Now easy to mentally connect line with label

%matplotlib inline
%config InlineBackend.figure_formats = {'png', 'retina'}
import matplotlib.pyplot as plt  
import seaborn as sns
import pandas as pd  

# I like my plots on a white background
sns.set_context("notebook", font_scale=1.2, rc={"lines.linewidth": 1.2})
sns.set_style("whitegrid")
custom_style = {
            'grid.color': '0.8',
            'grid.linestyle': '--',
            'grid.linewidth': 0.5,
}
sns.set_style(custom_style)

df = pd.read_csv("https://raw.githubusercontent.com/maxberggren/legend-right/master/percent-bachelors-degrees-women-usa.csv")

Here’s my function taking care of aligning the text to the lines.

def legend_positions(df, y):
    """ Calculate position of labels to the right in plot... """
    positions = {}
    for column in y:    
        positions[column] = df[column].values[-1] - 0.5    

    def push():
        """
        ...by puting them to the last y value and
        pushing until no overlap
        """
        collisions = 0
        for column1, value1 in positions.iteritems():
            for column2, value2 in positions.iteritems():
                if column1 != column2:
                    dist = abs(value1-value2)
                    if dist < 2.5:
                        collisions += 1
                        if value1 < value2:
                            positions[column1] -= .1
                            positions[column2] += .1
                        else:
                            positions[column1] += .1
                            positions[column2] -= .1
                        return True
    while True:
        pushed = push()
        if not pushed:
            break

    return positions

And we are ready to plot!

x = 'Year'
y = ['Health Professions', 'Public Administration', 'Education',
     'Psychology', 'Foreign Languages', 'English',    
     'Art and Performance', 'Biology', 'Agriculture',    
     'Social Sciences and History', 'Business', 'Math and Statistics',    
     'Architecture', 'Computer Science', 'Engineering']  
positions = legend_positions(df, y)

f, ax = plt.subplots(figsize=(8,11))        
cmap = plt.cm.get_cmap('Paired', len(y))

for i, (column, position) in enumerate(positions.items()):

    # Get a color
    color = cmap(float(i)/len(positions))
    # Plot each line separatly so we can be explicit about color
    ax = df.plot(x=x, y=column, legend=False, ax=ax, color=color)

    # Add the text to the right
    plt.text(
        df[x][df[column].last_valid_index()] + 0.5,
        position, column, fontsize=12,
        color=color # Same color as line
    )
ax.set_ylabel('Female bachelor degrees')
# Add percent signs
ax.set_yticklabels(['{:3.0f}%'.format(x) for x in ax.get_yticks()])
sns.despine()

UPDATE: Thanks /u/pybokeh for fix allowing for varying length of columns and for tip on Python 3.X compability.

Polling is failing us: How to collect less biased data with computational methods

Tue, 15 Nov 2016 21:39:50 +0100

In the aftermath of the US elections it is clear that polling has it’s flaws. Pollsters are good at compensating for things like underrepresented minorities, picking fair samples and with the popularization of polls of polls — even aggregating all known data to get a bigger picture. Thanks to the latter, most people now even can relate to probability of winning instead of share of votes.

This is great. But one big bias in polling makes all this skewed. The fact that there is a established moral high ground. Voting against that is something that might have social implications. So a good proportion of people prompted in a poll will therefore not give a truthful answer.

This is clear from both the Swedish election 2014, Brexit and now also the 2016 US presidential election. The majority of polls underestimated the less socially favorable alternative.

Referendum	Polls-of-polls ¹	Outcome	Abs. diff.	Rel. diff.
Sweden 2014	10 % Sweden Democrats²	~13 %	+3 %	+30 %
Brexit	~49 % leave³	~52 %	+3 %	+6 %
US 2016	~44 % Trump⁴	~56 %	+12 %	+27 %

¹Normalized for decided voters. ²Boten Ada. ³Financial times. ⁴Electoral votes 938.com.

So how do you get people to answer truthfully? One classic method is a Randomized Response Model. In it’s most simple form (two outcomes of the question, like for example the US presidential election) the respondent is asked to

First toss a coin.
If heads: tell what presidential candidate she will vote for
If tail
- Toss coin once again and say Clinton if heads and Trump if tail.

This gives the respondee the possibility to speak freely since the surveyor will not know if it’s an answer governed by chance or a truthful answer.

But what about the data? We can in fact still model the underlying percentage before the coin flips. We need more data since we are introducing noise. But it might be a small price to pay for unbiased data.

Let’s look at the different outcomes for this model.

From this graph we can make an expression for the probability of getting a Trump answer. If $P_T$ is the true proportion of that will vote for Trump, $T_O$ will be observed Trump answers.

$T_O = 0.5*P_T + 0.5*0.5$

Just for clarity sake, the corresponding expression for proportion observed Clinton votes is:

$C_O = 0.5*(1-P_T) + 0.5*0.5$

Now let’s create a Bayesian Markov chain Monte Carlo model out of this. First let’s generate 1000 true votes.

import pymc3 as pm
import numpy as np

N = 1000
true_p = 0.6
true_votes = np.random.choice([0, 1], size=(N,), p=[1-true_p, true_p])

The true votes is just an array looking something like [1 0 1 0 1 1 1 ... 0 0 1 0 1 0] with N entries. Given big enough N the percentage ones would be close to 60 % since that’s the percentage we choose for generating the random data.

Now starting from the true votes, we skew them in the same way as our algorithm would.

skewed_answers = true_votes.copy()
for i, vote in enumerate(skewed_answers):
    # If coin comes up heads - change answer
    if np.random.choice([0, 1], p=[.5, .5]) == 1:
        # Change to random flip of second coin
        skewed_answers[i] = np.random.choice([0, 1], p=[.5, .5])

Obviously we would not know the true votes in a real life situation, only the skewed_answers that’s now been generated.

Now that we have synthetic data let’s build a model that infers the true votes. This is heavily inspired by chapter 2 in Cam Davidson-Pilon’s great book Bayesian Methods for Hackers.

It works in a way that we construct a model that could have generated the data. And then gives it the data and asks questions about the variables in the model.

In this case $P_T$, the proportion of Trump voters is the variable we want to know about – but we create a model that given an (yet) unknown $P_T$ could generate the skewed data that we got. It follows the logic we just derived expressing the proportion observed Trump votes.

Now to the model

%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns

with pm.Model() as model:

    # Proportion Trump defined as between 0 and 1
    P_T = pm.Uniform("prop_trump_votes", 0, 1)

    # Observed proportion follows this expression
    P_O = pm.Deterministic(
        "prop_skewed",
        0.5 * P_T + 0.25)

    # A Binomial generates ones and zeros
    T_O = pm.Binomial(
        "number_trump_votes", N,
        P_O, # by the observed probability
        observed=sum(skewed_answers))
    # and we also gave it the observed skewed votes
    # so it know what to optimize for.

    # Now we run the inference
    step = pm.Metropolis(vars=[P_T])
    trace = pm.sample(40000, step=step)
    burned_trace = trace[2500:]

# Visualizing the distribution of the true proportion P_T
sns.distplot(
    burned_trace["prop_trump_votes"],
    bins=30, label='Posterior distribution')

sns.plt.xlim(0, 1)
sns.plt.legend()

 [-----------------100%-----------------] 40000 of 40000 complete in 3.2 sec

As you can see it’s not a clear answer saying 60 % as in true_p like the data we generated. We get a distribution of where the true proportion Trump voters $P_T$ lies because

our method/algorithm introduced noise in the data
and since there’s always a chance that the data would come up just by the random nature of the universe.

But from the graph, we can be quite confident that a larger proportion of the voters will choose Trump even with respecting the privacy of the respondents.

There are of course some problems with this method. Respondees could perhaps not understand what they are supposed to do in the flipping of coins here and there, and you need to collect more responses to battle the noise. But since it’s obvious that polling does not measure peoples true intentions in the case of a controversial alternative, data people needs to start thinking about alternative methods in understanding people.

Investigation of voting patterns demographic correlations in Sweden

Mon, 15 Aug 2016 21:39:50 +0200

Sweden have a long tradition of keeping statistics of different KPI:s by municipality through many institutions. Let’s see what KPI:s correlate with different voting patterns. And while we’re at it, demonstrate some Pandas method chaining – and do a Bayesian analysis of the Pearson product-moment correlation coefficient often denoted as Persons r.

Note that the conclusions might mainly be of intrest of Swedes, and there will probably slip in some Swedish. Quick wordlist: kommun = municipality, brott = crime, gravid = pregnant.

TLDR; Sweden Democrats (SD) shows high correlation with mothers smoking while pregnant, unemployment and motorcycles. The results are a the bottom of the post if you’re not interested in the process.

And as usual, the Jupyter Notebook (former Ipython notebook) is here in a repo together with the data.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
%matplotlib inline

from matplotlib import rcParams
rcParams['font.family'] = 'monospace'
rcParams['font.sans-serif'] = ['Courier']

Starting with the data from the Swedish election 2014, we can read it directly from an url with the very nice Pandas method read_html.

Encapsulating the DataFrame votes with parentheseses makes method chaining possible. Every method in fact returns a DataFrame which makes every preprocessing step just another step in the chain.

Percent votes in Swedish election 2014

votes = (pd.read_html('http://www.val.se/val/val2014/slutresultat/K/rike/index.html')[1]
           .rename(columns={u'Område': 'Kommun'})
           [['Kommun', 'SD', 'M', 'C', 'FP', 'KD', 'S', 'V', 'MP', 'FI', u'ÖVR', 'BLANK']]
           .set_index('Kommun')
           .apply(lambda x: x.str.replace(",", ".")
                             .str.replace("%", "").astype(float))
)
partier = votes.columns
votes.head(3)

	SD	M	C	FP	KD	S	V	MP	FI	ÖVR	BLANK
Kommun
Ale	10.64	21.08	4.95	3.28	3.50	33.59	6.25	5.70	0.18	10.83	1.39
Alingsås	7.48	21.58	6.17	10.30	6.44	29.19	6.85	10.81	0.30	0.89	1.21
Alvesta	14.44	19.70	14.89	1.74	3.12	28.77	4.08	3.49	0.20	9.56	1.98

Now let’s read some migration patterns straight of the web. I.e. what municipalitys that had a positive net growth in terms of people living there.

Net inhabitant growth

url = 'http://www.ekonomifakta.se/Fakta/Regional-statistik/Din-kommun-i-siffror/Nyckeltal-for-regioner/?var=17248'
col = u'Befolkningsökning, 2012-2015, procent, treårsbasis'
befolkningsokning = (
    pd.read_html(url, thousands=' ')[0]
      .assign(**{col: lambda x: x[col].str.replace(u",", u".").astype(float)})
      .set_index('Kommun')
)
befolkningsokning.head(3)

	Befolkningsökning, 2012-2015, procent, treårsbasis
Kommun
Ale	3.7
Alingsås	3.3
Alvesta	2.9

Rating of how good a municipality is for companies

url = 'http://www.ekonomifakta.se/Fakta/Regional-statistik/Din-kommun-i-siffror/Nyckeltal-for-regioner/?var=17257'
col = u'Företagsklimat, ranking 1-290, 2015, ordningstal'
foretagsklimat = (
    pd.read_html(url, thousands=' ')[0]
      .assign(**{col: lambda x: x[col].astype(int)})
      .set_index('Kommun')
)
foretagsklimat.head(3)

	Företagsklimat, ranking 1-290, 2015, ordningstal
Kommun
Ale	43
Alingsås	185
Alvesta	221

Percent people with a higher education

url = 'http://www.ekonomifakta.se/Fakta/Regional-statistik/Din-kommun-i-siffror/Nyckeltal-for-regioner/?var=17251'
col = u'Andel högskoleutbildade, 2015, procent'
hogskoleutbildning = (
    pd.read_html(url, thousands=' ')[0]
      .assign(**{col: lambda x: x[col].str.replace(u",", u".").astype(float)})
      .set_index('Kommun')
)
hogskoleutbildning.head(3)

	Andel högskoleutbildade, 2015, procent
Kommun
Ale	18.9
Alingsås	24.3
Alvesta	15.0

Percent early retirees

url = 'http://www.ekonomifakta.se/Fakta/Regional-statistik/Din-kommun-i-siffror/Nyckeltal-for-regioner/?var=17256'
col = u'Andel förtidspensionärer, 2015, procent'
fortidspensionarer = (
    pd.read_html(url, thousands=' ')[0]
      .assign(**{col: lambda x: x[col].str.replace(u",", u".").astype(float)})
      .set_index('Kommun')
)
fortidspensionarer.head(3)

	Andel förtidspensionärer, 2015, procent
Kommun
Ale	5.5
Alingsås	6.5
Alvesta	6.0

Median income

url = 'http://www.ekonomifakta.se/Fakta/Regional-statistik/Din-kommun-i-siffror/Nyckeltal-for-regioner/?var=17249'
col = u'Medianinkomst, 2014, kronor'
medianinkomst = (
    pd.read_html(url, thousands=' ')[0]
      .assign(**{col: lambda x: x[col].str.replace(u"\xa0", u"").astype(int)})
      .set_index('Kommun')
)
medianinkomst.head(3)

	Medianinkomst, 2014, kronor
Kommun
Ale	273684
Alingsås	259211
Alvesta	241077

Unemployment rate

url = 'http://www.ekonomifakta.se/Fakta/Regional-statistik/Din-kommun-i-siffror/Nyckeltal-for-regioner/?var=17255'
col = u'Öppen arbetslöshet (Arbetsförmedlingen), 2015, procent'
arbetsloshet = (
    pd.read_html(url, thousands=' ')[0]
      .assign(**{col: lambda x: x[col].str.replace(u",", u".").astype(float)})
      .set_index('Kommun')
)
arbetsloshet.head(3)

	Öppen arbetslöshet (Arbetsförmedlingen), 2015, procent
Kommun
Ale	4.9
Alingsås	5.5
Alvesta	9.3

Number of crimes per 100k inhabitants

brott = (pd.read_csv("brott_per_1000.txt", sep=";", encoding="latin-1")
        .rename(columns={"Region": "Kommun", "/100 000 inv": "Brott per 100k inv"})
        .groupby("Kommun")
        .sum()
        .reset_index()
        [['Kommun', 'Brott per 100k inv']]
        .assign(Kommun= lambda x: x['Kommun'].str.replace(" kommun", ""))
        .set_index('Kommun'))
brott.head(3)

	Brott per 100k inv
Kommun
Ale	8231
Alingsås	10511
Alvesta	8300

Violent crimes

valdsbrott = (pd.read_csv("vbrott_per_1000.txt", sep=";", encoding="latin-1")
        .rename(columns={"Region": "Kommun", "/100 000 inv": u"Våldsbrott per 100k inv"})
        .groupby("Kommun")
        .sum()
        .reset_index()
        [['Kommun', u'Våldsbrott per 100k inv']]
        .assign(Kommun= lambda x: x['Kommun'].str.replace(" kommun", ""))
        .set_index('Kommun'))
valdsbrott.head(3)

	Våldsbrott per 100k inv
Kommun
Ale	823
Alingsås	870
Alvesta	967

Drug crimes

knarkbrott = (pd.read_csv("knarkbrott_per_1000.txt", sep=";", encoding="latin-1")
        .rename(columns={"Region": "Kommun", "/100 000 inv": u"Narkotikabrott per 100k inv"})
        .groupby("Kommun")
        .sum()
        .reset_index()
        [['Kommun', u'Narkotikabrott per 100k inv']]
        .assign(Kommun= lambda x: x['Kommun'].str.replace(" kommun", ""))
        .set_index('Kommun'))
knarkbrott.head(3)

	Narkotikabrott per 100k inv
Kommun
Ale	402
Alingsås	1487
Alvesta	327

Mean age

url = 'http://www.ekonomifakta.se/Fakta/Regional-statistik/Din-kommun-i-siffror/Nyckeltal-for-regioner/?var=17247'
col = u'Medelålder, 2015, år'
medianalder = (
    pd.read_html(url, thousands=' ')[0]
      .assign(**{col: lambda x: x[col].str.replace(u",", u".").astype(float)})
      .set_index('Kommun')
)
medianalder.head(3)

	Medelålder, 2015, år
Kommun
Ale	39.9
Alingsås	42.0
Alvesta	42.1

Percent born in another country than Sweden

utrikesfodda = (pd.read_excel("BE0101-Utrikes-fodda-kom-fland.xlsx", header=5)
        .dropna(subset=[u'Kommun'])
        .assign(procent= lambda x: x[u'I procent'].astype(float))
        .rename(columns={'procent': u'Procent utrikesfödda'})
        [['Kommun', u'Procent utrikesfödda']]
        .set_index('Kommun')
)
utrikesfodda.head(3)

	Procent utrikesfödda
Kommun
Botkyrka	40.1
Danderyd	15.3
Ekerö	11.0

Percent mothers smoking while pregnant

rokande_grav = (
    pd.read_excel("Lev03.xlsx", header=3)
      .reset_index()
      .rename(columns={'index': 'Kommun', '2014': u'Rökande gravida, procent'})
      .assign(Kommun=lambda x: x['Kommun'].str.split())
      .assign(Kommun=lambda x: x['Kommun'].str[1:])
      .assign(Kommun=lambda x: x['Kommun'].str.join(" "))
      .dropna()
      .set_index('Kommun')
      .replace("..", 0)
      .assign(**{u'Rökande gravida, procent': lambda x: x[u'Rökande gravida, procent'].astype(float)})
)
rokande_grav.head(3)

	Rökande gravida, procent
Kommun
Upplands Väsby	3.6
Vallentuna	2.3
Österåker	2.9

Rapes per 10k inhabitants

valdtakter = (
    pd.read_excel("Sex03.xlsx", header=3)
      .reset_index()
      .rename(columns={'index': 'Kommun', '2015': u'Våldtäkter per 10k inv'})
      .assign(Kommun=lambda x: x['Kommun'].str.split())
      .assign(Kommun=lambda x: x['Kommun'].str[1:])
      .assign(Kommun=lambda x: x['Kommun'].str.join(" "))
      .dropna()
      .set_index('Kommun')
      .assign(**{u'Våldtäkter per 10k inv': lambda x: x[u'Våldtäkter per 10k inv'].astype(float)})
)
valdtakter.head(3)

	Våldtäkter per 10k inv
Kommun
Upplands Väsby	5.0
Vallentuna	1.9
Österåker	2.2

Asylum seekers per 1k inhabitants

asylsokande = (pd.read_excel("asylsdochm.xlsx", header=1)
        .dropna(subset=[u'Kommun'])
        .rename(columns={u'Antal asylsökande per': u'Antal asylsökande per 1000 invånare'})
        [['Kommun', u'Antal asylsökande per 1000 invånare']]
        .set_index('Kommun')
        .assign(**{u'Antal asylsökande per 1000 invånare': lambda x: x[u'Antal asylsökande per 1000 invånare'].astype(float)})

)
asylsokande.head(3)

	Antal asylsökande per 1000 invånare
Kommun
Ljusnarsberg	138.26
Norberg	113.94
Hultsfred	78.33

Types of vehicles

fordon = (pd.read_excel("Fordon_lan_och_kommuner_2015.xlsx", header=5, sheetname=2)
        .dropna(subset=[u'Kommun'])
        .set_index('Kommun')
        .drop(['Kommun-', 'Unnamed: 4', 'Unnamed: 5', 'Unnamed: 6'], axis=1)
        .apply(lambda x: x.astype(float))
        .apply(lambda x: x/sum(x), axis=1) # Convert to percentages
        .dropna()
        .rename(index=lambda x: x.strip())
        .rename(columns=lambda x: x.strip())
        .rename(columns=lambda x: x + ", 2015, procent")

)
fordon.head(3)

	Personbilar, 2015, procent	Lastbilar, 2015, procent	Bussar, 2015, procent	Motorcyklar, 2015, procent	Mopeder, 2015, procent	Traktorer, 2015, procent	Snöskotrar, 2015, procent	Terränghjulingar, 2015, procent	Terrängskotrar1), 2015, procent	Släpvagnar, 2015, procent
Kommun
Upplands Väsby	0.757944	0.069553	0.000088	0.042546	0.009148	0.007748	0.006128	0.005077	0.000175	0.101593
Vallentuna	0.672324	0.066290	0.000000	0.051673	0.009153	0.029813	0.008131	0.021327	0.000489	0.140801
Österåker	0.692083	0.069446	0.000252	0.047521	0.013716	0.015516	0.009396	0.010332	0.000252	0.141484

Since we now have every KPI as a DataFrame, let’s put them together with concat.

concat = pd.concat([
        votes,
        utrikesfodda,
        medianalder, 
        brott, 
        medianinkomst, 
        arbetsloshet, 
        fortidspensionarer, 
        hogskoleutbildning, 
        befolkningsokning,
        foretagsklimat, 
        valdsbrott,
        knarkbrott,
        asylsokande,
        fordon,
        valdtakter,
        rokande_grav,
    ], 
    axis=1,
    join='inner')
concat.head(5)

	SD	M	C	FP	KD	S	V	MP	FI	ÖVR	...	Bussar, 2015, procent	Motorcyklar, 2015, procent	Mopeder, 2015, procent	Traktorer, 2015, procent	Snöskotrar, 2015, procent	Terränghjulingar, 2015, procent	Terrängskotrar1), 2015, procent	Släpvagnar, 2015, procent	Våldtäkter per 10k inv	Rökande gravida, procent
Kommun
Ale	10.64	21.08	4.95	3.28	3.50	33.59	6.25	5.70	0.18	10.83	...	0.000141	0.054257	0.016427	0.039285	0.002159	0.009058	0.000188	0.150005	2.1	4.4
Alingsås	7.48	21.58	6.17	10.30	6.44	29.19	6.85	10.81	0.30	0.89	...	0.000068	0.052143	0.009086	0.053940	0.001729	0.014883	0.000203	0.154902	4.8	3.8
Alvesta	14.44	19.70	14.89	1.74	3.12	28.77	4.08	3.49	0.20	9.56	...	0.002950	0.042747	0.014082	0.082378	0.000946	0.015418	0.000000	0.182122	2.0	4.8
Aneby	9.38	11.19	20.46	4.06	14.82	31.30	2.19	5.75	0.05	0.80	...	0.003179	0.042438	0.011197	0.119712	0.000968	0.017556	0.000000	0.199198	1.5	0.0
Arboga	9.75	20.54	5.53	7.57	4.08	39.04	5.77	5.62	0.14	1.96	...	0.001810	0.043954	0.010256	0.060502	0.006722	0.009997	0.000172	0.170818	2.2	13.2

5 rows × 35 columns

Pandas gives us the method corr that by default uses Persons r to show correlations between columns. Let’s start of by using this simple frequentistic approach.

corr_matrix = concat.corr()
corr = concat.corr()

# Keep only parties as columns
corr = corr.loc[:, corr.columns.isin(partier)] 
# Keep only KPIs as rows
corr = corr[~corr.index.isin(partier)]

And use Seaborn:s heatmap to give us a nice visual representation of the correlation table.

fig1 = plt.figure(figsize=(8, 8))
ax = sns.heatmap(
    corr, 
    annot=True, fmt='.1f', linewidths=.5
)

ax.text(-1., 1.141, 
        u'Korrelation med andel riksdagsröster',
        verticalalignment='bottom', horizontalalignment='left',
        transform=ax.transAxes,
        color='#000000', fontsize=14)

ax.text(-1., -0.181, 
        u'Källa: BRÅ/SCB/Val.se/Ekonomifakta.se – Korrelationen är på kommunnivå och Pearson-korr har använts',
        verticalalignment='bottom', horizontalalignment='left',
        transform=ax.transAxes,
        color='#666666', fontsize=8)

plt.savefig('SD_korr.pdf', bbox_inches='tight')

First of, a Persons r correlation ranges from 0 to 1. Zero meaning no correlation and one full correlation. Depending on textbook, somewhere over 0.4 will be a correlation to take seriously. But let’s get back to that in a moment.

Some interesting correlations are though:

The right-wing populist party the Sweden Democrats (SD in the figure)
- Correlations with municipailtys having large porportions motorcycles,
- and women smoking while pregnant.
- Also high unemployment rate shows some correlation.
Right wing Moderaterna (M)
- High percentage mopeds
- High percentage cars
- High education
- High income
The Greens Miljöpartiet (MP)
- Pretty much the same as Moderaterna. Only not that high income.
Centerpartiet (C) historically a party for the farmers
- High correlation with tractors

So let’s go into detail what a correlation of 0.5 actually is. Is it a fully linear correlation between smoking while pregnant and votage on xenophobic parties? Let’s plot the municipalitys with these KPI:s on the axis.

def corrplot_SD(kpi):
    (concat.replace(0, np.nan).dropna()
           .plot(kind='scatter', 
                 x=kpi, 
                 y=u'SD'))

corrplot_SD(u'Rökande gravida, procent')

If you ask me, that looks like a correlation. But that’s also what I want to see since the Sweden Democrats is pretty much as far away from my personal political opinions you can go. I’d pretty much take any chance to misscredit them. So let’s go further down the rabbit hole, but first plot the other KPIs.

corrplot_SD(u'Motorcyklar, 2015, procent')

corrplot_SD(u'Öppen arbetslöshet (Arbetsförmedlingen), 2015, procent')

We are now going to calculate the r coefficient in a Bayesian fashion with Monte Carlo Markov Chains (MCMC). That will give us a distribution over r instead of a single number which allows us to give a upper and lower bound of the correlation. This closely follows the method described by Philipp Singer here.

You should read his blog post if you want more details. And in more general, Monte Carlo simulations is a really appealing way to solve problems. Whenever I get the chance, I recommend Bayesian methods for hackers. The topics covered there should really be in any data scientist tool belt.

import pymc as pymc
from pymc import Normal, Uniform, MvNormal, Exponential
from numpy.linalg import inv, det
from numpy import log, pi, dot
import numpy as np
from scipy.special import gammaln

def _model(data):
    mu = Normal('mu', 0, 0.000001, size=2)
    sigma = Uniform('sigma', 0, 1000, size=2)
    rho = Uniform('r', -1, 1)
    
    @pymc.deterministic
    def precision(sigma=sigma, rho=rho):
        ss1 = float(sigma[0] * sigma[0])
        ss2 = float(sigma[1] * sigma[1])
        rss = float(rho * sigma[0] * sigma[1])
        return inv(np.mat([[ss1, rss], 
                           [rss, ss2]]))
    
    mult_n = MvNormal('mult_n', mu=mu, tau=precision, value=data.T, observed=True)
    
    return locals()
    
def analyze(kpi):
    freq_corr = corr.loc[kpi, 'SD']
    print "Standard Persons r correlations gave {}".format(freq_corr)
    data = np.array([concat['SD'].values, concat[kpi].values])
    model = pymc.MCMC(_model(data)) 
    model.sample(10000, 5000) 
    
    print "\nMCMC gives an lower and upper bound for this of {} and {}".format(
        *model.stats()['r']['95% HPD interval']
    )

analyze(u'Rökande gravida, procent')

Standard Persons r correlations gave 0.483367702644
 [-----------------100%-----------------] 10000 of 10000 complete in 25.1 sec
MCMC gives an lower and upper bound for this of 0.384673816867 and 0.569554905111

This tells us that the correlation is statistically sound. No p-values, just a interval that the true r should be in with 95 % probability. Even if it’s just the lower bound 0.38 it’s a correlation worth mentioning. But of course, it does not tell us if correlation imply causation.

Mapping town names atomic parts in Sweden

Thu, 14 Jul 2016 21:39:50 +0200

Many towns in Sweden share the same atomic parts in their names. -hult is common in Småland -arp is common in Skåne. I got curious of how they cluster geographically. Let’s load up a DataFrame with the positions of Swedish towns that I made by querying Google for the position of every Swedish town I could find on Wikipedia.

This notebook is also available as a Jupyter Notebook here if you want to execute the code yourself.

Scroll past the code if you’re only here for the maps.

import pandas as pd

df = pd.read_csv('swe_towns_latlon.csv', encoding='utf-8')
df['town'] = df['town'].str.lower()

And then do some crappy character n-gram extractions.

def find_features(s):
    feats = [s[i:i+7] for i in xrange(len(s)-6)] 
    feats += [s[i:i+6] for i in xrange(len(s)-5)] 
    feats += [s[i:i+5] for i in xrange(len(s)-4)] 
    feats += [s[i:i+4] for i in xrange(len(s)-3)] 
    feats += [s[i:i+3] for i in xrange(len(s)-2)] 
    return feats

features = []
for town in df['town'].values:
    features.extend(find_features(town))

To be able to count the occurances.

features = list(set(features))
final_features = []
occurances = []

for feature in features:
    try:
        occurances.append(sum(df['town'].str.contains(feature)))
        final_features.append(feature)
    except:
        pass
    
df_features = pd.DataFrame({
    'feature': final_features, 
    'occurances': occurances 
})

So now we get interesting parts of town names. Sanity check (at least if you’re Swedish) is that sta, hult and so on are present.

print ", ".join(df_features.sort('occurances', ascending=False).head(100)['feature'].values)

sta, tor, ing, erg, nge, orp,  oc,  och , och , och, ch ,  och, vik, torp, ra , ber, ors, berg, und, sjö, tra, and, str, näs, ter, inge, stra, tad, stad, den, for, fors, mar, ken, sto, olm, ste, äst, storp, stor, lla, lle, all, dal, hol, ngs, holm, äll, lst, tra , ång, ers, orr, stra , red, ran, nda, arp, ham, ill, est, ung, äck, ten, ster, gen, jör, rby, jär, nne, cke, ult, amm, mma, lin, sun, ryd, ack, sby, äng, eby, byn, len, lan, sund, ling, ker, nna, mmar, löv, lun, ike, bäck, bro, tan, nga, bäc, ård, rst, tte

I went ahead and took out the ones I deemed interesting. Now let’s plot them to se how they group geographically.

from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
from matplotlib import rcParams
rcParams['font.family'] = 'sans-serif'
rcParams['font.sans-serif'] = ['Helvetica Neue']
%matplotlib inline

def plot(data, mult=0.9):
    """ Take data with 'part-of-towns-name' and color and plot it. Use `mult` for easy scaling of plot """
    
    def get_coordinates(townpart):
        try:
            townpart = townpart.decode('utf-8')
        except:
            pass
        return df[df['town'].str.contains(townpart)]['lat'].values, df[df['town'].str.contains(townpart)]['lon'].values

    fig = plt.figure(figsize=(20*mult, 16*mult), dpi=200)

    m = Basemap(
        projection='merc',
        resolution='i', 
        area_thresh=250,
        llcrnrlon=9.5, 
        llcrnrlat=54.5,
        urcrnrlon=24.5, 
        urcrnrlat=69.5
    ) 

    m.drawcoastlines(linewidth=0, color="#000000") 
    m.drawcountries()
    m.drawstates()
    m.drawmapboundary()
    m.fillcontinents(color='black', lake_color='white', zorder=0)
    m.drawmapboundary(fill_color='white')
    title = plt.title(u'Town names containing:', fontsize=16*mult) 
    title.set_y(1.01) 

    for townpart in data.keys():
        lats, lons = get_coordinates(townpart)
        x, y = m(lons, lats)
        m.scatter(x, y, marker='o', s=30*mult, alpha=1, label=townpart, edgecolors='none', c=data[townpart])

    plt.legend(loc=2, fontsize=16*mult)
    plt.show()


plot({
    'arp': '#00ff00',
    'holm': '#ff0000',
})

plot({
    'sta': '#ff00ff',
    'ing': '#00ffff',
})

plot({
    'hult': '#ffff00',
    'fors': '#00ff00',
})

plot({
    'vik': '#4f96c5',
    'torp': '#00ff00',
})

plot({
    u'näs': '#b15928',
    'ryd': '#08a060',
})

plot({
    'tuna': '#33cc33',
    'hammar': '#ff0066',
})

plot({
    u'köping': '#cc99ff',
    u'bruk': '#ffff00',
})

plot({
    'stor': '#00ccff',
    'sund': '#e3c471',
})

plot({
    'stad': '#ffcc00',
    'berg': '#9933ff',
})

Berg is really common in the old mining regions called Bergslagen.

plot({
    u'ån': '#ffccff',
    u'ås': '#ee5657'
})

Introducing Shamebot3

Wed, 05 Aug 2015 21:39:50 +0200

How we perceive our world is surely influenced by what kind of people we let represent it. Lead roles in movies is mainly held by men (at least in the top #100 imdb movies). That sets the stage for a world where men are perceived as in the lead role. This is perhaps not news, but quantifying this inequality might be a step in the direction of correcting it.

Therefore I built a bot that:

Daily scans the major Swedish news sites for names of the form Firstname Surname.
Checks every firstname against Sweden Statistics (SCB) name database, counting every name used by more than 95 % of exclusivly one gender. Unisex names are therefore excluded.
Calculates the percentage of women represented by each site and logs it in a timeseries.
Tweets once every week and month with a graph ranking the Swedish news sites.

It looks something like this.

So basically it shames the least equal newspaper, hoping for retweets to add on to the feel of a angry online mob. In this analogy, twitter is the public shame pole of the village.

The timeseries shows some general trends already, even though the bot was broken for some time (as you can see from the graph).

Etc.se which is a left wing newspaper works activily with equal representation in their texts and it obviously reflects on the data. Most other newspapers lies in the span 20 % - 35 % women represented. Svt.se which is the Swedish public service, that is funded by tax money (black line), comes out second.

When more data is avalible I will probably revisit this again.

Code for the bot can be found on my Github and the twitter account connected with it is Shamebot3 (shamebot one and two was already taken).

MAX BERGGREN

Linguistic analysis of who wrote the op-ed

Mapping clustered websites

Using ANNs on small data – Deep Learning vs. Xgboost

Telecom churn dataset (n=2325)

ANN

Xgboost

Three class wine dataset (n=59)

ANN

Xgboost

German Credit Data (n=1000)

ANN

Xgboost

Gender bias in language – analysis of bigrams in a news text corpus

Most male connected

Most female connected

Pick out the most skewed words for plotting

Engagement as a KPI in journalism

Plotting time series in Python with labels aligned to data

Now easy to mentally connect line with label

Polling is failing us: How to collect less biased data with computational methods

1Normalized for decided voters. 2Boten Ada. 3Financial times. 4Electoral votes 938.com.

Investigation of voting patterns demographic correlations in Sweden

Percent votes in Swedish election 2014

Net inhabitant growth

Rating of how good a municipality is for companies

Percent people with a higher education

Percent early retirees

Median income

Unemployment rate

Number of crimes per 100k inhabitants

Violent crimes

Drug crimes

Mean age

Percent born in another country than Sweden

Percent mothers smoking while pregnant

Rapes per 10k inhabitants

Asylum seekers per 1k inhabitants

Types of vehicles

Mapping town names atomic parts in Sweden

Introducing Shamebot3

¹Normalized for decided voters. ²Boten Ada. ³Financial times. ⁴Electoral votes 938.com.