Basil Shkara

App Sales Machine - pulls, parses, stores and emails App Store reports

2009-09-05T00:00:00+10:00

[UPDATE 20090910] Now parsing the Top 200 rankings.

[UPDATE 20090909] App Sales Machine now automatically generates graphs using Graphy and also sends HTML formatted emails.

I recently spent a couple days writing an App Engine application using the webapp framework. App Sales Machine does the following:

Imports a tarball of your existing sales reports and stores them in the datastore.
Logs into iTunes Connect every morning, downloads your latest sales report and stores it in the datastore.
Parses the downloaded report and stores the parsed data in the datastore.
Converts the reported income revenue of your sales to a configurable currency and stores that.
Pulls the rankings for your app every 8 hours (configurable) and stores that in the datastore.
Emails a report to a select group of recipients every morning with the latest cumulative figures, as well as the last downloaded report’s figures.

It utilises the task queues API to fetch your rankings asynchronously in a staggered approach. And if the daily sales report pull times out, it will add that task to the task queue to keep trying until it is downloaded.

All of this was made possible because of the following pieces of open software:

I made some modifications to itunes-connect-scraper to use the URL Fetch API so I could increase the connection timeout to the maximum currently allowed on App Engine (10 seconds).

The data parsing and currency converting code is borrowed and modified from iTunesConnectArchiver.

I also ported the app_rank.pl Perl script to Python. It was originally written by Ben Chatelain but it seems to have been taken down, however it still lives on in other forms albeit still in Perl (read the comments in his original blog post).

The variety of scripts available to download and parse your own report data is just an indication of the poor state of tools that Apple has provided iPhone developers with which to monitor their sales. In fact an entire ecosystem has sprung up to capitalise on this void (appFigures, heartbeat, TapMetrics, Appstatz, AppViz, My App Sales).

There are some free tools (as well as the above scripts) available such as MajicRank and AppSalesMobile but nothing really comparable to the kind of reporting you get from one of the above paid tools. App Sales Machine isn’t really comparable either at this stage and I don’t know if it ever will be.

There’s nothing wrong with any of the above tools, in fact I used appFigures for a while and was quite happy with it. App Sales Machine was inspired by their functionality but I just wanted to be in control of my own data.

In its current state App Sales Machine may offer all the functionality you require. It does for me currently but there is certainly room for extending it’s current feature set. There is currently a very minimal frontend interface which consists of just a file upload field to upload your existing sales reports. It wouldn’t be hard to integrate nice charts as email attachments (e.g. Graphy) or even as part of a frontend interface.

How to install

Clone my Git repository.

git clone git://github.com/baz/app-sales-machine.git

Configure settings.py located in the root directory. I’ve commented this file thoroughly to indicate which values need to be changed to be specific to the current app(s) you have on the App Store.

# SKU of the app when you uploaded it to the App Store
"<<SKU>>": {
	# Human-readable app name which corresponds to the above SKU
	"name": "<<app name>>",

	# App ID (can be found from the GET param of your iTunes URL i.e. ?id=xxxxxxxxx)
	"app_id": "<<app id>>",

	# Make sure your category name and corresponding ID exists in jobs/app_store_codes.py
	"category_name": "<<category name>>",

	# Is your app free?
	"paid": False,

	# List of dictionaries which represent the human-readable version number and the
	# date the version was released on the App Store
	"versions": [{'name': 'v1.0', 'date': datetime.date(2009, 6, 22)},
			{'name': 'v1.1', 'date': datetime.date(2009, 7, 18)}],

	# This must be the email address of a registered administrator for the application due to
	# mail API restrictions
	"from_address": "My Name <sender@example.com>",

	# List of email addresses you would like the daily reports sent to
	"to_addresses": [
		'Recipient Name <recipient@example.com>',
	]
}

You can specify more than one product SKU. It is just another key in this dictionary structure. Note: I have not tested App Sales Machine with more than one product as I only have one app in the App Store at the moment. However I designed it to accommodate multiple products so theoretically it should all just work.
I discovered the App Store category IDs in the app_rank.pl script were wrong/had changed whilst I was porting it. As such, lib/app_store_codes.py only contains category IDs for the “Top 100” and “Lifestyle” categories (categories I am monitoring for my current app). If you use App Sales Machine, you will have to update the CATEGORIES dictionary with your app’s particular category. From Ben’s blog entry on this:
```
http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewTop?id=$categoryID&popId=xx
$categoryID is the category id your app is in
```
The only other file you may want to configure is cron.yaml which describes the schedule of events. It is fairly self-explanatory but for more info, read this.

cron:
- description: Daily report pull
  url: /jobs/pull_report
  schedule: every day 06:00
  timezone: Australia/Sydney

- description: Rankings pull
  url: /jobs/pull_rankings
  # Must be larger than 1 hour, see email_report.py
  schedule: every 8 hours
  timezone: Australia/Sydney

- description: Daily report email push
  url: /jobs/email_report
  schedule: every day 07:00
  timezone: Australia/Sydney

Note: ensure your cron pull time is a time that occurs after the previous day’s report is made available on iTunes Connect otherwise your pull report job will fail for obvious reasons.
Visit the root URL of your App Engine application and upload a tarball with your existing iTunes Connect reports if you have any.
Then the only thing left to do is modify your app.yaml file to correspond to the App Engine application name you created and upload it using the command-line tool or the Mac Launcher tool.

What you get

If all goes well, the scheduled email task should push a daily report to your inbox that looks something like this:

Hello,

Here is your daily report for <<app name>>
--

Yesterday's (2009-09-03) download figures:
       - xx

Yesterday's (2009-09-03) upgrade figures:
       - xx

Total number of downloads (2009-06-22 to 2009-09-03):
       - xxxxx

Total number of upgrades (2009-07-18 to 2009-09-03):
       - xxxxx

Upgrade rate (over base of xxxxx):
       - xx.xx%

Approximate total income revenue (AUD):
       - 0.0

Rankings (as of 2009-09-04 16:59:27.615497 UTC):

       Country                  Category                 Ranking
       -------                  --------                 -------
       New Zealand              Lifestyle                67
       Australia                Lifestyle                6

I’ve been using App Sales Machine for the past week and it’s been working beautifully. It lets me and the other stakeholders in the app know about its progress and how it’s tracking.

And because everything is stored in GAE’s datastore under your own administrator account, you can write new webapp handlers and templates to query the data and generate graphs, etc. if you wish.

The usual disclaimers apply here. Although I’ve been using it happily for a while now, I take no responsibility for incorrect reporting or loss of data that App Sales Machine causes. With that being said, I hope you find a use for it.

Using the LLVM/Clang Static Analyzer for iPhone apps

2009-01-06T00:00:00+11:00

This set of instructions will help you get started with finding potential bugs in your app. They are provided for developers who do not have their provisioning certificate yet.

Why would you want to use this static code analysis tool? Why wouldn’t you? It’s another tool in your belt to detect bugs early and is probably a good idea to do before submission to the App Store.

Ensure the project active build configuration is the Debug profile that Xcode sets up for you when you create a new iPhone project.
```
Project > Set Active Build Configuration > Debug
```
Navigate to your project directory in your shell and check that you can run xcodebuild from the root directory of your project.

If you haven’t run this tool before, you may run into this output:

$ xcodebuild
=== BUILDING NATIVE TARGET Property Machine OF PROJECT Test Project WITH THE DEFAULT CONFIGURATION (Debug) ===

Checking Dependencies...
CodeSign error: Code Signing Identity 'iPhone Developer' does not match any code-signing certificate in your keychain.  Once added to the keychain, touch a file or clean the project to continue.
** BUILD FAILED **

In this particular case, the project was set up to build against the iPhone device and not the iPhone simulator. Building for the device requires you to use an iPhone provisioning certificate which can be had for USD 99 from the Apple iPhone Developer Program. If you wish to write and compile software without a provisioning certificate - you may still test your app in the simulator provided with the SDK however you will not be able to test your app on the device itself.

You must change your build settings for the project.
```
Project > Edit Project Settings
```
Ensure the Base SDK is set to Simulator - iPhone OS 2.2 (or whichever version of the iPhone OS you wish to build for).
Ensure the Code Signing Identity is set to Don't Code Sign (if you don’t have a provisioning certificate yet).
Check the code signing SDK (expand the Code Signing Identity disclosure) and ensure it is also set to a simulator SDK. Also ensure the value for this option is Don't Code Sign.
You should end up with something like this:

Back in your shell, you should be able to run xcodebuild inside your project’s directory and your project should now build.
Download the analyzer tool from here, extract it and optionally add it to your path for easy accessibility.
The analyzer must be run after a xcodebuild clean command.

Do so like this:

xcodebuild clean
scan-build -k -V xcodebuild

After the tool has run, it launches its own web server and opens your default browser allowing you to view the results of the analysis.
You will be pleasantly surprised at the quality of the results:

Overview of Jekyll - a static site generator written in Ruby

2008-12-27T00:00:00+11:00

GitHub Pages was announced a couple of weeks ago and seemed to be received quite well. Although I find this service extremely awesome, especially now that they’ve gone even further with it than their initial announcement, the bit of news that excited me more was released much earlier: Jekyll. In fact, I used Jekyll to generate this site for me.

The connection between GitHub Pages and Jekyll is that each Pages-bound repo is piped through Jekyll automatically, allowing you to create a site and host it at GitHub. The beauty of this system is that all your posts and site changes are stored in your Git repo allowing your site the benefits of a distributed VCS such as the ability to sync your site on any machine, pull or push changes to the GitHub repo and publishing posts remotely by pushing new posts in raw markdown form to GitHub.

Of course you can choose not to use Jekyll with GitHub and opt for another VCS or eschew VCS altogether. So far you may be thinking that these benefits don’t seem like much compared to a typical Wordpress install and you’d be right! After all Wordpress offers version controlled posts, the ability to submit new posts from any web browser and import/export tools for your posts. People who choose to use Jekyll will probably have these requirements:

Limited hosting facilities such as lack of a DBMS.
Command-line junkies.
Need to merge changes and track revisions VCS style.

In fact, the latter point seems ideal for software projects with large groups of people contributing to the site documentation and is what looks to be addressed by the latest GitHub Pages generation feature for existing hosted projects.

As one redditor commented, the barrier to entry for Jekyll with a VCS is high. You must have knowledge of VCS’s as well as being very comfortable in the command line. If you are happy with that, the end gain and primary advantage of Jekyll + a VCS seems to be an efficient and transparent way of storing your version controlled posts, plus it lets you blog like a hacker.

Templating Troubles

The worst aspect in setting up this site was dealing with the templating engine Liquid. For example I could not find a way to loop over the site topics like so:

{% for topics in site.topic_list %}
	{% for post in site.topics.topic %}
		{{ post.date | date_to_long_string }} » {{ post.url }} {{ post.title }}
	{% endfor %}
{% endfor %}

(Just a note: there is no site.topic_list attribute. I forked the project and added it in /lib/site.rb. Also, there is no site.topics attribute in v0.3.0. If you wish to use it we’ll have to see if Tom accepts my pull request. Or you can apply my patch, or you can clone the forked repo.)

.topic_list was just a list of all topics belonging to a site. In this case, I was using Jekyll’s topics as tags and foregoing categories. The difference seems to me to be semantical and as I don’t require 2 levels of categorisation for posts, I chose the latter as it appears to me to be a cleaner way of organising posts in terms of directory structure, i.e.

/_posts/tag1/tag2/yyyy-mm-dd-post-title.markdown

as opposed to

/cat1/cat2/_posts/yyyy-mm-dd-post-title.markdown

So I wanted a method of tagging posts and the ability to display all tags with corresponding posts in a site. For this, I used the pre-existing topics functionality and added the .topics attribute to the Site Liquid object. This now allows me to display my posts (use the ‘topics’ link in this page’s header) organised by tag/topic.

The disadvantage of this comes back to Liquid. I could not figure out a way to use the enumerator inside the Liquid for loop tags as an attribute name (see above). So currently this means I must specify the topic name manually like so:

{% for post in site.topics.TOPICNAME %}
	{{ post.date | date_to_long_string }} » {{ post.url }} {{ post.title }}
{% endfor %}

Where TOPICNAME is ruby or github (i.e. the topics for this post, and all the topics currently in use for this site).

This was the only issue that came up and was not a dealbreaker. That is as complicated as the Liquid markup got.

How do I get started with Jekyll?

So from all this, you can see that Jekyll is still a young project but showing promise and has the ability right now to publish fully blown blogs. It will appeal to people who like to get their hands dirty. However once you have set up the infrastructure of your site, the process of creating and pushing your posts should be very simple.

Dr Nic has created a generator that creates a simple Jekyll site structure for you with a choice of 2 themes. It even includes the Disqus and Lighthouse code you need. Unfortunately he didn’t count on the agile skills of the GitHub team who released something even more polished a couple days later in the shape of their very own Pages generator which allows existing hosted repo owners to generate a project page from the Admin tab of their account.

The way in which you get started depends on the initial level of control you wish to have over the site.

You can install the tool on your own server sudo gem install mojombo-jekyll -s http://gems.github.com/ and generate a Jekyll site structure with Dr Nic’s generator and then tweak it to your liking.
If you already have a hosted project on GitHub, you can generate a project page from the Admin tab of your repo. (Note: this feature is currently unavailable. In place of the generate link is this text: ‘Page generation is disabled while we sort out a bug.’.)
Or you can create a GitHub account if you don’t have one already, set up a new repo called your_username.github.com and push your Jekyll structure to it in the master branch.

Static-site generators

Having only used Jekyll, I found the whole process very satisfying. There is a minimalist simplicity in the ability to serve pre-generated static pages instead of relying on a DBMS to store content for you. As well, all your posts are in simple text files stored in a logical directory structure which is (hopefully) backed up to a VCS in the cloud, minimising the chance that you could ever lose your posts.

The Unix-like approach of storing everything in text seems to suit this particular application because in essence, there is little that needs to be done to produce content, able to be rendered by web browsers. As well, I am using Vim to author this post - it almost doesn’t feel like I’m writing text. It feels like I’m doing some sort of sysadmin task and that is probably part of the appeal and why I like it so much.