<CharlieDigital/> Programming, Politics, and uhh…pineapples

20Sep/13Off

Step by Step Android Development with Cordova and IntelliJ

Posted by Charles Chen

There are other guides that cover this, but I found them lacking a bit in the step-by-step department for a beginner like myself.

Why IntelliJ?  Well first, Google has chosen it as the basis of their Android Studio IDE.  And second, Eclipse sucks 😀

Okay, so let's get started.

Download Pre-Requisites

  1. Download the Java SDK from here: http://www.oracle.com/technetwork/java/javase/downloads/index.html
  2. Download IntelliJ IDEA Community Edition from here: http://www.jetbrains.com/idea/download/
  3. Download and unzip the Android SDK or ADT Bundle from here: http://developer.android.com/sdk/index.html
  4. Download and unzip the PhoneGap package from here: http://phonegap.com/install/

If you download the Cordova package directly from the Apache site, you'll have to deal with the build process since it comes as source.  It's just all around easier to get started with the PhoneGap binaries.

Install Java

Follow the standard installation for the SDK, but after you finish, you will want to update your environment settings.

First, add the path to the Java binaries to your Path variable (right click on Computer -> Properties -> Advanced system settings -> Environment Variables).

Find the Path variable and add the path to the bin directory under where you installed the SDK:

path-var

Then click on New and add a variable called JAVA_HOME pointing to the directory where the SDK is installed:

java-home

If you open a command window and run javac, you should see the following:

javac-command

If not, you need to double check your settings.

Install IntelliJ

Not much to see here as the install is straight forward.  The installer will prompt you to run IntelliJ after it completes.

Create a New Project

Now we can create a new project and select Application Module under Android and then enter a project name and location:

new-project

Next, we need to select the Project SDK.  Click on the New button to create a new SDK mapping.  On your first run, it will prompt you to configure a Java SDK:

new-sdk

Click OK and select the location where the Java SDK was installed (C:\Program Files\Java\jdk1.7.0_04 for me).

Once you select that, you will be prompted to select the location of the Android SDK (E:\Experimental\android\adt-bundle-windows-x86_64-20130911\sdk for me).

It's the same dialog, so pay attention to the title.

Finally, you will see a dialog prompting you to Created New Android SDK:

create-android-sdk

When you are all done, it should look like so:

selected-android-sdk

Click Next and click Finish on the final screen:

finish-project-creation

Set Up Cordova

Now that we have our project set up, we need to hook Cordova into it.

Find the location where you unzipped the PhoneGap binaries and go to lib\android; you will find cordova.js and cordova-2.9.0.jar here.

Under the assets directory in the project, create a directory called www and paste the cordova.js file here.

Under the libs directory, paste the cordova-2.9.0.jar file here.

You need to right click on this file and select Add as Library.

Finally, under the res directory, paste the xml directory from the PhoneGap SDK and your directory should look like this:

project-setup

Create Content Files

Now we're ready to create our content files!  In the assets\www directory, add an HTML file called index.html:

create-html-file

Now update the HTML file with the following:

<!DOCTYPE html>
<html>
<head>
    <title>Demo Phonegap</title>
    <script type="text/javascript" charset="utf-8" src="cordova.js">
    </script>
</head>
<body>
<h2>Hello Android</h2>
</body>
</html>

Locate the file MyActivity.java and paste the following:

package com.example.CordovaExample;

import android.os.Bundle;
import org.apache.cordova.DroidGap;

public class MyActivity extends DroidGap {
    /**
     * Called when the activity is first created.
     */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        super.loadUrl("file:///android_asset/www/index.html");
    }
}

You'll noticed that even though the directory is assets\www, it must be androd_asset/www for the code to work.

According to the official Cordova docs, you need to update the file AndroidManifest.xml file (however, I didn't need to add it to run it at this stage).  You need to paste the following into your AndroidManifest.xml file:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
          package="com.example.CordovaExample"
          android:versionCode="1"
          android:versionName="1.0">
    <uses-sdk android:minSdkVersion="18"/>
    <supports-screens
            android:largeScreens="true"
            android:normalScreens="true"
            android:smallScreens="true"
            android:resizeable="true"
            android:anyDensity="true"/>
    <uses-permission android:name="android.permission.VIBRATE"/>
    <uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION"/>
    <uses-permission android:name="android.permission.ACCESS_FINE_LOCATION"/>
    <uses-permission android:name="android.permission.ACCESS_LOCATION_EXTRA_COMMANDS"/>
    <uses-permission android:name="android.permission.READ_PHONE_STATE"/>
    <uses-permission android:name="android.permission.INTERNET"/>
    <uses-permission android:name="android.permission.RECEIVE_SMS"/>
    <uses-permission android:name="android.permission.RECORD_AUDIO"/>
    <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>
    <uses-permission android:name="android.permission.READ_CONTACTS"/>
    <uses-permission android:name="android.permission.WRITE_CONTACTS"/>
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>
    <uses-permission android:name="android.permission.GET_ACCOUNTS"/>
    <uses-permission android:name="android.permission.BROADCAST_STICKY"/>
    <application android:label="@string/app_name" android:icon="@drawable/ic_launcher">
        <activity android:name="MyActivity"
                  android:label="@string/app_name"
                  android:screenOrientation="sensor"
                  android:configChanges="orientation|keyboardHidden|screenSize">
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>
                <category android:name="android.intent.category.LAUNCHER"/>
            </intent-filter>
        </activity>
    </application>
</manifest>

Set Up Virtual Device

Now you're ready to run!  You'll need to set up an Android Virtual Device (AVD) by selecting Tools -> Android -> AVD Manager:

create-avd

In this dialog, click on the Device Definitions tab and select Nexus S by Google and then click Create AVD:

selected-avd

Accept the defaults and enter 10 in the SD Card Size.  Now you're ready to build and run.

Build and Run

From the Build menu, select Make Project.  Once this completes successfully, click on Run -> Run 'CordovaExample'.

You should see your virtual device boot up now 😀 (it could take a while...)

The cool thing is that as you update your code and build, IntelliJ will automatically push the changes to the virtual device.

Filed under: Android, Dev, DevTools No Comments
19Sep/13Off

The Nuclear Arms Race and Deep Space Exploration

Posted by Charles Chen

Wired has a fascinating article which focuses on the world's dwindling supply of plutonium-238, a veritable super fuel in the field of deep space exploration:

In 1977, the Voyager 1 spacecraft left Earth on a four-year mission to explore Jupiter and Saturn. Thirty-six years later, the car-size probe is still exploring, still sending its findings home. It has now put more than 19 billion kilometers between itself and the sun. Last week NASA announced that Voyager 1 had become the first man-made object to reach interstellar space.

None of this would be possible without the spacecraft’s three batteries filled with plutonium-238. In fact, Most of what humanity knows about the outer planets came back to Earth on plutonium power.  Cassini’s ongoing exploration of Saturn, Galileo’s trip to Jupiter, Curiosity’s exploration of the surface of Mars, and the 2015 flyby of Pluto by the New Horizons spacecraft are all fueled by the stuff.

But there’s a problem: We’ve almost run out.

Most of the US supply of plutonium-238 was a byproduct of producing bomb-grade nuclear material.  Nowadays, the material is in incredibly short supply with demand, unyielding.

Of course, we have the capability to make more, except for the reality that pitiful funding that is required has been so difficult to obtain:

Since 1994, scientists have pleaded with lawmakers for the money to restart production. The DOE believes a relatively modest $10 to 20 million in funding each year through 2020 could yield an operation capable of making between 3.3 and 11 pounds of plutonium-238 annually — plenty to keep a steady stream of spacecraft in business.

It took countless scientists and their lobbyists more than 15 years just to get lawmakers’ attention. Congressional committees squabbled over if and how to spend $20 million of taxpayers’ money — it took them three years to make up their minds

Any hiccups in funding for plutonium-238 production could put planetary science into a tailspin and delay, strip down, or smother nuclear-powered missions.

Support science and share this story!

Filed under: Awesome, Science No Comments
12Sep/13Off

The Dichotomy of Change Control and Quality Software

Posted by Charles Chen

When processes attack!

When processes attack!
(Sybren A. Stüvel via photopin cc)

It may seem counter-intuitive to the old guard that change control can actually negatively affect the quality of software, but when implemented in such a way that it dramatically increases the cost and effort of making software improvements, it leads a team down a path of mediocrity.

You see, good developers naturally want to fix things that they find wrong with the software.  Tweak it to be a little faster.  Make a button bigger so it's easier to see.  Change the alignment of some form labels.  There are endless ways that good engineers will reflect upon their work or the work of their peers and find better ways to deliver a superior user experience.  Good teams will also find ways to quickly fold user feedback into the UX design and improve the usability of the system.

Indeed, there are times when changes are drastic and require that proportionally sufficient effort is put into testing, but there are times when changes are small and well contained where running the same process for big fixes simply yields frustration and drives a team to ultimately "settle" for fear of waking the slumbering giant of change control.

Consider a doctor performing open heart surgery.  Of course, the protocol for sanitation and safety checks are going to be far more rigorous than for administering a flu shot.  But this makes sense because the risk is far, far greater with one than the other.  If a doctor's protocols for preparing for surgery were one-size-fits all and applied for administering a flu shot, I suspect the quality of care in the US would decline drastically while causing the cost to skyrocket!  Or consider the permits and documentation required if you want to add a walkout entrance to your basement.  Of course you will need engineering specs, signed permits, and so on because the risk is high.  No one would think of requiring certification from a structural engineer and permits from your municipality to hang a picture frame on your wall because it would be absurd!

Likewise, common sense, pragmatism, and a sensible balance is a requirement for any software quality process; a one-size-fits-all approach creates a wall of effort for even minor tweaks that simply means that the small fixes that can make a big difference in the usability and functionality of the system aren't made for fear of generating a ton of e-paperwork.  The Broken Windows Theory, popularized by Steven D. Levitt and Stephen J. Dubner in Freakonomics, explains this phenomenon:

In an anonymous, urban environment, with few or no other people around, social norms and monitoring are not clearly known. Individuals thus look for signals within the environment as to the social norms in the setting and the risk of getting caught violating those norms; one of those signals is the area's general appearance.

Under the broken windows theory, an ordered and clean environment – one which is maintained – sends the signal that the area is monitored and that criminal behavior will not be tolerated. Conversely, a disordered environment – one which is not maintained (broken windows, graffiti, excessive litter) – sends the signal that the area is not monitored and that one can engage in criminal behavior with little risk of detection.

In the education realm, the broken windows theory is used to promote order in classrooms and school cultures. The belief is that students are signaled by disorder or rule-breaking and that they, in turn, imitate the disorder.

When change is made expensive, niggling things here and there in the code and UX that don't get fixed simply accumulate over time and promotes discord and disorder.  When the processes discourages innovation and excellence, a team ends up simply mired in mediocrity with no real quality to show for it.  It's great that there's now a trail of paperwork long enough to circle the Earth for the changes that were made, but the real shame are the fixes, improvements, and ideas that weren't implemented because of the cost in paperwork.

Again, this is not to say that no quality processes or change control should exist, but that their application must be proportional to the risk and pragmatic about the priorities of the project.

One way to solve this problem is via rigorous automation for it enables teams to condense the processes into robotic automatons that take minutes, not hours to execute.  It enables teams to conceptualize, develop, and deliver more rapidly through continuous deployment.

Wired had a good article on how LinkedIn is able to build and release new features quickly:

LinkedIn is a Wall Street darling, its stock up more than threefold in two years on soaring revenue, spiking profits, and seven straight quarters beating bankers’ estimates. But LinkedIn’s success isn’t just about numbers: an impressive acceleration of LinkedIn’s product cycle and a corresponding revolution in how LinkedIn writes software is a huge component in the company’s winning streak.

Much of LinkedIn’s success can be traced to changes made by Kevin Scott, the senior vice president of engineering and longtime Google veteran lured to LinkedIn in Feb. 2011, just before the buttoned-down social network went public. It was Scott and his team of programmers who completely overhauled how LinkedIn develops and ships new updates to its website and apps, taking a system that required a full month to release new features and turning it into one that pushes out updates multiple times per day.

By creating the tools and process and training the team to operate under continuous deployment, LinkedIn was able to quickly bring concepts and ideas to life; it is because the cost of making changes and improvements to the software have become cheap that they can be made readily and without friction.

True software quality can never be achieved solely through heavy-handed processes (unless they are automated!); such processes are simply paper tigers that lead to the appearance of conformance but instead, create an impedance to creating better software.

6Sep/13Off

SharePoint, Large Lists, Content Iterator, and an Alternative

Posted by Charles Chen

When retrieving large datasets from large SharePoint lists, SharePoint 2010 provides a new class called ContentIterator (CI) that is supposed to help make large dataset retrievals from large lists possible.

There's a bunch of great documentation on the web regarding this class, but one interesting observation that I've made is that it seems that it limits your query to one field only.  This means that in your query's where clause, you can only include one field, even when used with the order clause generated by ContentIterator.ItemEnumerationOrderByNVPField.

I tested with a list containing over 22,000 items with the default thresholds:

list-settings

And randomly generated data like so:

random-data

It turns out that if I use more than one field in the query, even with an index on each field in the query and using SPQueryThrottleOption.Override, the CI will fail the query with a threshold error.

What's one to do if you need to get all of the items in a list?

It seems that you should be able to just simply write a loop that executes the query and retrieves data, page-by-page, until you reach the end of the set.  So I rigged up the code myself:

/// <summary>
///     Executes a query and returns the result in batches.
/// </summary>
public class BatchQueryExector
{
    private SPQuery _query;

    private BatchQueryExector() {}

    /// <summary>
    ///     Creates an instance of the executor against the specified query.
    /// </summary>
    /// <param name="query">The query to execute.</param>
    /// <returns>The instance of the executor.</returns>
    public static BatchQueryExector WithQuery(SPQuery query)
    {
        BatchQueryExector executor = new BatchQueryExector();

        executor._query = query;

        return executor;
    }

    /// <summary>
    ///     Specifies the list the query will be executed over.
    /// </summary>
    /// <param name="list">The SharePoint list that contains the data.</param>
    /// <returns>An instance of <c>ExecutionContext</c>.</returns>
    public ExecutionContext OverList(SPList list)
    {
        return new ExecutionContext(_query, list);
    }

    /// <summary>
    ///     Inner class used to encapsulate the execution logic.
    /// </summary>
    public class ExecutionContext
    {
        private readonly SPList _list;
        private readonly SPQuery _query;

        /// <summary>
        ///     Creates a new instance of the context.
        /// </summary>
        /// <param name="query">The query to execute.</param>
        /// <param name="list">The SharePoint list that contains the data.</param>
        public ExecutionContext(SPQuery query, SPList list)
        {
            _query = query;
            _list = list;
        }

        /// <summary>
        ///     Retrieves the items in the list in batches based on the <c>RowLimit</c> and 
        ///     invokes the handler for each item.
        /// </summary>
        /// <param name="handler">A method which is invoked for each item.</param>
        public void GetItems(Action<SPListItem> handler)
        {
            string pagingToken = string.Empty;

            while (true)
            {
                _query.ListItemCollectionPosition = new SPListItemCollectionPosition(pagingToken);

                SPListItemCollection results = _list.GetItems(_query);

                foreach (SPListItem item in results)
                {
                    handler(item);
                }

                if (results.ListItemCollectionPosition == null)
                {
                    break; // EXIT; no more pages.
                }

                pagingToken = results.ListItemCollectionPosition.PagingInfo;
            }
        }
    }
}

This can be invoked like so:

internal class Program
{
    private static void Main(string[] args)
    {
        Program program = new Program();

        program.Run();
    }

    private void Run()
    {
        using (SPSite site = new SPSite("http://internal.dev.com"))
        using (SPWeb web = site.OpenWeb())
        {
            SPList list = web.Lists.TryGetList("TestPaging");

            SPQuery query = new SPQuery();
            query.Query = @"
<Where>        
        <And>
            <Eq>
                <FieldRef Name=""TstProgram"" />
                <Value Type=""Text"">Program 1</Value>
            </Eq>
            <Eq>
                <FieldRef Name=""TstDocumentType"" />
                <Value Type=""Text"">15 Day SUSAR</Value>
            </Eq>
        </And>      
</Where>";
            query.RowLimit = 100; // Effective batch size.
            query.QueryThrottleMode = SPQueryThrottleOption.Override;

            query.Query += ContentIterator.ItemEnumerationOrderByNVPField;

            Stopwatch stopwatch = new Stopwatch();

            stopwatch.Start();

            int count = 0;

            BatchQueryExector.WithQuery(query).OverList(list).GetItems(i => { count++; });

            stopwatch.Stop();

            Console.Out.WriteLine("{0}ms, {1} items", stopwatch.ElapsedMilliseconds, count);
        }
    }
}

It turns out that this works just fine provided that at least one of the columns in your query has an index. I tested with indices on all columns, on two columns, on one column, and on no columns.  With no indices, this query will fail as well (must have at least one index on one of the columns in your query in my testing, but my guess is that you will need more as the number of items increases).  With one to three indices, it made no difference in performance.  In fact, it got a little slower with three indices.

The batch size also had an impact.  Larger batch sizes were more efficient, which makes sense given that another database roundtrip is made for each batch.  For 32,000 items (I added more to test), a batch of 1000 (seems to be a sweet spot in my testing) completed in 2487ms for 7902 items.  A batch of 500 completed in 2550ms.  A batch of 100 completed in 3116ms.

It doesn't have the fancy bits of the CI, but it will work with multiple columns in your where clause.

To test for yourself, you can download this handy-dandy, multi-threaded list filler

Filed under: SharePoint No Comments
   
  • vocal
  • trap
  • trance
  • techno
  • symphonic-rock
  • spain
  • soundtrack
  • soul
  • singer
  • score
  • rock
  • rnb
  • reggaeton
  • reggae
  • rap
  • punk
  • progressive
  • post-grunge
  • pop
  • other
  • new-audio
  • metalcore
  • lounge
  • latino
  • jazz
  • instrumental
  • indie
  • house
  • hip-hop
  • heavy-metal
  • hard-rock
  • funk
  • folk
  • electronic
  • dubstep
  • drum-and-bass
  • downtempo
  • deep-house
  • dance
  • country
  • club-house
  • classical
  • classic-rock
  • chillout
  • breakbeat
  • blues
  • ambient
  • alternative-rock